CatalogManagerConfigurations/MasterStressTest.Test/0: ==693==ERROR: AddressSanitizer: heap-use-after-free on address 0x6190000f0088 at pc 0x0000009f3834 bp 0x7fe7687b94b0 sp 0x7fe7687b94a8
READ of size 4 at 0x6190000f0088 thread T56 (rpc worker-756)
W20250901 01:52:08.845160 994 connection.cc:537] client connection to 127.0.104.60:44231 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250901 01:52:08.845316 450 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_bb2ecd6ebafe402b85b81e6a112e6297" } new_table_name: "default.Table_3753991862534114bc316536ffba68c0" modify_external_catalogs: true
W20250901 01:52:08.846063 1103 heartbeater.cc:646] Failed to heartbeat to 127.0.104.60:44231 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250901 01:52:08.817392 903 tablet_service.cc:1468] Processing CreateTablet for tablet c445c300d71643609071f0f36528a9c4 (DEFAULT_TABLE table=default.Table_c766b74c950b433b963d93d396521bcd [id=739509ac727245c9a2d63f409e8f0f52]), partition=RANGE (key) PARTITION UNBOUNDED
W20250901 01:52:08.848481 716 connection.cc:537] client connection to 127.0.104.60:44231 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250901 01:52:08.848919 903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c445c300d71643609071f0f36528a9c4. 1 dirs total, 0 dirs full, 0 dirs failed
W20250901 01:52:08.849051 827 heartbeater.cc:646] Failed to heartbeat to 127.0.104.60:44231 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250901 01:52:08.849104 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_ac3b76f15c7743d0900e9a7a00348ec2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:08.849640 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ac3b76f15c7743d0900e9a7a00348ec2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250901 01:52:08.850175 442 connection.cc:537] client connection to 127.0.104.60:44231 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250901 01:52:08.851162 1164 raft_consensus.cc:2241] T 15c4c43fd20b46e4a337f5ac9c455537 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:08.851415 1164 raft_consensus.cc:2270] T 15c4c43fd20b46e4a337f5ac9c455537 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:08.817974 1150 ts_tablet_manager.cc:1428] T e9b24dde06be4bbdba74c559eb0d0275 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.122s user 0.000s sys 0.002s
I20250901 01:52:08.852774 1150 tablet_bootstrap.cc:492] T e6646f6a02cf426cb606312a5817684c P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:08.853338 452 master_service.cc:948] ReplaceTablet: received request to replace tablet 15c4c43fd20b46e4a337f5ac9c455537 from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:08.853276 758 tablet_service.cc:1468] Processing CreateTablet for tablet 8d5b10ede9c84be098795c0a787e5a78 (DEFAULT_TABLE table=default.Table_fadac7a1c3ae42c995aa1cc4414bbb7a [id=017c724f98ec404bab6e882ead9b8541]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:08.853827 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d5b10ede9c84be098795c0a787e5a78. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:08.854509 1150 tablet_bootstrap.cc:654] T e6646f6a02cf426cb606312a5817684c P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:08.822152 924 raft_consensus.cc:1273] T 0dce1909aa4848c4800910f50a95d1c6 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Refusing update from remote peer 0cf599f74322478d91475df234e58965: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250901 01:52:08.830132 902 tablet_service.cc:1515] Processing DeleteTablet for tablet 514f64dfb4a64103b60b084c6416171f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:08 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:08.841326 901 tablet_service.cc:1515] Processing DeleteTablet for tablet e9b24dde06be4bbdba74c559eb0d0275 with delete_type TABLET_DATA_DELETED (replaced by tablet 24ddc13d1bb04db4a7850acbff091577) from {username='slave'} at 127.0.0.1:52846
W20250901 01:52:08.848138 969 heartbeater.cc:646] Failed to heartbeat to 127.0.104.60:44231 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.0.104.60:44231 (error 108)
I20250901 01:52:08.861753 901 tablet_service.cc:1468] Processing CreateTablet for tablet 8d5b10ede9c84be098795c0a787e5a78 (DEFAULT_TABLE table=default.Table_fadac7a1c3ae42c995aa1cc4414bbb7a [id=017c724f98ec404bab6e882ead9b8541]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:08.862005 755 tablet_service.cc:1468] Processing CreateTablet for tablet 4f8e3fcaa19740d397f4d118ac4cbdb0 (DEFAULT_TABLE table=default.Table_aadaeb5310a74963a0c558f887b018f4 [id=1d0bc46740444f1fbf2129270e806259]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:08.862141 901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d5b10ede9c84be098795c0a787e5a78. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:08.862429 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f8e3fcaa19740d397f4d118ac4cbdb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:08.864766 1034 tablet_service.cc:1468] Processing CreateTablet for tablet 8d5b10ede9c84be098795c0a787e5a78 (DEFAULT_TABLE table=default.Table_fadac7a1c3ae42c995aa1cc4414bbb7a [id=017c724f98ec404bab6e882ead9b8541]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:08.865021 1164 ts_tablet_manager.cc:1905] T 15c4c43fd20b46e4a337f5ac9c455537 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:08.865242 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d5b10ede9c84be098795c0a787e5a78. 1 dirs total, 0 dirs full, 0 dirs failed
W20250901 01:52:08.870194 1195 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 01:52:08.846677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:44231 (local address 127.0.104.3:35921)
0901 01:52:08.846953 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0901 01:52:08.846976 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0901 01:52:08.847099 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 01:52:08.869218 (+ 22119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:44231: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":141}
I20250901 01:52:08.870124 1032 tablet_service.cc:1468] Processing CreateTablet for tablet 4f8e3fcaa19740d397f4d118ac4cbdb0 (DEFAULT_TABLE table=default.Table_aadaeb5310a74963a0c558f887b018f4 [id=1d0bc46740444f1fbf2129270e806259]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:08.870641 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f8e3fcaa19740d397f4d118ac4cbdb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:08.874068 1038 tablet_service.cc:1468] Processing CreateTablet for tablet 97e40c318f64455d91b0e4f8866990ce (DEFAULT_TABLE table=default.Table_bb2ecd6ebafe402b85b81e6a112e6297 [id=9af016a062a64218ad3b62e131d8b65d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:08.874490 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97e40c318f64455d91b0e4f8866990ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:08.877142 901 tablet_service.cc:1468] Processing CreateTablet for tablet 4f8e3fcaa19740d397f4d118ac4cbdb0 (DEFAULT_TABLE table=default.Table_aadaeb5310a74963a0c558f887b018f4 [id=1d0bc46740444f1fbf2129270e806259]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:08.877331 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 0dce1909aa4848c4800910f50a95d1c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:08 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:08.877632 901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f8e3fcaa19740d397f4d118ac4cbdb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:08.877966 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet 0dce1909aa4848c4800910f50a95d1c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:08 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:08.878203 1166 tablet_replica.cc:331] T 0dce1909aa4848c4800910f50a95d1c6 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:08.885121 1038 tablet_service.cc:1468] Processing CreateTablet for tablet 24ddc13d1bb04db4a7850acbff091577 (DEFAULT_TABLE table=default.Table_8c1924d21a91425cbbbd8429d56cf0ab [id=18aecbf3fe4643c4b589ee47706161bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:08.885749 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24ddc13d1bb04db4a7850acbff091577. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:08.891358 903 tablet_service.cc:1468] Processing CreateTablet for tablet 97e40c318f64455d91b0e4f8866990ce (DEFAULT_TABLE table=default.Table_bb2ecd6ebafe402b85b81e6a112e6297 [id=9af016a062a64218ad3b62e131d8b65d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:08.891878 903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97e40c318f64455d91b0e4f8866990ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:08.896349 902 tablet_service.cc:1468] Processing CreateTablet for tablet 7a8a3aca289c4735abeb9c95c414ab76 (DEFAULT_TABLE table=default.Table_8da589a069464922b2fc6b875a684173 [id=1d8f34cc14cb4e90a562ff4424b8577a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:08.896736 895 tablet_service.cc:1468] Processing CreateTablet for tablet 24ddc13d1bb04db4a7850acbff091577 (DEFAULT_TABLE table=default.Table_8c1924d21a91425cbbbd8429d56cf0ab [id=18aecbf3fe4643c4b589ee47706161bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:08.896826 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a8a3aca289c4735abeb9c95c414ab76. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:08.900763 1151 tablet_bootstrap.cc:492] T e9b24dde06be4bbdba74c559eb0d0275 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:08.900949 1151 ts_tablet_manager.cc:1397] T e9b24dde06be4bbdba74c559eb0d0275 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.204s user 0.023s sys 0.037s
I20250901 01:52:08.901582 1150 tablet_bootstrap.cc:492] T e6646f6a02cf426cb606312a5817684c P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:08.901821 1150 ts_tablet_manager.cc:1397] T e6646f6a02cf426cb606312a5817684c P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.049s user 0.002s sys 0.007s
I20250901 01:52:08.901777 1151 raft_consensus.cc:357] T e9b24dde06be4bbdba74c559eb0d0275 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:08.901966 1151 raft_consensus.cc:383] T e9b24dde06be4bbdba74c559eb0d0275 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:08.902099 1151 raft_consensus.cc:738] T e9b24dde06be4bbdba74c559eb0d0275 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:08.902428 1151 consensus_queue.cc:260] T e9b24dde06be4bbdba74c559eb0d0275 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:08.902807 1150 raft_consensus.cc:357] T e6646f6a02cf426cb606312a5817684c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:08.903050 1150 raft_consensus.cc:383] T e6646f6a02cf426cb606312a5817684c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:08.903205 1150 raft_consensus.cc:738] T e6646f6a02cf426cb606312a5817684c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:08.903535 1150 consensus_queue.cc:260] T e6646f6a02cf426cb606312a5817684c P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:08.904093 1151 ts_tablet_manager.cc:1428] T e9b24dde06be4bbdba74c559eb0d0275 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250901 01:52:08.905225 1150 ts_tablet_manager.cc:1428] T e6646f6a02cf426cb606312a5817684c P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250901 01:52:08.905396 1151 tablet_bootstrap.cc:492] T b11e8f77243e4764b96548d1132b8e5a P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:08.905603 1150 tablet_bootstrap.cc:492] T 81fa34655a71439a85b8a932b7603d7d P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:08.907116 1151 tablet_bootstrap.cc:654] T b11e8f77243e4764b96548d1132b8e5a P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:08.908334 442 consensus_peers.cc:489] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227 -> Peer 7ce2288f784044eea010664e3def133f (127.0.104.60:44231): Couldn't send request to peer 7ce2288f784044eea010664e3def133f. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.60:44231: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:08.906688 899 tablet_service.cc:1515] Processing DeleteTablet for tablet e6646f6a02cf426cb606312a5817684c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:08 UTC) from {username='slave'} at 127.0.0.1:52846
W20250901 01:52:08.910028 1215 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 01:52:08.856321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:44231 (local address 127.0.104.2:55647)
0901 01:52:08.908574 (+ 52253us) negotiation.cc:107] Waiting for socket to connect
0901 01:52:08.908757 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:44231: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":52096,"thread_start_us":52291,"threads_started":1}
I20250901 01:52:08.911228 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 8e6e96b4de214cadba1ccd19390cffb0 (DEFAULT_TABLE table=default.Table_05707bc6c5ca46c69e791188350c8615 [id=c1b1e9e3876244f68d31c63ce7bdebf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:08.911514 1150 tablet_bootstrap.cc:654] T 81fa34655a71439a85b8a932b7603d7d P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:08.911729 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e6e96b4de214cadba1ccd19390cffb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:08.916464 449 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_3063e0e5f956483db9161270ce39f4b9" } modify_external_catalogs: true
I20250901 01:52:08.916760 449 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_3063e0e5f956483db9161270ce39f4b9" } modify_external_catalogs: true
I20250901 01:52:08.918934 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24ddc13d1bb04db4a7850acbff091577. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:08.920706 448 master_service.cc:948] ReplaceTablet: received request to replace tablet e6646f6a02cf426cb606312a5817684c from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:08.922353 1150 tablet_bootstrap.cc:492] T 81fa34655a71439a85b8a932b7603d7d P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:08.922585 1150 ts_tablet_manager.cc:1397] T 81fa34655a71439a85b8a932b7603d7d P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.005s
I20250901 01:52:08.923417 1150 raft_consensus.cc:357] T 81fa34655a71439a85b8a932b7603d7d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:08.923715 1150 raft_consensus.cc:383] T 81fa34655a71439a85b8a932b7603d7d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:08.923849 1150 raft_consensus.cc:738] T 81fa34655a71439a85b8a932b7603d7d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:08.924180 1150 consensus_queue.cc:260] T 81fa34655a71439a85b8a932b7603d7d P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:08.926970 901 tablet_service.cc:1468] Processing CreateTablet for tablet 8e6e96b4de214cadba1ccd19390cffb0 (DEFAULT_TABLE table=default.Table_05707bc6c5ca46c69e791188350c8615 [id=c1b1e9e3876244f68d31c63ce7bdebf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:08.926869 761 tablet_service.cc:1468] Processing CreateTablet for tablet 24ddc13d1bb04db4a7850acbff091577 (DEFAULT_TABLE table=default.Table_8c1924d21a91425cbbbd8429d56cf0ab [id=18aecbf3fe4643c4b589ee47706161bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:08.927438 901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e6e96b4de214cadba1ccd19390cffb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:08.927350 1031 tablet_service.cc:1468] Processing CreateTablet for tablet 7a8a3aca289c4735abeb9c95c414ab76 (DEFAULT_TABLE table=default.Table_8da589a069464922b2fc6b875a684173 [id=1d8f34cc14cb4e90a562ff4424b8577a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:08.927614 761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24ddc13d1bb04db4a7850acbff091577. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:08.927799 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a8a3aca289c4735abeb9c95c414ab76. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:08.932931 759 tablet_service.cc:1468] Processing CreateTablet for tablet 7a8a3aca289c4735abeb9c95c414ab76 (DEFAULT_TABLE table=default.Table_8da589a069464922b2fc6b875a684173 [id=1d8f34cc14cb4e90a562ff4424b8577a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:08.933492 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a8a3aca289c4735abeb9c95c414ab76. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:08.934087 1150 ts_tablet_manager.cc:1428] T 81fa34655a71439a85b8a932b7603d7d P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20250901 01:52:08.934458 1150 tablet_bootstrap.cc:492] T 2dded0466e394e44b0303cefd1b9bc7e P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:08.934603 1164 ts_tablet_manager.cc:1918] T 15c4c43fd20b46e4a337f5ac9c455537 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:08.934727 1164 log.cc:1199] T 15c4c43fd20b46e4a337f5ac9c455537 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/15c4c43fd20b46e4a337f5ac9c455537
I20250901 01:52:08.935104 1164 ts_tablet_manager.cc:1939] T 15c4c43fd20b46e4a337f5ac9c455537 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:08.935770 1158 raft_consensus.cc:491] T e9b24dde06be4bbdba74c559eb0d0275 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:08.935922 1158 raft_consensus.cc:513] T e9b24dde06be4bbdba74c559eb0d0275 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:08.936149 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 15c4c43fd20b46e4a337f5ac9c455537 (table default.Table_cc9b70845c694a44bcf5116bce3512ca [id=d8d3ad39fdb347e9bb8769a5a9e72646]) successfully deleted
I20250901 01:52:08.937197 754 tablet_service.cc:1468] Processing CreateTablet for tablet 8e6e96b4de214cadba1ccd19390cffb0 (DEFAULT_TABLE table=default.Table_05707bc6c5ca46c69e791188350c8615 [id=c1b1e9e3876244f68d31c63ce7bdebf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:08.937729 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e6e96b4de214cadba1ccd19390cffb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:08.937984 1158 leader_election.cc:290] T e9b24dde06be4bbdba74c559eb0d0275 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:08.939419 450 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 97e40c318f64455d91b0e4f8866990ce consensus state has no leader
I20250901 01:52:08.940359 1150 tablet_bootstrap.cc:654] T 2dded0466e394e44b0303cefd1b9bc7e P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:08.941949 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 8f2e702121c845e0a09a254bbb81c142 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8f2e702121c845e0a09a254bbb81c142 already in progress: creating tablet
W20250901 01:52:08.942548 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet fab3c1a8928a4066a15868ba708267f1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fab3c1a8928a4066a15868ba708267f1 already in progress: creating tablet
W20250901 01:52:08.943146 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet c31993d35207443288835f90091f73cf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c31993d35207443288835f90091f73cf already in progress: creating tablet
I20250901 01:52:08.945434 447 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_84b7ac25a6b94ac0bbf0865c39abac94" } modify_external_catalogs: true
I20250901 01:52:08.945607 447 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_84b7ac25a6b94ac0bbf0865c39abac94" } modify_external_catalogs: true
I20250901 01:52:08.946177 450 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_cf1c9f5fc5104126b163225918707284" } modify_external_catalogs: true
I20250901 01:52:08.946354 450 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_cf1c9f5fc5104126b163225918707284" } modify_external_catalogs: true
W20250901 01:52:08.946445 440 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 4b14925e9dcb4c0db1170ae5e299d631 because the tablet was not found. No further retry: Not found: Tablet not found: 4b14925e9dcb4c0db1170ae5e299d631
W20250901 01:52:08.947090 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 514f64dfb4a64103b60b084c6416171f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 514f64dfb4a64103b60b084c6416171f already in progress: creating tablet
I20250901 01:52:08.947580 1164 tablet_replica.cc:331] T e9b24dde06be4bbdba74c559eb0d0275 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:08.948338 449 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet bf302e76746b4f7bbe4469914f83720f
I20250901 01:52:08.948520 1164 raft_consensus.cc:2241] T e9b24dde06be4bbdba74c559eb0d0275 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:08.948787 1164 raft_consensus.cc:2270] T e9b24dde06be4bbdba74c559eb0d0275 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:08.949288 446 master_service.cc:948] ReplaceTablet: received request to replace tablet 77405097e68344628f4eda667afaed21 from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:08.950157 895 tablet_service.cc:1515] Processing DeleteTablet for tablet bf302e76746b4f7bbe4469914f83720f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:08 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:08.950537 1038 tablet_service.cc:1515] Processing DeleteTablet for tablet bf302e76746b4f7bbe4469914f83720f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:08 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:08.951287 1164 ts_tablet_manager.cc:1905] T e9b24dde06be4bbdba74c559eb0d0275 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:08.960144 1164 ts_tablet_manager.cc:1918] T e9b24dde06be4bbdba74c559eb0d0275 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:08.960374 1164 log.cc:1199] T e9b24dde06be4bbdba74c559eb0d0275 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/e9b24dde06be4bbdba74c559eb0d0275
I20250901 01:52:08.961232 1164 ts_tablet_manager.cc:1939] T e9b24dde06be4bbdba74c559eb0d0275 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:08.961194 760 tablet_service.cc:1515] Processing DeleteTablet for tablet bf302e76746b4f7bbe4469914f83720f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:08 UTC) from {username='slave'} at 127.0.0.1:36178
I20250901 01:52:08.961715 753 tablet_service.cc:1468] Processing CreateTablet for tablet 97e40c318f64455d91b0e4f8866990ce (DEFAULT_TABLE table=default.Table_bb2ecd6ebafe402b85b81e6a112e6297 [id=9af016a062a64218ad3b62e131d8b65d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:08.962111 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet e9b24dde06be4bbdba74c559eb0d0275 (table default.Table_8c1924d21a91425cbbbd8429d56cf0ab [id=18aecbf3fe4643c4b589ee47706161bd]) successfully deleted
I20250901 01:52:08.962456 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97e40c318f64455d91b0e4f8866990ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:08.962548 1164 tablet_replica.cc:331] T 0dce1909aa4848c4800910f50a95d1c6 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:08.962775 1164 raft_consensus.cc:2241] T 0dce1909aa4848c4800910f50a95d1c6 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:08.963014 1164 raft_consensus.cc:2270] T 0dce1909aa4848c4800910f50a95d1c6 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:08.963595 1164 ts_tablet_manager.cc:1905] T 0dce1909aa4848c4800910f50a95d1c6 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:08.970861 1166 raft_consensus.cc:2241] T 0dce1909aa4848c4800910f50a95d1c6 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:08.974303 449 master_service.cc:948] ReplaceTablet: received request to replace tablet e6646f6a02cf426cb606312a5817684c from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:08.976222 1166 pending_rounds.cc:70] T 0dce1909aa4848c4800910f50a95d1c6 P fd27ab6665304aaea53adbd102dd4f58: Trying to abort 2 pending ops.
I20250901 01:52:08.976676 1166 pending_rounds.cc:77] T 0dce1909aa4848c4800910f50a95d1c6 P fd27ab6665304aaea53adbd102dd4f58: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7195408502041333760 op_type: ALTER_SCHEMA_OP alter_schema_request { tablet_id: "0dce1909aa4848c4800910f50a95d1c6" schema { columns { id: 10 name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } schema_version: 1 new_table_name: "default.Table_28fc024b8ecb4bd996601bc5035929c5" dest_uuid: "0cf599f74322478d91475df234e58965" new_extra_config { } }
I20250901 01:52:08.976958 1164 ts_tablet_manager.cc:1918] T 0dce1909aa4848c4800910f50a95d1c6 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:08.977078 1164 log.cc:1199] T 0dce1909aa4848c4800910f50a95d1c6 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/0dce1909aa4848c4800910f50a95d1c6
I20250901 01:52:08.977419 1164 ts_tablet_manager.cc:1939] T 0dce1909aa4848c4800910f50a95d1c6 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:08.977452 1166 pending_rounds.cc:77] T 0dce1909aa4848c4800910f50a95d1c6 P fd27ab6665304aaea53adbd102dd4f58: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7195408501114703872 op_type: NO_OP noop_request { }
I20250901 01:52:08.977614 1166 raft_consensus.cc:2887] T 0dce1909aa4848c4800910f50a95d1c6 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250901 01:52:08.977758 1166 raft_consensus.cc:2270] T 0dce1909aa4848c4800910f50a95d1c6 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:08.978540 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 0dce1909aa4848c4800910f50a95d1c6 (table default.Table_28fc024b8ecb4bd996601bc5035929c5 [id=22a4aeb52a4747d593d4a316583626b8]) successfully deleted
I20250901 01:52:08.980050 1164 tablet_replica.cc:331] T e6646f6a02cf426cb606312a5817684c P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:08.980334 1164 raft_consensus.cc:2241] T e6646f6a02cf426cb606312a5817684c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:08.980589 1164 raft_consensus.cc:2270] T e6646f6a02cf426cb606312a5817684c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:08.981446 1164 ts_tablet_manager.cc:1905] T e6646f6a02cf426cb606312a5817684c P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:08.983772 448 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 81fa34655a71439a85b8a932b7603d7d (table default.Table_196019dc3a1d4cbea8d5a87f139c899d [id=928ead2844c646a881f5f327cca0d3b9]). Expected version 2 got 0
I20250901 01:52:08.986104 895 tablet_service.cc:1468] Processing CreateTablet for tablet 4d333b670df84f308f0cbf6da636b768 (DEFAULT_TABLE table=default.Table_84b7ac25a6b94ac0bbf0865c39abac94 [id=c4ac57f676cd4799a98ad922c26ef14b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:08.986553 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d333b670df84f308f0cbf6da636b768. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:08.986668 448 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 81fa34655a71439a85b8a932b7603d7d (table default.Table_196019dc3a1d4cbea8d5a87f139c899d [id=928ead2844c646a881f5f327cca0d3b9]). Expected version 2 got 0
I20250901 01:52:08.990116 1151 tablet_bootstrap.cc:492] T b11e8f77243e4764b96548d1132b8e5a P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:08.990614 1151 ts_tablet_manager.cc:1397] T b11e8f77243e4764b96548d1132b8e5a P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.085s user 0.002s sys 0.001s
I20250901 01:52:08.992090 1164 ts_tablet_manager.cc:1918] T e6646f6a02cf426cb606312a5817684c P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:08.993021 1164 log.cc:1199] T e6646f6a02cf426cb606312a5817684c P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/e6646f6a02cf426cb606312a5817684c
I20250901 01:52:08.992993 1031 tablet_service.cc:1468] Processing CreateTablet for tablet 4d333b670df84f308f0cbf6da636b768 (DEFAULT_TABLE table=default.Table_84b7ac25a6b94ac0bbf0865c39abac94 [id=c4ac57f676cd4799a98ad922c26ef14b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:08.993430 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d333b670df84f308f0cbf6da636b768. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:08.993607 1164 ts_tablet_manager.cc:1939] T e6646f6a02cf426cb606312a5817684c P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:08.994681 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet e6646f6a02cf426cb606312a5817684c (table default.Table_59eb427e8a564f55a63a4c05bd7dbaf1 [id=5c8570beb0544e28877a757f16afb5bf]) successfully deleted
W20250901 01:52:08.996452 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet bf302e76746b4f7bbe4469914f83720f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bf302e76746b4f7bbe4469914f83720f already in progress: creating tablet
I20250901 01:52:08.998315 1038 tablet_service.cc:1468] Processing CreateTablet for tablet f0affc7a78634607852a48458412761d (DEFAULT_TABLE table=default.Table_72d74def1ee74948a379b9d1b6e51781 [id=535cf5f83c56443d93935ce80ff87efb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:08.998730 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0affc7a78634607852a48458412761d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:08.999135 901 tablet_service.cc:1468] Processing CreateTablet for tablet 447ffe73741640ceb73f35676232a094 (DEFAULT_TABLE table=default.Table_ac8663b3415c4c668e298a84b2deb8c3 [id=33cd13ca007c4308957250255f985e70]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:08.999526 901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 447ffe73741640ceb73f35676232a094. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:09.000712 1032 tablet_service.cc:1468] Processing CreateTablet for tablet 447ffe73741640ceb73f35676232a094 (DEFAULT_TABLE table=default.Table_ac8663b3415c4c668e298a84b2deb8c3 [id=33cd13ca007c4308957250255f985e70]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:09.001076 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 447ffe73741640ceb73f35676232a094. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:09.003821 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 4cdd369214ef4f5ebabddbf28b2ae3f7
I20250901 01:52:09.004760 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 342b7fde3ede494b9eec6e361db255dc (DEFAULT_TABLE table=default.Table_e6bab04ba03d4d9fa029e550a887e303 [id=405abb093fe44077b374dc89acb88328]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:09.004765 903 tablet_service.cc:1468] Processing CreateTablet for tablet 342b7fde3ede494b9eec6e361db255dc (DEFAULT_TABLE table=default.Table_e6bab04ba03d4d9fa029e550a887e303 [id=405abb093fe44077b374dc89acb88328]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:09.004931 757 tablet_service.cc:1468] Processing CreateTablet for tablet 342b7fde3ede494b9eec6e361db255dc (DEFAULT_TABLE table=default.Table_e6bab04ba03d4d9fa029e550a887e303 [id=405abb093fe44077b374dc89acb88328]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:09.005188 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 342b7fde3ede494b9eec6e361db255dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:09.005229 903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 342b7fde3ede494b9eec6e361db255dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:09.005240 1036 tablet_service.cc:1515] Processing DeleteTablet for tablet 4cdd369214ef4f5ebabddbf28b2ae3f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:08 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:09.005515 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 342b7fde3ede494b9eec6e361db255dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:09.006592 760 tablet_service.cc:1468] Processing CreateTablet for tablet f0affc7a78634607852a48458412761d (DEFAULT_TABLE table=default.Table_72d74def1ee74948a379b9d1b6e51781 [id=535cf5f83c56443d93935ce80ff87efb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:09.008731 1030 tablet_service.cc:1468] Processing CreateTablet for tablet eeae5294bd474f368f5766c260ccf3fe (DEFAULT_TABLE table=default.Table_2f7b3be40c0240f9af56af1fec427da1 [id=a48c2b3f8c4847cf9f9cf29548db5c50]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:09.009044 760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0affc7a78634607852a48458412761d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:09.009111 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eeae5294bd474f368f5766c260ccf3fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:09.009275 1166 ts_tablet_manager.cc:1905] T 0dce1909aa4848c4800910f50a95d1c6 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:09.010422 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet e9b24dde06be4bbdba74c559eb0d0275 with delete_type TABLET_DATA_DELETED (replaced by tablet 24ddc13d1bb04db4a7850acbff091577) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:09.010715 902 tablet_service.cc:1515] Processing DeleteTablet for tablet 4cdd369214ef4f5ebabddbf28b2ae3f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:08 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:09.008701 1034 tablet_service.cc:1468] Processing CreateTablet for tablet a4f5685260304d589dd467953d9bebfc (DEFAULT_TABLE table=default.Table_a2bf411424c74109a7c2baa28a4db8b9 [id=7c904bfb6cc94518873d6806d1fda600]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:09.010823 752 tablet_service.cc:1468] Processing CreateTablet for tablet a4f5685260304d589dd467953d9bebfc (DEFAULT_TABLE table=default.Table_a2bf411424c74109a7c2baa28a4db8b9 [id=7c904bfb6cc94518873d6806d1fda600]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:09.011348 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4f5685260304d589dd467953d9bebfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:09.012003 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4f5685260304d589dd467953d9bebfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:09.012699 899 tablet_service.cc:1468] Processing CreateTablet for tablet a4f5685260304d589dd467953d9bebfc (DEFAULT_TABLE table=default.Table_a2bf411424c74109a7c2baa28a4db8b9 [id=7c904bfb6cc94518873d6806d1fda600]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:09.013034 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4f5685260304d589dd467953d9bebfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:09.013944 750 tablet_service.cc:1468] Processing CreateTablet for tablet eeae5294bd474f368f5766c260ccf3fe (DEFAULT_TABLE table=default.Table_2f7b3be40c0240f9af56af1fec427da1 [id=a48c2b3f8c4847cf9f9cf29548db5c50]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:09.014482 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eeae5294bd474f368f5766c260ccf3fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:09.015106 450 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_9cf6434d754f45aabc920c0b3841f0c8" } new_table_name: "default.Table_947ec0e1a97a40b59be1e234bcaf4e0d" modify_external_catalogs: true
I20250901 01:52:09.015614 1158 raft_consensus.cc:491] T 81fa34655a71439a85b8a932b7603d7d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:09.015764 1158 raft_consensus.cc:513] T 81fa34655a71439a85b8a932b7603d7d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:09.015836 751 tablet_service.cc:1515] Processing DeleteTablet for tablet 4cdd369214ef4f5ebabddbf28b2ae3f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:08 UTC) from {username='slave'} at 127.0.0.1:36178
I20250901 01:52:09.016378 1158 leader_election.cc:290] T 81fa34655a71439a85b8a932b7603d7d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:09.017768 900 tablet_service.cc:1468] Processing CreateTablet for tablet eeae5294bd474f368f5766c260ccf3fe (DEFAULT_TABLE table=default.Table_2f7b3be40c0240f9af56af1fec427da1 [id=a48c2b3f8c4847cf9f9cf29548db5c50]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:09.018239 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eeae5294bd474f368f5766c260ccf3fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:09.018726 762 tablet_service.cc:1468] Processing CreateTablet for tablet 447ffe73741640ceb73f35676232a094 (DEFAULT_TABLE table=default.Table_ac8663b3415c4c668e298a84b2deb8c3 [id=33cd13ca007c4308957250255f985e70]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:09.019160 762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 447ffe73741640ceb73f35676232a094. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:09.019680 448 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_4121e29ecfc848658eb0b8e07b0ff24e" } modify_external_catalogs: true
I20250901 01:52:09.019847 448 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_4121e29ecfc848658eb0b8e07b0ff24e" } modify_external_catalogs: true
I20250901 01:52:09.022356 902 tablet_service.cc:1468] Processing CreateTablet for tablet 4cdd369214ef4f5ebabddbf28b2ae3f7 (DEFAULT_TABLE table=default.Table_cf1c9f5fc5104126b163225918707284 [id=00ae7e46c65d4a56b1ffa193b853421a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:09.022773 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cdd369214ef4f5ebabddbf28b2ae3f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:09.025072 751 tablet_service.cc:1468] Processing CreateTablet for tablet 4cdd369214ef4f5ebabddbf28b2ae3f7 (DEFAULT_TABLE table=default.Table_cf1c9f5fc5104126b163225918707284 [id=00ae7e46c65d4a56b1ffa193b853421a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:09.026573 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cdd369214ef4f5ebabddbf28b2ae3f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:09.030089 1166 ts_tablet_manager.cc:1918] T 0dce1909aa4848c4800910f50a95d1c6 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 01:52:09.030241 1166 log.cc:1199] T 0dce1909aa4848c4800910f50a95d1c6 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/0dce1909aa4848c4800910f50a95d1c6
I20250901 01:52:09.030715 1166 ts_tablet_manager.cc:1939] T 0dce1909aa4848c4800910f50a95d1c6 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:09.031643 1032 tablet_service.cc:1468] Processing CreateTablet for tablet 4cdd369214ef4f5ebabddbf28b2ae3f7 (DEFAULT_TABLE table=default.Table_cf1c9f5fc5104126b163225918707284 [id=00ae7e46c65d4a56b1ffa193b853421a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:09.032070 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cdd369214ef4f5ebabddbf28b2ae3f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:09.032011 441 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 0dce1909aa4848c4800910f50a95d1c6 (table default.Table_28fc024b8ecb4bd996601bc5035929c5 [id=22a4aeb52a4747d593d4a316583626b8]) successfully deleted
I20250901 01:52:09.035504 1158 raft_consensus.cc:491] T b11e8f77243e4764b96548d1132b8e5a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:09.035676 1158 raft_consensus.cc:513] T b11e8f77243e4764b96548d1132b8e5a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:09.036363 1158 leader_election.cc:290] T b11e8f77243e4764b96548d1132b8e5a P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:09.004760 895 tablet_service.cc:1468] Processing CreateTablet for tablet f0affc7a78634607852a48458412761d (DEFAULT_TABLE table=default.Table_72d74def1ee74948a379b9d1b6e51781 [id=535cf5f83c56443d93935ce80ff87efb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:09.038841 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0affc7a78634607852a48458412761d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:09.038960 449 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_776830720839484cbcd1934fb885056d" } new_table_name: "default.Table_1e00e9f110c7483bb1a077829c64f85b" modify_external_catalogs: true
I20250901 01:52:09.040354 454 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_fadac7a1c3ae42c995aa1cc4414bbb7a" } new_table_name: "default.Table_472dac9061664ce0afb468db1d8fa1de" modify_external_catalogs: true
I20250901 01:52:09.041673 451 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_3753991862534114bc316536ffba68c0" } new_table_name: "default.Table_0c6a1616a7b64455a3ae44c9983dc6c0" modify_external_catalogs: true
I20250901 01:52:09.042896 455 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_f8793d5bb6ca480491a9729761f3f61b" } modify_external_catalogs: true
I20250901 01:52:09.043059 455 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_f8793d5bb6ca480491a9729761f3f61b" } modify_external_catalogs: true
I20250901 01:52:09.043866 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_2447596ecd4242ceae8590ca488e4616"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:09.044430 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2447596ecd4242ceae8590ca488e4616 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:09.046260 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_95c28a4758e148368604152622fe51be"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:09.046789 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_95c28a4758e148368604152622fe51be in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250901 01:52:09.048508 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 4cdd369214ef4f5ebabddbf28b2ae3f7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4cdd369214ef4f5ebabddbf28b2ae3f7 already in progress: creating tablet
I20250901 01:52:08.992130 1151 raft_consensus.cc:357] T b11e8f77243e4764b96548d1132b8e5a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
W20250901 01:52:09.049455 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet bf302e76746b4f7bbe4469914f83720f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bf302e76746b4f7bbe4469914f83720f already in progress: creating tablet
I20250901 01:52:09.049643 1151 raft_consensus.cc:383] T b11e8f77243e4764b96548d1132b8e5a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:09.049943 1151 raft_consensus.cc:738] T b11e8f77243e4764b96548d1132b8e5a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
W20250901 01:52:09.050069 441 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 4cdd369214ef4f5ebabddbf28b2ae3f7 because the tablet was not found. No further retry: Not found: Tablet not found: 4cdd369214ef4f5ebabddbf28b2ae3f7
I20250901 01:52:09.050382 1166 tablet_replica.cc:331] T e9b24dde06be4bbdba74c559eb0d0275 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:09.050611 1166 raft_consensus.cc:2241] T e9b24dde06be4bbdba74c559eb0d0275 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:09.050812 1166 raft_consensus.cc:2270] T e9b24dde06be4bbdba74c559eb0d0275 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:09.050846 1151 consensus_queue.cc:260] T b11e8f77243e4764b96548d1132b8e5a P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:09.051762 1150 tablet_bootstrap.cc:492] T 2dded0466e394e44b0303cefd1b9bc7e P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:09.051900 1150 ts_tablet_manager.cc:1397] T 2dded0466e394e44b0303cefd1b9bc7e P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.118s user 0.007s sys 0.026s
I20250901 01:52:09.052682 1150 raft_consensus.cc:357] T 2dded0466e394e44b0303cefd1b9bc7e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:09.052861 1150 raft_consensus.cc:383] T 2dded0466e394e44b0303cefd1b9bc7e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:09.052932 1150 raft_consensus.cc:738] T 2dded0466e394e44b0303cefd1b9bc7e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:09.053140 1150 consensus_queue.cc:260] T 2dded0466e394e44b0303cefd1b9bc7e P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:09.054219 1150 ts_tablet_manager.cc:1428] T 2dded0466e394e44b0303cefd1b9bc7e P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250901 01:52:09.054479 1150 tablet_bootstrap.cc:492] T ce2a1ba75d674c95bdba9d6f12273bee P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:09.054229 1151 ts_tablet_manager.cc:1428] T b11e8f77243e4764b96548d1132b8e5a P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.063s user 0.001s sys 0.001s
I20250901 01:52:09.057246 1151 tablet_bootstrap.cc:492] T c33dea020a5e43d8ad8b52c39cb89a8b P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:09.060837 1150 tablet_bootstrap.cc:654] T ce2a1ba75d674c95bdba9d6f12273bee P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:09.065450 1150 tablet_bootstrap.cc:492] T ce2a1ba75d674c95bdba9d6f12273bee P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:09.065622 1150 ts_tablet_manager.cc:1397] T ce2a1ba75d674c95bdba9d6f12273bee P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20250901 01:52:09.064556 781 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "81fa34655a71439a85b8a932b7603d7d" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf599f74322478d91475df234e58965" is_pre_election: true
I20250901 01:52:09.066089 781 raft_consensus.cc:2466] T 81fa34655a71439a85b8a932b7603d7d P 0cf599f74322478d91475df234e58965 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00c282f779c1477d920f24dc0dc16628 in term 0.
I20250901 01:52:09.066298 1150 raft_consensus.cc:357] T ce2a1ba75d674c95bdba9d6f12273bee P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:09.066490 1150 raft_consensus.cc:383] T ce2a1ba75d674c95bdba9d6f12273bee P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:09.066556 1150 raft_consensus.cc:738] T ce2a1ba75d674c95bdba9d6f12273bee P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:09.066759 1150 consensus_queue.cc:260] T ce2a1ba75d674c95bdba9d6f12273bee P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:09.067167 858 leader_election.cc:304] T 81fa34655a71439a85b8a932b7603d7d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965; no voters:
I20250901 01:52:09.067628 1158 raft_consensus.cc:2802] T 81fa34655a71439a85b8a932b7603d7d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:09.067762 1158 raft_consensus.cc:491] T 81fa34655a71439a85b8a932b7603d7d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:09.067812 1150 ts_tablet_manager.cc:1428] T ce2a1ba75d674c95bdba9d6f12273bee P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250901 01:52:09.067885 1158 raft_consensus.cc:3058] T 81fa34655a71439a85b8a932b7603d7d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:09.068135 1150 tablet_bootstrap.cc:492] T c31993d35207443288835f90091f73cf P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:09.068291 1151 tablet_bootstrap.cc:654] T c33dea020a5e43d8ad8b52c39cb89a8b P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:09.069908 1150 tablet_bootstrap.cc:654] T c31993d35207443288835f90091f73cf P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:09.071549 1158 raft_consensus.cc:513] T 81fa34655a71439a85b8a932b7603d7d P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:09.072700 1158 leader_election.cc:290] T 81fa34655a71439a85b8a932b7603d7d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:09.073323 1151 tablet_bootstrap.cc:492] T c33dea020a5e43d8ad8b52c39cb89a8b P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:09.076593 1151 ts_tablet_manager.cc:1397] T c33dea020a5e43d8ad8b52c39cb89a8b P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.001s
I20250901 01:52:09.077723 1151 raft_consensus.cc:357] T c33dea020a5e43d8ad8b52c39cb89a8b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:09.078759 447 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 4d333b670df84f308f0cbf6da636b768
I20250901 01:52:09.078967 1151 raft_consensus.cc:383] T c33dea020a5e43d8ad8b52c39cb89a8b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:09.079241 1151 raft_consensus.cc:738] T c33dea020a5e43d8ad8b52c39cb89a8b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:09.079767 1151 consensus_queue.cc:260] T c33dea020a5e43d8ad8b52c39cb89a8b P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:09.080127 446 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 77405097e68344628f4eda667afaed21
I20250901 01:52:09.080602 446 catalog_manager.cc:6652] ReplaceTablet: tablet 77405097e68344628f4eda667afaed21 deleted and replaced by tablet 7f2546a3f6da4b3297ebcbf006eeb3f2
I20250901 01:52:09.082051 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 4d333b670df84f308f0cbf6da636b768 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:08 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:09.082276 900 tablet_service.cc:1515] Processing DeleteTablet for tablet 77405097e68344628f4eda667afaed21 with delete_type TABLET_DATA_DELETED (replaced by tablet 7f2546a3f6da4b3297ebcbf006eeb3f2) from {username='slave'} at 127.0.0.1:52846
W20250901 01:52:09.083060 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 4d333b670df84f308f0cbf6da636b768 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4d333b670df84f308f0cbf6da636b768 already in progress: creating tablet
W20250901 01:52:09.083300 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 77405097e68344628f4eda667afaed21 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 77405097e68344628f4eda667afaed21 already in progress: creating tablet
I20250901 01:52:09.084156 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet 4d333b670df84f308f0cbf6da636b768 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:08 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:09.084425 1034 tablet_service.cc:1515] Processing DeleteTablet for tablet 77405097e68344628f4eda667afaed21 with delete_type TABLET_DATA_DELETED (replaced by tablet 7f2546a3f6da4b3297ebcbf006eeb3f2) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:09.086688 1151 ts_tablet_manager.cc:1428] T c33dea020a5e43d8ad8b52c39cb89a8b P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20250901 01:52:09.088217 1150 tablet_bootstrap.cc:492] T c31993d35207443288835f90091f73cf P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:09.088361 1150 ts_tablet_manager.cc:1397] T c31993d35207443288835f90091f73cf P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.006s
I20250901 01:52:09.089239 1150 raft_consensus.cc:357] T c31993d35207443288835f90091f73cf P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:09.089424 1150 raft_consensus.cc:383] T c31993d35207443288835f90091f73cf P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:09.089510 1150 raft_consensus.cc:738] T c31993d35207443288835f90091f73cf P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:09.089745 1150 consensus_queue.cc:260] T c31993d35207443288835f90091f73cf P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:09.090301 1150 ts_tablet_manager.cc:1428] T c31993d35207443288835f90091f73cf P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250901 01:52:09.090550 1150 tablet_bootstrap.cc:492] T 514f64dfb4a64103b60b084c6416171f P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
W20250901 01:52:09.091434 442 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.61:42375, user_credentials={real_user=slave}} blocked reactor thread for 82449.8us
I20250901 01:52:09.091991 1151 tablet_bootstrap.cc:492] T 2dded0466e394e44b0303cefd1b9bc7e P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:09.092191 1150 tablet_bootstrap.cc:654] T 514f64dfb4a64103b60b084c6416171f P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:09.093642 441 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 81fa34655a71439a85b8a932b7603d7d (table default.Table_196019dc3a1d4cbea8d5a87f139c899d [id=928ead2844c646a881f5f327cca0d3b9], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 81fa34655a71439a85b8a932b7603d7d consensus state has no leader
I20250901 01:52:09.098153 1151 tablet_bootstrap.cc:654] T 2dded0466e394e44b0303cefd1b9bc7e P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:09.100085 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 77405097e68344628f4eda667afaed21 with delete_type TABLET_DATA_DELETED (replaced by tablet 7f2546a3f6da4b3297ebcbf006eeb3f2) from {username='slave'} at 127.0.0.1:36178
W20250901 01:52:09.101087 858 leader_election.cc:343] T e9b24dde06be4bbdba74c559eb0d0275 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Not found: Tablet not found: e9b24dde06be4bbdba74c559eb0d0275
I20250901 01:52:09.098663 780 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b11e8f77243e4764b96548d1132b8e5a" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf599f74322478d91475df234e58965" is_pre_election: true
I20250901 01:52:09.102121 780 raft_consensus.cc:2466] T b11e8f77243e4764b96548d1132b8e5a P 0cf599f74322478d91475df234e58965 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00c282f779c1477d920f24dc0dc16628 in term 0.
I20250901 01:52:09.102433 781 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "81fa34655a71439a85b8a932b7603d7d" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf599f74322478d91475df234e58965"
I20250901 01:52:09.102638 781 raft_consensus.cc:3058] T 81fa34655a71439a85b8a932b7603d7d P 0cf599f74322478d91475df234e58965 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:09.103451 858 leader_election.cc:304] T b11e8f77243e4764b96548d1132b8e5a P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965; no voters:
I20250901 01:52:09.103816 1158 raft_consensus.cc:2802] T b11e8f77243e4764b96548d1132b8e5a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:09.103920 1158 raft_consensus.cc:491] T b11e8f77243e4764b96548d1132b8e5a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:09.103854 1151 tablet_bootstrap.cc:492] T 2dded0466e394e44b0303cefd1b9bc7e P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:09.104048 1158 raft_consensus.cc:3058] T b11e8f77243e4764b96548d1132b8e5a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:09.104240 1151 ts_tablet_manager.cc:1397] T 2dded0466e394e44b0303cefd1b9bc7e P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20250901 01:52:09.105266 1151 raft_consensus.cc:357] T 2dded0466e394e44b0303cefd1b9bc7e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:09.106448 1151 raft_consensus.cc:383] T 2dded0466e394e44b0303cefd1b9bc7e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:09.106694 1151 raft_consensus.cc:738] T 2dded0466e394e44b0303cefd1b9bc7e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:09.107187 1151 consensus_queue.cc:260] T 2dded0466e394e44b0303cefd1b9bc7e P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:09.107287 1158 raft_consensus.cc:513] T b11e8f77243e4764b96548d1132b8e5a P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:09.108332 1158 leader_election.cc:290] T b11e8f77243e4764b96548d1132b8e5a P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:09.111361 1151 ts_tablet_manager.cc:1428] T 2dded0466e394e44b0303cefd1b9bc7e P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250901 01:52:09.114655 1150 tablet_bootstrap.cc:492] T 514f64dfb4a64103b60b084c6416171f P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:09.114899 1150 ts_tablet_manager.cc:1397] T 514f64dfb4a64103b60b084c6416171f P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.021s
I20250901 01:52:09.114840 780 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b11e8f77243e4764b96548d1132b8e5a" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf599f74322478d91475df234e58965"
I20250901 01:52:09.115048 1151 tablet_bootstrap.cc:492] T ce2a1ba75d674c95bdba9d6f12273bee P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:09.115053 780 raft_consensus.cc:3058] T b11e8f77243e4764b96548d1132b8e5a P 0cf599f74322478d91475df234e58965 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:09.116086 1150 raft_consensus.cc:357] T 514f64dfb4a64103b60b084c6416171f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:09.116387 1150 raft_consensus.cc:383] T 514f64dfb4a64103b60b084c6416171f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:09.116495 1150 raft_consensus.cc:738] T 514f64dfb4a64103b60b084c6416171f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:09.116808 1150 consensus_queue.cc:260] T 514f64dfb4a64103b60b084c6416171f P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:09.117681 1150 ts_tablet_manager.cc:1428] T 514f64dfb4a64103b60b084c6416171f P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250901 01:52:09.118095 1150 tablet_bootstrap.cc:492] T 8f2e702121c845e0a09a254bbb81c142 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:09.118975 1151 tablet_bootstrap.cc:654] T ce2a1ba75d674c95bdba9d6f12273bee P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:09.120661 1150 tablet_bootstrap.cc:654] T 8f2e702121c845e0a09a254bbb81c142 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:09.143493 1166 ts_tablet_manager.cc:1905] T e9b24dde06be4bbdba74c559eb0d0275 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:09.153157 762 tablet_service.cc:1515] Processing DeleteTablet for tablet 4d333b670df84f308f0cbf6da636b768 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:08 UTC) from {username='slave'} at 127.0.0.1:36178
I20250901 01:52:09.153818 1166 ts_tablet_manager.cc:1918] T e9b24dde06be4bbdba74c559eb0d0275 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:09.154093 1166 log.cc:1199] T e9b24dde06be4bbdba74c559eb0d0275 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/e9b24dde06be4bbdba74c559eb0d0275
I20250901 01:52:09.154623 1166 ts_tablet_manager.cc:1939] T e9b24dde06be4bbdba74c559eb0d0275 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:09.156111 441 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet e9b24dde06be4bbdba74c559eb0d0275 (table default.Table_8c1924d21a91425cbbbd8429d56cf0ab [id=18aecbf3fe4643c4b589ee47706161bd]) successfully deleted
W20250901 01:52:09.156440 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 4d333b670df84f308f0cbf6da636b768 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4d333b670df84f308f0cbf6da636b768 already in progress: creating tablet
W20250901 01:52:09.157577 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 77405097e68344628f4eda667afaed21 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 77405097e68344628f4eda667afaed21 already in progress: creating tablet
I20250901 01:52:09.157811 780 raft_consensus.cc:2466] T b11e8f77243e4764b96548d1132b8e5a P 0cf599f74322478d91475df234e58965 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c282f779c1477d920f24dc0dc16628 in term 1.
I20250901 01:52:09.158344 858 leader_election.cc:304] T b11e8f77243e4764b96548d1132b8e5a P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965; no voters:
I20250901 01:52:09.158753 1158 raft_consensus.cc:2802] T b11e8f77243e4764b96548d1132b8e5a P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:09.163854 1058 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "81fa34655a71439a85b8a932b7603d7d" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:09.163882 1055 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "81fa34655a71439a85b8a932b7603d7d" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58"
W20250901 01:52:09.163995 859 leader_election.cc:343] T e9b24dde06be4bbdba74c559eb0d0275 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Not found: Tablet not found: e9b24dde06be4bbdba74c559eb0d0275
I20250901 01:52:09.164157 859 leader_election.cc:304] T e9b24dde06be4bbdba74c559eb0d0275 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:09.164201 1058 raft_consensus.cc:2466] T 81fa34655a71439a85b8a932b7603d7d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00c282f779c1477d920f24dc0dc16628 in term 0.
I20250901 01:52:09.164139 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b11e8f77243e4764b96548d1132b8e5a" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:09.164429 859 raft_consensus.cc:2684] T e9b24dde06be4bbdba74c559eb0d0275 P 00c282f779c1477d920f24dc0dc16628: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250901 01:52:09.164451 1056 raft_consensus.cc:2466] T b11e8f77243e4764b96548d1132b8e5a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00c282f779c1477d920f24dc0dc16628 in term 0.
I20250901 01:52:09.164722 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b11e8f77243e4764b96548d1132b8e5a" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58"
I20250901 01:52:09.164880 1057 raft_consensus.cc:3058] T b11e8f77243e4764b96548d1132b8e5a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:09.167845 1057 raft_consensus.cc:2466] T b11e8f77243e4764b96548d1132b8e5a P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c282f779c1477d920f24dc0dc16628 in term 1.
I20250901 01:52:09.171005 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_68682089b6364db98892ca5d82ed1322"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:09.171849 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_68682089b6364db98892ca5d82ed1322 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:09.174052 1151 tablet_bootstrap.cc:492] T ce2a1ba75d674c95bdba9d6f12273bee P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:09.174218 1151 ts_tablet_manager.cc:1397] T ce2a1ba75d674c95bdba9d6f12273bee P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.060s user 0.015s sys 0.008s
I20250901 01:52:09.175019 1151 raft_consensus.cc:357] T ce2a1ba75d674c95bdba9d6f12273bee P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:09.175422 447 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet b11e8f77243e4764b96548d1132b8e5a (table default.Table_16b3ac9fb2304c8095e0270038501db0 [id=a39e8b171a874df1890d072732e55df9]). Expected version 1 got 0
I20250901 01:52:09.176009 447 master_service.cc:948] ReplaceTablet: received request to replace tablet 4f8e3fcaa19740d397f4d118ac4cbdb0 from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:09.177381 900 tablet_service.cc:1515] Processing DeleteTablet for tablet 2dded0466e394e44b0303cefd1b9bc7e with delete_type TABLET_DATA_DELETED (replaced by tablet 7a8a3aca289c4735abeb9c95c414ab76) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:09.177803 1164 tablet_replica.cc:331] T 2dded0466e394e44b0303cefd1b9bc7e P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:09.178056 1164 raft_consensus.cc:2241] T 2dded0466e394e44b0303cefd1b9bc7e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:09.178294 1164 raft_consensus.cc:2270] T 2dded0466e394e44b0303cefd1b9bc7e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:09.175217 1151 raft_consensus.cc:383] T ce2a1ba75d674c95bdba9d6f12273bee P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:09.179214 1151 raft_consensus.cc:738] T ce2a1ba75d674c95bdba9d6f12273bee P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:09.179257 1164 ts_tablet_manager.cc:1905] T 2dded0466e394e44b0303cefd1b9bc7e P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:09.179764 1151 consensus_queue.cc:260] T ce2a1ba75d674c95bdba9d6f12273bee P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:09.184420 1151 ts_tablet_manager.cc:1428] T ce2a1ba75d674c95bdba9d6f12273bee P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20250901 01:52:09.187662 1150 tablet_bootstrap.cc:492] T 8f2e702121c845e0a09a254bbb81c142 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:09.188014 1150 ts_tablet_manager.cc:1397] T 8f2e702121c845e0a09a254bbb81c142 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.070s user 0.015s sys 0.035s
I20250901 01:52:09.188899 1151 tablet_bootstrap.cc:492] T 514f64dfb4a64103b60b084c6416171f P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:09.189438 1150 raft_consensus.cc:357] T 8f2e702121c845e0a09a254bbb81c142 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:09.189843 1150 raft_consensus.cc:383] T 8f2e702121c845e0a09a254bbb81c142 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:09.190042 1150 raft_consensus.cc:738] T 8f2e702121c845e0a09a254bbb81c142 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:09.190433 1150 consensus_queue.cc:260] T 8f2e702121c845e0a09a254bbb81c142 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:09.192814 1151 tablet_bootstrap.cc:654] T 514f64dfb4a64103b60b084c6416171f P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:09.198310 1150 ts_tablet_manager.cc:1428] T 8f2e702121c845e0a09a254bbb81c142 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.010s user 0.006s sys 0.002s
I20250901 01:52:09.198905 1150 tablet_bootstrap.cc:492] T 3eaf8baac9f1485f944cd9e74b8b5569 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:09.200469 1164 ts_tablet_manager.cc:1918] T 2dded0466e394e44b0303cefd1b9bc7e P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:09.200616 1164 log.cc:1199] T 2dded0466e394e44b0303cefd1b9bc7e P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/2dded0466e394e44b0303cefd1b9bc7e
I20250901 01:52:09.201082 1164 ts_tablet_manager.cc:1939] T 2dded0466e394e44b0303cefd1b9bc7e P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:09.201261 1151 tablet_bootstrap.cc:492] T 514f64dfb4a64103b60b084c6416171f P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:09.201452 1151 ts_tablet_manager.cc:1397] T 514f64dfb4a64103b60b084c6416171f P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.004s
I20250901 01:52:09.201751 1150 tablet_bootstrap.cc:654] T 3eaf8baac9f1485f944cd9e74b8b5569 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:09.201782 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 2dded0466e394e44b0303cefd1b9bc7e (table default.Table_8da589a069464922b2fc6b875a684173 [id=1d8f34cc14cb4e90a562ff4424b8577a]) successfully deleted
I20250901 01:52:09.202354 1151 raft_consensus.cc:357] T 514f64dfb4a64103b60b084c6416171f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:09.202541 1151 raft_consensus.cc:383] T 514f64dfb4a64103b60b084c6416171f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:09.202636 1151 raft_consensus.cc:738] T 514f64dfb4a64103b60b084c6416171f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:09.204649 758 tablet_service.cc:1468] Processing CreateTablet for tablet 4d333b670df84f308f0cbf6da636b768 (DEFAULT_TABLE table=default.Table_84b7ac25a6b94ac0bbf0865c39abac94 [id=c4ac57f676cd4799a98ad922c26ef14b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:09.204941 1151 consensus_queue.cc:260] T 514f64dfb4a64103b60b084c6416171f P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:09.205628 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d333b670df84f308f0cbf6da636b768. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:09.209314 1151 ts_tablet_manager.cc:1428] T 514f64dfb4a64103b60b084c6416171f P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
W20250901 01:52:09.210189 450 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3eaf8baac9f1485f944cd9e74b8b5569 consensus state has no leader
I20250901 01:52:09.210883 1150 tablet_bootstrap.cc:492] T 3eaf8baac9f1485f944cd9e74b8b5569 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:09.211128 1150 ts_tablet_manager.cc:1397] T 3eaf8baac9f1485f944cd9e74b8b5569 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20250901 01:52:09.212236 1150 raft_consensus.cc:357] T 3eaf8baac9f1485f944cd9e74b8b5569 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:09.212584 1150 raft_consensus.cc:383] T 3eaf8baac9f1485f944cd9e74b8b5569 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:09.212713 1150 raft_consensus.cc:738] T 3eaf8baac9f1485f944cd9e74b8b5569 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:09.213070 1150 consensus_queue.cc:260] T 3eaf8baac9f1485f944cd9e74b8b5569 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:09.213369 1151 tablet_bootstrap.cc:492] T 3eaf8baac9f1485f944cd9e74b8b5569 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:09.213991 1150 ts_tablet_manager.cc:1428] T 3eaf8baac9f1485f944cd9e74b8b5569 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.003s user 0.001s sys 0.003s
I20250901 01:52:09.214411 1150 tablet_bootstrap.cc:492] T e8621a0c15a94ed7bd63b035ba2fded7 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:09.217674 1150 tablet_bootstrap.cc:654] T e8621a0c15a94ed7bd63b035ba2fded7 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:09.218586 1151 tablet_bootstrap.cc:654] T 3eaf8baac9f1485f944cd9e74b8b5569 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:09.225028 781 raft_consensus.cc:2466] T 81fa34655a71439a85b8a932b7603d7d P 0cf599f74322478d91475df234e58965 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c282f779c1477d920f24dc0dc16628 in term 1.
I20250901 01:52:09.225687 858 leader_election.cc:304] T 81fa34655a71439a85b8a932b7603d7d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965; no voters: fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:09.226003 1151 tablet_bootstrap.cc:492] T 3eaf8baac9f1485f944cd9e74b8b5569 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:09.226069 1234 raft_consensus.cc:2802] T 81fa34655a71439a85b8a932b7603d7d P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:09.226166 1151 ts_tablet_manager.cc:1397] T 3eaf8baac9f1485f944cd9e74b8b5569 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.003s
I20250901 01:52:09.226282 1234 raft_consensus.cc:695] T 81fa34655a71439a85b8a932b7603d7d P 00c282f779c1477d920f24dc0dc16628 [term 1 LEADER]: Becoming Leader. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Running, Role: LEADER
I20250901 01:52:09.227042 1151 raft_consensus.cc:357] T 3eaf8baac9f1485f944cd9e74b8b5569 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:09.227276 1151 raft_consensus.cc:383] T 3eaf8baac9f1485f944cd9e74b8b5569 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:09.226949 1234 consensus_queue.cc:237] T 81fa34655a71439a85b8a932b7603d7d P 00c282f779c1477d920f24dc0dc16628 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:09.227345 1151 raft_consensus.cc:738] T 3eaf8baac9f1485f944cd9e74b8b5569 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:09.229625 1151 consensus_queue.cc:260] T 3eaf8baac9f1485f944cd9e74b8b5569 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:09.233670 1151 ts_tablet_manager.cc:1428] T 3eaf8baac9f1485f944cd9e74b8b5569 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250901 01:52:09.237568 1151 tablet_bootstrap.cc:492] T fab3c1a8928a4066a15868ba708267f1 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:09.238799 1163 ts_tablet_manager.cc:1918] T 0dce1909aa4848c4800910f50a95d1c6 P 0cf599f74322478d91475df234e58965: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 01:52:09.239058 1163 log.cc:1199] T 0dce1909aa4848c4800910f50a95d1c6 P 0cf599f74322478d91475df234e58965: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-0/wal/wals/0dce1909aa4848c4800910f50a95d1c6
I20250901 01:52:09.239321 1151 tablet_bootstrap.cc:654] T fab3c1a8928a4066a15868ba708267f1 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:09.239779 1163 ts_tablet_manager.cc:1939] T 0dce1909aa4848c4800910f50a95d1c6 P 0cf599f74322478d91475df234e58965: Deleting consensus metadata
I20250901 01:52:09.241863 1150 tablet_bootstrap.cc:492] T e8621a0c15a94ed7bd63b035ba2fded7 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:09.242111 1150 ts_tablet_manager.cc:1397] T e8621a0c15a94ed7bd63b035ba2fded7 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.028s user 0.007s sys 0.007s
I20250901 01:52:09.243368 1150 raft_consensus.cc:357] T e8621a0c15a94ed7bd63b035ba2fded7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:09.243669 1150 raft_consensus.cc:383] T e8621a0c15a94ed7bd63b035ba2fded7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:09.243846 1150 raft_consensus.cc:738] T e8621a0c15a94ed7bd63b035ba2fded7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:09.244889 1151 tablet_bootstrap.cc:492] T fab3c1a8928a4066a15868ba708267f1 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:09.245064 1151 ts_tablet_manager.cc:1397] T fab3c1a8928a4066a15868ba708267f1 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.001s
I20250901 01:52:09.246034 1151 raft_consensus.cc:357] T fab3c1a8928a4066a15868ba708267f1 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:09.246274 1151 raft_consensus.cc:383] T fab3c1a8928a4066a15868ba708267f1 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:09.246376 1151 raft_consensus.cc:738] T fab3c1a8928a4066a15868ba708267f1 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:09.246994 1150 consensus_queue.cc:260] T e8621a0c15a94ed7bd63b035ba2fded7 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:09.247767 1158 raft_consensus.cc:695] T b11e8f77243e4764b96548d1132b8e5a P 00c282f779c1477d920f24dc0dc16628 [term 1 LEADER]: Becoming Leader. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Running, Role: LEADER
I20250901 01:52:09.248186 1158 consensus_queue.cc:237] T b11e8f77243e4764b96548d1132b8e5a P 00c282f779c1477d920f24dc0dc16628 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:09.248725 1151 consensus_queue.cc:260] T fab3c1a8928a4066a15868ba708267f1 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:09.253376 440 catalog_manager.cc:4928] TS 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): tablet 0dce1909aa4848c4800910f50a95d1c6 (table default.Table_28fc024b8ecb4bd996601bc5035929c5 [id=22a4aeb52a4747d593d4a316583626b8]) successfully deleted
I20250901 01:52:09.254196 1151 ts_tablet_manager.cc:1428] T fab3c1a8928a4066a15868ba708267f1 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20250901 01:52:09.257818 1151 tablet_bootstrap.cc:492] T e8621a0c15a94ed7bd63b035ba2fded7 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:09.258246 450 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet b11e8f77243e4764b96548d1132b8e5a (table default.Table_16b3ac9fb2304c8095e0270038501db0 [id=a39e8b171a874df1890d072732e55df9]). Expected version 1 got 0
I20250901 01:52:09.258914 1176 raft_consensus.cc:491] T ce2a1ba75d674c95bdba9d6f12273bee P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:09.259074 1176 raft_consensus.cc:513] T ce2a1ba75d674c95bdba9d6f12273bee P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:09.259688 1176 leader_election.cc:290] T ce2a1ba75d674c95bdba9d6f12273bee P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:09.260118 450 master_service.cc:948] ReplaceTablet: received request to replace tablet 7a8a3aca289c4735abeb9c95c414ab76 from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:09.262669 1034 tablet_service.cc:1515] Processing DeleteTablet for tablet c33dea020a5e43d8ad8b52c39cb89a8b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:08 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:09.262970 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet 2dded0466e394e44b0303cefd1b9bc7e with delete_type TABLET_DATA_DELETED (replaced by tablet 7a8a3aca289c4735abeb9c95c414ab76) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:09.263037 1166 tablet_replica.cc:331] T c33dea020a5e43d8ad8b52c39cb89a8b P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:09.263356 1166 raft_consensus.cc:2241] T c33dea020a5e43d8ad8b52c39cb89a8b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:09.263577 1166 raft_consensus.cc:2270] T c33dea020a5e43d8ad8b52c39cb89a8b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:09.262688 1151 tablet_bootstrap.cc:654] T e8621a0c15a94ed7bd63b035ba2fded7 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:09.265192 1166 ts_tablet_manager.cc:1905] T c33dea020a5e43d8ad8b52c39cb89a8b P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:09.265218 781 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ce2a1ba75d674c95bdba9d6f12273bee" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf599f74322478d91475df234e58965" is_pre_election: true
W20250901 01:52:09.266077 992 leader_election.cc:343] T ce2a1ba75d674c95bdba9d6f12273bee P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:09.274466 1166 ts_tablet_manager.cc:1918] T c33dea020a5e43d8ad8b52c39cb89a8b P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:09.274523 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 288e8029382f4154bc1e2103b3035a60
I20250901 01:52:09.274633 1166 log.cc:1199] T c33dea020a5e43d8ad8b52c39cb89a8b P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/c33dea020a5e43d8ad8b52c39cb89a8b
I20250901 01:52:09.274961 1166 ts_tablet_manager.cc:1939] T c33dea020a5e43d8ad8b52c39cb89a8b P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:09.275941 1166 tablet_replica.cc:331] T 2dded0466e394e44b0303cefd1b9bc7e P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:09.276225 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet 288e8029382f4154bc1e2103b3035a60 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:09 UTC) from {username='slave'} at 127.0.0.1:44246
W20250901 01:52:09.276731 449 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b29e846dfae44f0e9ce3e5fea3763a4f has no consensus state
I20250901 01:52:09.276149 1166 raft_consensus.cc:2241] T 2dded0466e394e44b0303cefd1b9bc7e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:09.277402 1166 raft_consensus.cc:2270] T 2dded0466e394e44b0303cefd1b9bc7e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:09.278133 441 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet c33dea020a5e43d8ad8b52c39cb89a8b (table default.Table_906620c5ed1e475a983304809ea3786b [id=155470f2b494476695807c790028f60b]) successfully deleted
I20250901 01:52:09.278704 1166 ts_tablet_manager.cc:1905] T 2dded0466e394e44b0303cefd1b9bc7e P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:09.279808 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_45ba5a130a824accac159d040d1b6a9d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:09.280578 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_45ba5a130a824accac159d040d1b6a9d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:09.283028 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_87bd0229edd44590b0b77f0b9dea7b1f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:09.283746 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_87bd0229edd44590b0b77f0b9dea7b1f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:09.287468 698 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "e3add7edb97a40ef9f7c6bc011d13227" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3add7edb97a40ef9f7c6bc011d13227" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 43871 } } peers { permanent_uuid: "d2ae2cb6af0f4b3a8f47ab1cdd6567d3" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42375 } } peers { permanent_uuid: "7ce2288f784044eea010664e3def133f" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 44231 } } }
I20250901 01:52:09.287731 698 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227 [sys.catalog]: This master's current role is: LEADER
I20250901 01:52:09.294754 1166 ts_tablet_manager.cc:1918] T 2dded0466e394e44b0303cefd1b9bc7e P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:09.295001 1166 log.cc:1199] T 2dded0466e394e44b0303cefd1b9bc7e P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/2dded0466e394e44b0303cefd1b9bc7e
I20250901 01:52:09.295502 1166 ts_tablet_manager.cc:1939] T 2dded0466e394e44b0303cefd1b9bc7e P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:09.297421 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 288e8029382f4154bc1e2103b3035a60 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:09 UTC) from {username='slave'} at 127.0.0.1:36178
I20250901 01:52:09.304527 441 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 2dded0466e394e44b0303cefd1b9bc7e (table default.Table_8da589a069464922b2fc6b875a684173 [id=1d8f34cc14cb4e90a562ff4424b8577a]) successfully deleted
W20250901 01:52:09.305898 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 288e8029382f4154bc1e2103b3035a60 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 288e8029382f4154bc1e2103b3035a60 already in progress: creating tablet
I20250901 01:52:09.260548 924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ce2a1ba75d674c95bdba9d6f12273bee" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:09.307317 924 raft_consensus.cc:2466] T ce2a1ba75d674c95bdba9d6f12273bee P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:09.264039 900 tablet_service.cc:1515] Processing DeleteTablet for tablet 514f64dfb4a64103b60b084c6416171f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:08 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:09.264317 895 tablet_service.cc:1515] Processing DeleteTablet for tablet c31993d35207443288835f90091f73cf with delete_type TABLET_DATA_DELETED (replaced by tablet 4cdd369214ef4f5ebabddbf28b2ae3f7) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:09.307824 1164 tablet_replica.cc:331] T 514f64dfb4a64103b60b084c6416171f P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:09.307888 992 leader_election.cc:304] T ce2a1ba75d674c95bdba9d6f12273bee P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:09.308030 1164 raft_consensus.cc:2241] T 514f64dfb4a64103b60b084c6416171f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:09.308192 1164 raft_consensus.cc:2270] T 514f64dfb4a64103b60b084c6416171f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:09.276360 901 tablet_service.cc:1515] Processing DeleteTablet for tablet 288e8029382f4154bc1e2103b3035a60 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:09 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:09.308301 1176 raft_consensus.cc:2802] T ce2a1ba75d674c95bdba9d6f12273bee P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:09.308425 1176 raft_consensus.cc:491] T ce2a1ba75d674c95bdba9d6f12273bee P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:09.308549 1176 raft_consensus.cc:3058] T ce2a1ba75d674c95bdba9d6f12273bee P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:09.309252 1164 ts_tablet_manager.cc:1905] T 514f64dfb4a64103b60b084c6416171f P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:09.309283 440 catalog_manager.cc:4908] TS 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): delete failed for tablet bf302e76746b4f7bbe4469914f83720f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bf302e76746b4f7bbe4469914f83720f already in progress: creating tablet
W20250901 01:52:09.310596 454 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8d5b10ede9c84be098795c0a787e5a78 consensus state has no leader
I20250901 01:52:09.311237 1176 raft_consensus.cc:513] T ce2a1ba75d674c95bdba9d6f12273bee P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:09.311892 1176 leader_election.cc:290] T ce2a1ba75d674c95bdba9d6f12273bee P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:09.312849 924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ce2a1ba75d674c95bdba9d6f12273bee" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:09.313099 924 raft_consensus.cc:3058] T ce2a1ba75d674c95bdba9d6f12273bee P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:09.316906 924 raft_consensus.cc:2466] T ce2a1ba75d674c95bdba9d6f12273bee P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:09.317435 992 leader_election.cc:304] T ce2a1ba75d674c95bdba9d6f12273bee P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:09.318053 1176 raft_consensus.cc:2802] T ce2a1ba75d674c95bdba9d6f12273bee P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:09.323295 781 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ce2a1ba75d674c95bdba9d6f12273bee" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf599f74322478d91475df234e58965"
W20250901 01:52:09.325577 992 leader_election.cc:343] T ce2a1ba75d674c95bdba9d6f12273bee P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:09.325816 1164 ts_tablet_manager.cc:1918] T 514f64dfb4a64103b60b084c6416171f P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:09.326031 1164 log.cc:1199] T 514f64dfb4a64103b60b084c6416171f P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/514f64dfb4a64103b60b084c6416171f
I20250901 01:52:09.326814 1164 ts_tablet_manager.cc:1939] T 514f64dfb4a64103b60b084c6416171f P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:09.327183 1176 raft_consensus.cc:695] T ce2a1ba75d674c95bdba9d6f12273bee P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:09.328023 1176 consensus_queue.cc:237] T ce2a1ba75d674c95bdba9d6f12273bee P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:09.328240 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 514f64dfb4a64103b60b084c6416171f (table default.Table_05d73d27c8ca4ebf932d4ea423c62e31 [id=8ef09c6e67cb4cce94429d901157f07c]) successfully deleted
I20250901 01:52:09.329272 1164 tablet_replica.cc:331] T c31993d35207443288835f90091f73cf P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:09.329630 1164 raft_consensus.cc:2241] T c31993d35207443288835f90091f73cf P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:09.329892 1164 raft_consensus.cc:2270] T c31993d35207443288835f90091f73cf P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:09.331116 1164 ts_tablet_manager.cc:1905] T c31993d35207443288835f90091f73cf P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:09.334599 451 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 97e40c318f64455d91b0e4f8866990ce consensus state has no leader
I20250901 01:52:09.334897 451 master_service.cc:948] ReplaceTablet: received request to replace tablet e8621a0c15a94ed7bd63b035ba2fded7 from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:09.340502 1164 ts_tablet_manager.cc:1918] T c31993d35207443288835f90091f73cf P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:09.342651 455 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 7f2546a3f6da4b3297ebcbf006eeb3f2
I20250901 01:52:09.347100 1164 log.cc:1199] T c31993d35207443288835f90091f73cf P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/c31993d35207443288835f90091f73cf
I20250901 01:52:09.349823 1164 ts_tablet_manager.cc:1939] T c31993d35207443288835f90091f73cf P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:09.351624 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet c31993d35207443288835f90091f73cf (table default.Table_cf1c9f5fc5104126b163225918707284 [id=00ae7e46c65d4a56b1ffa193b853421a]) successfully deleted
W20250901 01:52:09.352072 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 288e8029382f4154bc1e2103b3035a60 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 288e8029382f4154bc1e2103b3035a60 already in progress: creating tablet
W20250901 01:52:09.358480 440 catalog_manager.cc:4908] TS 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): delete failed for tablet 4cdd369214ef4f5ebabddbf28b2ae3f7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4cdd369214ef4f5ebabddbf28b2ae3f7 already in progress: creating tablet
W20250901 01:52:09.358950 440 catalog_manager.cc:4895] TS 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): delete failed for tablet 4d333b670df84f308f0cbf6da636b768 because the tablet was not found. No further retry: Not found: Tablet not found: 4d333b670df84f308f0cbf6da636b768
W20250901 01:52:09.359864 440 catalog_manager.cc:4908] TS 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): delete failed for tablet 288e8029382f4154bc1e2103b3035a60 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 288e8029382f4154bc1e2103b3035a60 already in progress: creating tablet
I20250901 01:52:09.367113 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_dcacb44eed4241d3acbae153ebaad663"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:09.368006 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dcacb44eed4241d3acbae153ebaad663 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:09.369570 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_64b711e409d7434c8b176bf1a3ef8c86"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:09.370150 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_64b711e409d7434c8b176bf1a3ef8c86 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:09.371904 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_914637ff818441a6bf62eb9d217404b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:09.372534 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_914637ff818441a6bf62eb9d217404b7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:09.374045 446 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_72d74def1ee74948a379b9d1b6e51781" } new_table_name: "default.Table_b7bffcbb78614d3cad2b1d375d7a97cc" modify_external_catalogs: true
I20250901 01:52:09.405035 1150 ts_tablet_manager.cc:1428] T e8621a0c15a94ed7bd63b035ba2fded7 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.163s user 0.019s sys 0.054s
I20250901 01:52:09.405702 1150 tablet_bootstrap.cc:492] T 288e8029382f4154bc1e2103b3035a60 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:09.406760 447 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 4f8e3fcaa19740d397f4d118ac4cbdb0
I20250901 01:52:09.407423 447 catalog_manager.cc:6652] ReplaceTablet: tablet 4f8e3fcaa19740d397f4d118ac4cbdb0 deleted and replaced by tablet 88a6ab29d00542aba54f0e4185fbf615
I20250901 01:52:09.407716 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet 4f8e3fcaa19740d397f4d118ac4cbdb0 with delete_type TABLET_DATA_DELETED (replaced by tablet 88a6ab29d00542aba54f0e4185fbf615) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:09.408193 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 4f8e3fcaa19740d397f4d118ac4cbdb0 with delete_type TABLET_DATA_DELETED (replaced by tablet 88a6ab29d00542aba54f0e4185fbf615) from {username='slave'} at 127.0.0.1:36178
W20250901 01:52:09.408497 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 4f8e3fcaa19740d397f4d118ac4cbdb0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f8e3fcaa19740d397f4d118ac4cbdb0 already in progress: creating tablet
I20250901 01:52:09.408630 901 tablet_service.cc:1515] Processing DeleteTablet for tablet 4f8e3fcaa19740d397f4d118ac4cbdb0 with delete_type TABLET_DATA_DELETED (replaced by tablet 88a6ab29d00542aba54f0e4185fbf615) from {username='slave'} at 127.0.0.1:52846
W20250901 01:52:09.408975 440 catalog_manager.cc:4908] TS 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): delete failed for tablet 4f8e3fcaa19740d397f4d118ac4cbdb0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f8e3fcaa19740d397f4d118ac4cbdb0 already in progress: creating tablet
W20250901 01:52:09.409696 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 4f8e3fcaa19740d397f4d118ac4cbdb0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f8e3fcaa19740d397f4d118ac4cbdb0 already in progress: creating tablet
I20250901 01:52:09.410480 1150 tablet_bootstrap.cc:654] T 288e8029382f4154bc1e2103b3035a60 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:09.411620 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 7a8a3aca289c4735abeb9c95c414ab76
I20250901 01:52:09.412209 450 catalog_manager.cc:6652] ReplaceTablet: tablet 7a8a3aca289c4735abeb9c95c414ab76 deleted and replaced by tablet 85322a864d154f299f2d670fdd2a7bfb
I20250901 01:52:09.412650 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet 7a8a3aca289c4735abeb9c95c414ab76 with delete_type TABLET_DATA_DELETED (replaced by tablet 85322a864d154f299f2d670fdd2a7bfb) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:09.413139 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_b3ec3b285fac447f8c5c4a9ac4196628"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:09.413434 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 7a8a3aca289c4735abeb9c95c414ab76 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7a8a3aca289c4735abeb9c95c414ab76 already in progress: creating tablet
I20250901 01:52:09.413839 901 tablet_service.cc:1515] Processing DeleteTablet for tablet 7a8a3aca289c4735abeb9c95c414ab76 with delete_type TABLET_DATA_DELETED (replaced by tablet 85322a864d154f299f2d670fdd2a7bfb) from {username='slave'} at 127.0.0.1:52846
W20250901 01:52:09.413964 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b3ec3b285fac447f8c5c4a9ac4196628 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250901 01:52:09.414644 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 7a8a3aca289c4735abeb9c95c414ab76 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7a8a3aca289c4735abeb9c95c414ab76 already in progress: creating tablet
I20250901 01:52:09.414932 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 7a8a3aca289c4735abeb9c95c414ab76 with delete_type TABLET_DATA_DELETED (replaced by tablet 85322a864d154f299f2d670fdd2a7bfb) from {username='slave'} at 127.0.0.1:36178
W20250901 01:52:09.415552 440 catalog_manager.cc:4908] TS 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): delete failed for tablet 7a8a3aca289c4735abeb9c95c414ab76 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7a8a3aca289c4735abeb9c95c414ab76 already in progress: creating tablet
I20250901 01:52:09.429813 1150 tablet_bootstrap.cc:492] T 288e8029382f4154bc1e2103b3035a60 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:09.430060 1150 ts_tablet_manager.cc:1397] T 288e8029382f4154bc1e2103b3035a60 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.003s
I20250901 01:52:09.430091 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_6400e401888045348632f924f636d011"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:09.430913 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6400e401888045348632f924f636d011 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:09.431097 1150 raft_consensus.cc:357] T 288e8029382f4154bc1e2103b3035a60 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:09.431370 1150 raft_consensus.cc:383] T 288e8029382f4154bc1e2103b3035a60 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:09.431434 1150 raft_consensus.cc:738] T 288e8029382f4154bc1e2103b3035a60 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:09.433542 1151 tablet_bootstrap.cc:492] T e8621a0c15a94ed7bd63b035ba2fded7 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:09.433779 1151 ts_tablet_manager.cc:1397] T e8621a0c15a94ed7bd63b035ba2fded7 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.176s user 0.014s sys 0.031s
I20250901 01:52:09.435194 1151 raft_consensus.cc:357] T e8621a0c15a94ed7bd63b035ba2fded7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:09.435266 1150 consensus_queue.cc:260] T 288e8029382f4154bc1e2103b3035a60 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:09.435485 1151 raft_consensus.cc:383] T e8621a0c15a94ed7bd63b035ba2fded7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:09.435581 1151 raft_consensus.cc:738] T e8621a0c15a94ed7bd63b035ba2fded7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:09.435918 1151 consensus_queue.cc:260] T e8621a0c15a94ed7bd63b035ba2fded7 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:09.436595 1150 ts_tablet_manager.cc:1428] T 288e8029382f4154bc1e2103b3035a60 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.006s user 0.002s sys 0.002s
I20250901 01:52:09.437049 1150 tablet_bootstrap.cc:492] T fab3c1a8928a4066a15868ba708267f1 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:09.437476 1151 ts_tablet_manager.cc:1428] T e8621a0c15a94ed7bd63b035ba2fded7 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.004s user 0.002s sys 0.002s
I20250901 01:52:09.437824 1151 tablet_bootstrap.cc:492] T c31993d35207443288835f90091f73cf P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:09.439471 1151 tablet_bootstrap.cc:654] T c31993d35207443288835f90091f73cf P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:09.439456 1150 tablet_bootstrap.cc:654] T fab3c1a8928a4066a15868ba708267f1 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:09.451632 449 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_1a83c4b1e2734d038d0309401d0ab08e" } modify_external_catalogs: true
I20250901 01:52:09.451931 449 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_1a83c4b1e2734d038d0309401d0ab08e" } modify_external_catalogs: true
I20250901 01:52:09.458838 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet e8621a0c15a94ed7bd63b035ba2fded7
I20250901 01:52:09.459568 451 catalog_manager.cc:6652] ReplaceTablet: tablet e8621a0c15a94ed7bd63b035ba2fded7 deleted and replaced by tablet 31390f38258c465cb1798615a1fcc790
I20250901 01:52:09.460247 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet e8621a0c15a94ed7bd63b035ba2fded7 with delete_type TABLET_DATA_DELETED (replaced by tablet 31390f38258c465cb1798615a1fcc790) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:09.460711 1166 tablet_replica.cc:331] T e8621a0c15a94ed7bd63b035ba2fded7 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:09.461037 1166 raft_consensus.cc:2241] T e8621a0c15a94ed7bd63b035ba2fded7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:09.461282 1166 raft_consensus.cc:2270] T e8621a0c15a94ed7bd63b035ba2fded7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:09.462204 1166 ts_tablet_manager.cc:1905] T e8621a0c15a94ed7bd63b035ba2fded7 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:09.465452 901 tablet_service.cc:1515] Processing DeleteTablet for tablet e8621a0c15a94ed7bd63b035ba2fded7 with delete_type TABLET_DATA_DELETED (replaced by tablet 31390f38258c465cb1798615a1fcc790) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:09.465804 1164 tablet_replica.cc:331] T e8621a0c15a94ed7bd63b035ba2fded7 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:09.466017 1164 raft_consensus.cc:2241] T e8621a0c15a94ed7bd63b035ba2fded7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:09.466176 1164 raft_consensus.cc:2270] T e8621a0c15a94ed7bd63b035ba2fded7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:09.466993 1164 ts_tablet_manager.cc:1905] T e8621a0c15a94ed7bd63b035ba2fded7 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:09.470088 1166 ts_tablet_manager.cc:1918] T e8621a0c15a94ed7bd63b035ba2fded7 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:09.470247 1166 log.cc:1199] T e8621a0c15a94ed7bd63b035ba2fded7 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/e8621a0c15a94ed7bd63b035ba2fded7
I20250901 01:52:09.470641 1166 ts_tablet_manager.cc:1939] T e8621a0c15a94ed7bd63b035ba2fded7 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:09.471431 758 tablet_service.cc:1515] Processing DeleteTablet for tablet e8621a0c15a94ed7bd63b035ba2fded7 with delete_type TABLET_DATA_DELETED (replaced by tablet 31390f38258c465cb1798615a1fcc790) from {username='slave'} at 127.0.0.1:36178
I20250901 01:52:09.471745 441 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet e8621a0c15a94ed7bd63b035ba2fded7 (table default.Table_1a83c4b1e2734d038d0309401d0ab08e [id=c1902e6ac21b4a8a8af96d647f85e288]) successfully deleted
W20250901 01:52:09.472520 440 catalog_manager.cc:4908] TS 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): delete failed for tablet e8621a0c15a94ed7bd63b035ba2fded7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e8621a0c15a94ed7bd63b035ba2fded7 already in progress: creating tablet
I20250901 01:52:09.472646 451 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_196019dc3a1d4cbea8d5a87f139c899d" } modify_external_catalogs: true
I20250901 01:52:09.472921 451 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_196019dc3a1d4cbea8d5a87f139c899d" } modify_external_catalogs: true
I20250901 01:52:09.474608 1164 ts_tablet_manager.cc:1918] T e8621a0c15a94ed7bd63b035ba2fded7 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:09.474756 1164 log.cc:1199] T e8621a0c15a94ed7bd63b035ba2fded7 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/e8621a0c15a94ed7bd63b035ba2fded7
I20250901 01:52:09.474853 1150 tablet_bootstrap.cc:492] T fab3c1a8928a4066a15868ba708267f1 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:09.475013 1150 ts_tablet_manager.cc:1397] T fab3c1a8928a4066a15868ba708267f1 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.038s user 0.004s sys 0.000s
I20250901 01:52:09.475099 1164 ts_tablet_manager.cc:1939] T e8621a0c15a94ed7bd63b035ba2fded7 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:09.475904 1150 raft_consensus.cc:357] T fab3c1a8928a4066a15868ba708267f1 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:09.476079 1150 raft_consensus.cc:383] T fab3c1a8928a4066a15868ba708267f1 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:09.476179 1150 raft_consensus.cc:738] T fab3c1a8928a4066a15868ba708267f1 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:09.476455 1150 consensus_queue.cc:260] T fab3c1a8928a4066a15868ba708267f1 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:09.476863 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet e8621a0c15a94ed7bd63b035ba2fded7 (table default.Table_1a83c4b1e2734d038d0309401d0ab08e [id=c1902e6ac21b4a8a8af96d647f85e288]) successfully deleted
I20250901 01:52:09.482693 1150 ts_tablet_manager.cc:1428] T fab3c1a8928a4066a15868ba708267f1 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250901 01:52:09.483022 1150 tablet_bootstrap.cc:492] T 061b760d8e6647c6ab0b5d538840a230 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:09.483196 455 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_1e00e9f110c7483bb1a077829c64f85b" } modify_external_catalogs: true
I20250901 01:52:09.483419 455 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_1e00e9f110c7483bb1a077829c64f85b" } modify_external_catalogs: true
I20250901 01:52:09.485204 1150 tablet_bootstrap.cc:654] T 061b760d8e6647c6ab0b5d538840a230 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:09.498389 453 master_service.cc:948] ReplaceTablet: received request to replace tablet 8e6e96b4de214cadba1ccd19390cffb0 from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:09.502735 1150 tablet_bootstrap.cc:492] T 061b760d8e6647c6ab0b5d538840a230 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:09.503032 1150 ts_tablet_manager.cc:1397] T 061b760d8e6647c6ab0b5d538840a230 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.000s
I20250901 01:52:09.506352 1150 raft_consensus.cc:357] T 061b760d8e6647c6ab0b5d538840a230 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:09.506742 1150 raft_consensus.cc:383] T 061b760d8e6647c6ab0b5d538840a230 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:09.506861 1150 raft_consensus.cc:738] T 061b760d8e6647c6ab0b5d538840a230 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:09.508296 1150 consensus_queue.cc:260] T 061b760d8e6647c6ab0b5d538840a230 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:09.511933 452 master_service.cc:948] ReplaceTablet: received request to replace tablet 81fa34655a71439a85b8a932b7603d7d from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:09.512813 1150 ts_tablet_manager.cc:1428] T 061b760d8e6647c6ab0b5d538840a230 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.010s user 0.002s sys 0.002s
I20250901 01:52:09.513283 1150 tablet_bootstrap.cc:492] T bf302e76746b4f7bbe4469914f83720f P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
W20250901 01:52:09.521446 446 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f0affc7a78634607852a48458412761d consensus state has no leader
I20250901 01:52:09.524820 1150 tablet_bootstrap.cc:654] T bf302e76746b4f7bbe4469914f83720f P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:09.526180 446 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_aadaeb5310a74963a0c558f887b018f4" } new_table_name: "default.Table_aa569ededbee4b1bb3c3a2f20f9748cb" modify_external_catalogs: true
I20250901 01:52:09.529776 1150 tablet_bootstrap.cc:492] T bf302e76746b4f7bbe4469914f83720f P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:09.530011 1150 ts_tablet_manager.cc:1397] T bf302e76746b4f7bbe4469914f83720f P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.002s
I20250901 01:52:09.530814 1151 tablet_bootstrap.cc:492] T c31993d35207443288835f90091f73cf P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:09.530978 1151 ts_tablet_manager.cc:1397] T c31993d35207443288835f90091f73cf P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.093s user 0.003s sys 0.000s
I20250901 01:52:09.531172 1150 raft_consensus.cc:357] T bf302e76746b4f7bbe4469914f83720f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:09.531445 1150 raft_consensus.cc:383] T bf302e76746b4f7bbe4469914f83720f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:09.531572 1150 raft_consensus.cc:738] T bf302e76746b4f7bbe4469914f83720f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:09.531814 1151 raft_consensus.cc:357] T c31993d35207443288835f90091f73cf P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:09.532008 1151 raft_consensus.cc:383] T c31993d35207443288835f90091f73cf P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:09.531925 1150 consensus_queue.cc:260] T bf302e76746b4f7bbe4469914f83720f P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:09.532100 1151 raft_consensus.cc:738] T c31993d35207443288835f90091f73cf P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:09.532357 1151 consensus_queue.cc:260] T c31993d35207443288835f90091f73cf P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:09.537202 1150 ts_tablet_manager.cc:1428] T bf302e76746b4f7bbe4469914f83720f P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.007s user 0.001s sys 0.002s
I20250901 01:52:09.537644 1150 tablet_bootstrap.cc:492] T 116f65a42ab34bf6b8f8113cb7afdc21 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:09.539059 450 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_ac8663b3415c4c668e298a84b2deb8c3" } modify_external_catalogs: true
I20250901 01:52:09.539260 450 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_ac8663b3415c4c668e298a84b2deb8c3" } modify_external_catalogs: true
I20250901 01:52:09.540015 1150 tablet_bootstrap.cc:654] T 116f65a42ab34bf6b8f8113cb7afdc21 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:09.541514 1151 ts_tablet_manager.cc:1428] T c31993d35207443288835f90091f73cf P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20250901 01:52:09.543758 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskddMNdm/build/asan/bin/kudu
/tmp/dist-test-taskddMNdm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:44231
--webserver_interface=127.0.104.60
--webserver_port=46373
--builtin_ntp_servers=127.0.104.20:46333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:43871,127.0.104.61:42375,127.0.104.60:44231
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250901 01:52:09.544718 1151 tablet_bootstrap.cc:492] T 8f2e702121c845e0a09a254bbb81c142 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:09.546847 1151 tablet_bootstrap.cc:654] T 8f2e702121c845e0a09a254bbb81c142 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:09.558450 448 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_2447596ecd4242ceae8590ca488e4616" } new_table_name: "default.Table_6a5316d194864096bdfdf11e382f69d2" modify_external_catalogs: true
I20250901 01:52:09.563520 1150 tablet_bootstrap.cc:492] T 116f65a42ab34bf6b8f8113cb7afdc21 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:09.563829 1150 ts_tablet_manager.cc:1397] T 116f65a42ab34bf6b8f8113cb7afdc21 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.002s
I20250901 01:52:09.566130 1241 raft_consensus.cc:491] T fab3c1a8928a4066a15868ba708267f1 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:09.566298 1241 raft_consensus.cc:513] T fab3c1a8928a4066a15868ba708267f1 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:09.567180 1241 leader_election.cc:290] T fab3c1a8928a4066a15868ba708267f1 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:09.567863 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fab3c1a8928a4066a15868ba708267f1" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:09.568104 1057 raft_consensus.cc:2466] T fab3c1a8928a4066a15868ba708267f1 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00c282f779c1477d920f24dc0dc16628 in term 0.
I20250901 01:52:09.568799 859 leader_election.cc:304] T fab3c1a8928a4066a15868ba708267f1 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:09.571434 1241 raft_consensus.cc:2802] T fab3c1a8928a4066a15868ba708267f1 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:09.571619 1241 raft_consensus.cc:491] T fab3c1a8928a4066a15868ba708267f1 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:09.571748 1241 raft_consensus.cc:3058] T fab3c1a8928a4066a15868ba708267f1 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:09.573032 454 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 3eaf8baac9f1485f944cd9e74b8b5569 (table default.Table_947ec0e1a97a40b59be1e234bcaf4e0d [id=5d14706774d345c48ca3575feabb17b0]). Expected version 1 got 0
I20250901 01:52:09.575904 1241 raft_consensus.cc:513] T fab3c1a8928a4066a15868ba708267f1 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:09.577098 1241 leader_election.cc:290] T fab3c1a8928a4066a15868ba708267f1 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:09.577641 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fab3c1a8928a4066a15868ba708267f1" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58"
I20250901 01:52:09.577845 1057 raft_consensus.cc:3058] T fab3c1a8928a4066a15868ba708267f1 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:09.577732 1150 raft_consensus.cc:357] T 116f65a42ab34bf6b8f8113cb7afdc21 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:09.578001 1150 raft_consensus.cc:383] T 116f65a42ab34bf6b8f8113cb7afdc21 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:09.578102 1150 raft_consensus.cc:738] T 116f65a42ab34bf6b8f8113cb7afdc21 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:09.578411 1150 consensus_queue.cc:260] T 116f65a42ab34bf6b8f8113cb7afdc21 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:09.580747 755 tablet_service.cc:1468] Processing CreateTablet for tablet 4fd0e4f9d86c4362af96ce9964b72328 (DEFAULT_TABLE table=default.Table_7a731ba38b4048009b421d8da7f58089 [id=f48fd53522234290b6c0373fd7b02570]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:09.581187 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fd0e4f9d86c4362af96ce9964b72328. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:09.581167 758 tablet_service.cc:1468] Processing CreateTablet for tablet 467c2bf4a8cb4937b6f627bea28d8ddb (DEFAULT_TABLE table=default.Table_ac3b76f15c7743d0900e9a7a00348ec2 [id=f3735a7cf9634c11abc2c8121e2ed930]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:09.581571 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 467c2bf4a8cb4937b6f627bea28d8ddb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:09.581724 1150 ts_tablet_manager.cc:1428] T 116f65a42ab34bf6b8f8113cb7afdc21 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.018s user 0.003s sys 0.000s
I20250901 01:52:09.582043 1150 tablet_bootstrap.cc:492] T c297e8bf11ff467a854eab41117ac376 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:09.583812 1150 tablet_bootstrap.cc:654] T c297e8bf11ff467a854eab41117ac376 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:09.584733 750 tablet_service.cc:1468] Processing CreateTablet for tablet b29e846dfae44f0e9ce3e5fea3763a4f (DEFAULT_TABLE table=default.Table_1e00e9f110c7483bb1a077829c64f85b [id=4ef5558babf64de7930fadf065ef6866]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:09.585139 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b29e846dfae44f0e9ce3e5fea3763a4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:09.585294 1057 raft_consensus.cc:2466] T fab3c1a8928a4066a15868ba708267f1 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c282f779c1477d920f24dc0dc16628 in term 1.
I20250901 01:52:09.589025 901 tablet_service.cc:1468] Processing CreateTablet for tablet 467c2bf4a8cb4937b6f627bea28d8ddb (DEFAULT_TABLE table=default.Table_ac3b76f15c7743d0900e9a7a00348ec2 [id=f3735a7cf9634c11abc2c8121e2ed930]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:09.589754 901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 467c2bf4a8cb4937b6f627bea28d8ddb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:09.590226 1031 tablet_service.cc:1468] Processing CreateTablet for tablet b29e846dfae44f0e9ce3e5fea3763a4f (DEFAULT_TABLE table=default.Table_1e00e9f110c7483bb1a077829c64f85b [id=4ef5558babf64de7930fadf065ef6866]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:09.590343 859 leader_election.cc:304] T fab3c1a8928a4066a15868ba708267f1 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:09.590480 1032 tablet_service.cc:1468] Processing CreateTablet for tablet 467c2bf4a8cb4937b6f627bea28d8ddb (DEFAULT_TABLE table=default.Table_ac3b76f15c7743d0900e9a7a00348ec2 [id=f3735a7cf9634c11abc2c8121e2ed930]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:09.590658 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b29e846dfae44f0e9ce3e5fea3763a4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:09.590987 900 tablet_service.cc:1468] Processing CreateTablet for tablet b29e846dfae44f0e9ce3e5fea3763a4f (DEFAULT_TABLE table=default.Table_1e00e9f110c7483bb1a077829c64f85b [id=4ef5558babf64de7930fadf065ef6866]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:09.591007 1034 tablet_service.cc:1468] Processing CreateTablet for tablet 4fd0e4f9d86c4362af96ce9964b72328 (DEFAULT_TABLE table=default.Table_7a731ba38b4048009b421d8da7f58089 [id=f48fd53522234290b6c0373fd7b02570]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:09.591392 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fd0e4f9d86c4362af96ce9964b72328. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:09.591418 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b29e846dfae44f0e9ce3e5fea3763a4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:09.590696 895 tablet_service.cc:1468] Processing CreateTablet for tablet 4fd0e4f9d86c4362af96ce9964b72328 (DEFAULT_TABLE table=default.Table_7a731ba38b4048009b421d8da7f58089 [id=f48fd53522234290b6c0373fd7b02570]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:09.593628 1241 raft_consensus.cc:2802] T fab3c1a8928a4066a15868ba708267f1 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:09.593815 1241 raft_consensus.cc:695] T fab3c1a8928a4066a15868ba708267f1 P 00c282f779c1477d920f24dc0dc16628 [term 1 LEADER]: Becoming Leader. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Running, Role: LEADER
I20250901 01:52:09.594484 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet fab3c1a8928a4066a15868ba708267f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:08 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:09.594607 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fd0e4f9d86c4362af96ce9964b72328. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:09.594756 1038 tablet_service.cc:1515] Processing DeleteTablet for tablet 514f64dfb4a64103b60b084c6416171f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:08 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:09.594998 1166 tablet_replica.cc:331] T fab3c1a8928a4066a15868ba708267f1 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:09.595191 1166 raft_consensus.cc:2241] T fab3c1a8928a4066a15868ba708267f1 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:09.595371 1166 raft_consensus.cc:2270] T fab3c1a8928a4066a15868ba708267f1 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:09.597224 1241 consensus_queue.cc:237] T fab3c1a8928a4066a15868ba708267f1 P 00c282f779c1477d920f24dc0dc16628 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:09.597677 1166 ts_tablet_manager.cc:1905] T fab3c1a8928a4066a15868ba708267f1 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:09.598968 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 467c2bf4a8cb4937b6f627bea28d8ddb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:09.606750 1166 ts_tablet_manager.cc:1918] T fab3c1a8928a4066a15868ba708267f1 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:09.607795 1166 log.cc:1199] T fab3c1a8928a4066a15868ba708267f1 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/fab3c1a8928a4066a15868ba708267f1
I20250901 01:52:09.608302 1166 ts_tablet_manager.cc:1939] T fab3c1a8928a4066a15868ba708267f1 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:09.609517 1166 tablet_replica.cc:331] T 514f64dfb4a64103b60b084c6416171f P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:09.609799 1166 raft_consensus.cc:2241] T 514f64dfb4a64103b60b084c6416171f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:09.609728 755 tablet_service.cc:1468] Processing CreateTablet for tablet 2a0d125423a043dcb11e67345599e468 (DEFAULT_TABLE table=default.Table_b27678f366d74143b716da42e7ca32c3 [id=500406ff1eb342f6af09fbb6f818e160]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:09.609766 753 tablet_service.cc:1468] Processing CreateTablet for tablet d945b310fe6344af9cc9dd68f84c4bdf (DEFAULT_TABLE table=default.Table_e2ea7aa90231448888c6cfead7e3d9c1 [id=c264ed0bf1d84bc9bd5c2bb0d1e8a45b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:09.610085 1166 raft_consensus.cc:2270] T 514f64dfb4a64103b60b084c6416171f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:09.610158 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a0d125423a043dcb11e67345599e468. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:09.610745 903 tablet_service.cc:1468] Processing CreateTablet for tablet 2a0d125423a043dcb11e67345599e468 (DEFAULT_TABLE table=default.Table_b27678f366d74143b716da42e7ca32c3 [id=500406ff1eb342f6af09fbb6f818e160]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:09.611126 903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a0d125423a043dcb11e67345599e468. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:09.611451 1038 tablet_service.cc:1468] Processing CreateTablet for tablet d945b310fe6344af9cc9dd68f84c4bdf (DEFAULT_TABLE table=default.Table_e2ea7aa90231448888c6cfead7e3d9c1 [id=c264ed0bf1d84bc9bd5c2bb0d1e8a45b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:09.611805 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d945b310fe6344af9cc9dd68f84c4bdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:09.611987 1031 tablet_service.cc:1468] Processing CreateTablet for tablet 2a0d125423a043dcb11e67345599e468 (DEFAULT_TABLE table=default.Table_b27678f366d74143b716da42e7ca32c3 [id=500406ff1eb342f6af09fbb6f818e160]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:09.612352 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a0d125423a043dcb11e67345599e468. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:09.614930 902 tablet_service.cc:1468] Processing CreateTablet for tablet d945b310fe6344af9cc9dd68f84c4bdf (DEFAULT_TABLE table=default.Table_e2ea7aa90231448888c6cfead7e3d9c1 [id=c264ed0bf1d84bc9bd5c2bb0d1e8a45b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:09.615306 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d945b310fe6344af9cc9dd68f84c4bdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:09.616304 1166 ts_tablet_manager.cc:1905] T 514f64dfb4a64103b60b084c6416171f P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:09.616469 441 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet fab3c1a8928a4066a15868ba708267f1 (table default.Table_2a6cc96cb6b44ff59a573b691d22c7da [id=dd30bd2e6bee462894768b9341db0ff2]) successfully deleted
I20250901 01:52:09.616979 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d945b310fe6344af9cc9dd68f84c4bdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:09.637095 1150 tablet_bootstrap.cc:492] T c297e8bf11ff467a854eab41117ac376 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:09.637264 1150 ts_tablet_manager.cc:1397] T c297e8bf11ff467a854eab41117ac376 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.055s user 0.003s sys 0.003s
I20250901 01:52:09.637678 454 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_8da589a069464922b2fc6b875a684173" } new_table_name: "default.Table_a8b4fe01e7c54c3e95663bced8b3bbe4" modify_external_catalogs: true
I20250901 01:52:09.638091 1150 raft_consensus.cc:357] T c297e8bf11ff467a854eab41117ac376 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:09.638275 1150 raft_consensus.cc:383] T c297e8bf11ff467a854eab41117ac376 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:09.638365 1150 raft_consensus.cc:738] T c297e8bf11ff467a854eab41117ac376 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:09.642937 1166 ts_tablet_manager.cc:1918] T 514f64dfb4a64103b60b084c6416171f P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:09.643105 1166 log.cc:1199] T 514f64dfb4a64103b60b084c6416171f P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/514f64dfb4a64103b60b084c6416171f
I20250901 01:52:09.643440 1166 ts_tablet_manager.cc:1939] T 514f64dfb4a64103b60b084c6416171f P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:09.644627 441 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 514f64dfb4a64103b60b084c6416171f (table default.Table_05d73d27c8ca4ebf932d4ea423c62e31 [id=8ef09c6e67cb4cce94429d901157f07c]) successfully deleted
I20250901 01:52:09.645470 1150 consensus_queue.cc:260] T c297e8bf11ff467a854eab41117ac376 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:09.654424 1150 ts_tablet_manager.cc:1428] T c297e8bf11ff467a854eab41117ac376 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
W20250901 01:52:09.659519 446 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 88a6ab29d00542aba54f0e4185fbf615 has no consensus state
I20250901 01:52:09.660341 1150 tablet_bootstrap.cc:492] T 77405097e68344628f4eda667afaed21 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:09.663283 1150 tablet_bootstrap.cc:654] T 77405097e68344628f4eda667afaed21 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:09.664793 446 catalog_manager.cc:5582] T ce2a1ba75d674c95bdba9d6f12273bee P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } }
W20250901 01:52:09.681512 448 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 84fe314502444bd88b038b87eead9cb5 has no consensus state
I20250901 01:52:09.681895 1150 tablet_bootstrap.cc:492] T 77405097e68344628f4eda667afaed21 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:09.682147 1150 ts_tablet_manager.cc:1397] T 77405097e68344628f4eda667afaed21 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.022s user 0.000s sys 0.010s
I20250901 01:52:09.683346 1150 raft_consensus.cc:357] T 77405097e68344628f4eda667afaed21 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:09.683676 1150 raft_consensus.cc:383] T 77405097e68344628f4eda667afaed21 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:09.683770 1150 raft_consensus.cc:738] T 77405097e68344628f4eda667afaed21 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:09.687463 1150 consensus_queue.cc:260] T 77405097e68344628f4eda667afaed21 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:09.688246 448 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_914637ff818441a6bf62eb9d217404b7" } new_table_name: "default.Table_dd24f0428569419fad1bbe8a7f9f08c1" modify_external_catalogs: true
I20250901 01:52:09.690698 1150 ts_tablet_manager.cc:1428] T 77405097e68344628f4eda667afaed21 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.008s user 0.002s sys 0.001s
I20250901 01:52:09.702212 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 447ffe73741640ceb73f35676232a094
I20250901 01:52:09.704440 902 tablet_service.cc:1515] Processing DeleteTablet for tablet 447ffe73741640ceb73f35676232a094 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:09 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:09.708424 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet 447ffe73741640ceb73f35676232a094 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:09 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:09.708856 758 tablet_service.cc:1515] Processing DeleteTablet for tablet 447ffe73741640ceb73f35676232a094 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:09 UTC) from {username='slave'} at 127.0.0.1:36178
W20250901 01:52:09.709231 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 447ffe73741640ceb73f35676232a094 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 447ffe73741640ceb73f35676232a094 already in progress: creating tablet
W20250901 01:52:09.709939 440 catalog_manager.cc:4908] TS 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): delete failed for tablet 447ffe73741640ceb73f35676232a094 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 447ffe73741640ceb73f35676232a094 already in progress: creating tablet
I20250901 01:52:09.709864 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_457273ab524846e6954d3e60f698e178"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:09.710659 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_457273ab524846e6954d3e60f698e178 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:09.719795 1150 tablet_bootstrap.cc:492] T c445c300d71643609071f0f36528a9c4 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
W20250901 01:52:09.721459 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 447ffe73741640ceb73f35676232a094 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 447ffe73741640ceb73f35676232a094 already in progress: creating tablet
I20250901 01:52:09.722420 1241 raft_consensus.cc:491] T 061b760d8e6647c6ab0b5d538840a230 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:09.722695 1241 raft_consensus.cc:513] T 061b760d8e6647c6ab0b5d538840a230 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:09.723603 1241 leader_election.cc:290] T 061b760d8e6647c6ab0b5d538840a230 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:09.724496 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "061b760d8e6647c6ab0b5d538840a230" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:09.725281 859 leader_election.cc:343] T 061b760d8e6647c6ab0b5d538840a230 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:09.728075 1150 tablet_bootstrap.cc:654] T c445c300d71643609071f0f36528a9c4 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:09.729487 1151 tablet_bootstrap.cc:492] T 8f2e702121c845e0a09a254bbb81c142 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:09.729702 1151 ts_tablet_manager.cc:1397] T 8f2e702121c845e0a09a254bbb81c142 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.185s user 0.014s sys 0.047s
I20250901 01:52:09.730471 1151 raft_consensus.cc:357] T 8f2e702121c845e0a09a254bbb81c142 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:09.730698 1151 raft_consensus.cc:383] T 8f2e702121c845e0a09a254bbb81c142 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:09.730787 1151 raft_consensus.cc:738] T 8f2e702121c845e0a09a254bbb81c142 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:09.732836 1151 consensus_queue.cc:260] T 8f2e702121c845e0a09a254bbb81c142 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:09.736974 1151 ts_tablet_manager.cc:1428] T 8f2e702121c845e0a09a254bbb81c142 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250901 01:52:09.740756 1151 tablet_bootstrap.cc:492] T 288e8029382f4154bc1e2103b3035a60 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:09.742697 1151 tablet_bootstrap.cc:654] T 288e8029382f4154bc1e2103b3035a60 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:09.745795 1150 tablet_bootstrap.cc:492] T c445c300d71643609071f0f36528a9c4 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:09.746104 1150 ts_tablet_manager.cc:1397] T c445c300d71643609071f0f36528a9c4 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.026s user 0.007s sys 0.006s
I20250901 01:52:09.747285 1150 raft_consensus.cc:357] T c445c300d71643609071f0f36528a9c4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:09.747633 1150 raft_consensus.cc:383] T c445c300d71643609071f0f36528a9c4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:09.747834 1150 raft_consensus.cc:738] T c445c300d71643609071f0f36528a9c4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:09.748296 1150 consensus_queue.cc:260] T c445c300d71643609071f0f36528a9c4 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:09.749500 453 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 8e6e96b4de214cadba1ccd19390cffb0
I20250901 01:52:09.750066 453 catalog_manager.cc:6652] ReplaceTablet: tablet 8e6e96b4de214cadba1ccd19390cffb0 deleted and replaced by tablet 3b18a7084727490380d26b3485e4e799
I20250901 01:52:09.750573 1176 raft_consensus.cc:491] T 3eaf8baac9f1485f944cd9e74b8b5569 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:09.750749 1176 raft_consensus.cc:513] T 3eaf8baac9f1485f944cd9e74b8b5569 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:09.751438 1176 leader_election.cc:290] T 3eaf8baac9f1485f944cd9e74b8b5569 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:09.752187 902 tablet_service.cc:1515] Processing DeleteTablet for tablet 8e6e96b4de214cadba1ccd19390cffb0 with delete_type TABLET_DATA_DELETED (replaced by tablet 3b18a7084727490380d26b3485e4e799) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:09.752615 924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3eaf8baac9f1485f944cd9e74b8b5569" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:09.752825 924 raft_consensus.cc:2466] T 3eaf8baac9f1485f944cd9e74b8b5569 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
W20250901 01:52:09.752902 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 8e6e96b4de214cadba1ccd19390cffb0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8e6e96b4de214cadba1ccd19390cffb0 already in progress: creating tablet
I20250901 01:52:09.753913 992 leader_election.cc:304] T 3eaf8baac9f1485f944cd9e74b8b5569 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:09.754320 1176 raft_consensus.cc:2802] T 3eaf8baac9f1485f944cd9e74b8b5569 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:09.754417 1176 raft_consensus.cc:491] T 3eaf8baac9f1485f944cd9e74b8b5569 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:09.754518 1176 raft_consensus.cc:3058] T 3eaf8baac9f1485f944cd9e74b8b5569 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:09.755527 1150 ts_tablet_manager.cc:1428] T c445c300d71643609071f0f36528a9c4 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20250901 01:52:09.757901 1176 raft_consensus.cc:513] T 3eaf8baac9f1485f944cd9e74b8b5569 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:09.760263 1176 leader_election.cc:290] T 3eaf8baac9f1485f944cd9e74b8b5569 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:09.760319 924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3eaf8baac9f1485f944cd9e74b8b5569" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:09.760650 924 raft_consensus.cc:3058] T 3eaf8baac9f1485f944cd9e74b8b5569 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:09.760650 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet 8e6e96b4de214cadba1ccd19390cffb0 with delete_type TABLET_DATA_DELETED (replaced by tablet 3b18a7084727490380d26b3485e4e799) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:09.761003 758 tablet_service.cc:1515] Processing DeleteTablet for tablet 8e6e96b4de214cadba1ccd19390cffb0 with delete_type TABLET_DATA_DELETED (replaced by tablet 3b18a7084727490380d26b3485e4e799) from {username='slave'} at 127.0.0.1:36178
W20250901 01:52:09.761340 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 8e6e96b4de214cadba1ccd19390cffb0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8e6e96b4de214cadba1ccd19390cffb0 already in progress: creating tablet
W20250901 01:52:09.761790 440 catalog_manager.cc:4908] TS 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): delete failed for tablet 8e6e96b4de214cadba1ccd19390cffb0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8e6e96b4de214cadba1ccd19390cffb0 already in progress: creating tablet
I20250901 01:52:09.762989 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_e066e678c3a8406583dbe851d3424bac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:09.763821 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e066e678c3a8406583dbe851d3424bac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:09.765225 924 raft_consensus.cc:2466] T 3eaf8baac9f1485f944cd9e74b8b5569 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:09.765957 1150 tablet_bootstrap.cc:492] T 8d5b10ede9c84be098795c0a787e5a78 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:09.765973 992 leader_election.cc:304] T 3eaf8baac9f1485f944cd9e74b8b5569 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:09.766359 1176 raft_consensus.cc:2802] T 3eaf8baac9f1485f944cd9e74b8b5569 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:09.766623 1176 raft_consensus.cc:695] T 3eaf8baac9f1485f944cd9e74b8b5569 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:09.766971 1176 consensus_queue.cc:237] T 3eaf8baac9f1485f944cd9e74b8b5569 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:09.768312 1150 tablet_bootstrap.cc:654] T 8d5b10ede9c84be098795c0a787e5a78 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:09.768818 1241 raft_consensus.cc:491] T 116f65a42ab34bf6b8f8113cb7afdc21 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:09.768947 1241 raft_consensus.cc:513] T 116f65a42ab34bf6b8f8113cb7afdc21 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:09.769706 1241 leader_election.cc:290] T 116f65a42ab34bf6b8f8113cb7afdc21 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:09.770589 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "116f65a42ab34bf6b8f8113cb7afdc21" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:09.771220 859 leader_election.cc:343] T 116f65a42ab34bf6b8f8113cb7afdc21 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:09.775503 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet b29e846dfae44f0e9ce3e5fea3763a4f
I20250901 01:52:09.777603 902 tablet_service.cc:1515] Processing DeleteTablet for tablet b29e846dfae44f0e9ce3e5fea3763a4f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:09 UTC) from {username='slave'} at 127.0.0.1:52846
W20250901 01:52:09.778391 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet b29e846dfae44f0e9ce3e5fea3763a4f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b29e846dfae44f0e9ce3e5fea3763a4f already in progress: creating tablet
I20250901 01:52:09.779742 1150 tablet_bootstrap.cc:492] T 8d5b10ede9c84be098795c0a787e5a78 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:09.779960 1150 ts_tablet_manager.cc:1397] T 8d5b10ede9c84be098795c0a787e5a78 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.003s
I20250901 01:52:09.780754 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet b29e846dfae44f0e9ce3e5fea3763a4f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:09 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:09.781080 1150 raft_consensus.cc:357] T 8d5b10ede9c84be098795c0a787e5a78 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:09.781374 1150 raft_consensus.cc:383] T 8d5b10ede9c84be098795c0a787e5a78 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:09.781476 1150 raft_consensus.cc:738] T 8d5b10ede9c84be098795c0a787e5a78 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
W20250901 01:52:09.781698 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet b29e846dfae44f0e9ce3e5fea3763a4f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b29e846dfae44f0e9ce3e5fea3763a4f already in progress: creating tablet
I20250901 01:52:09.782143 758 tablet_service.cc:1515] Processing DeleteTablet for tablet b29e846dfae44f0e9ce3e5fea3763a4f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:09 UTC) from {username='slave'} at 127.0.0.1:36178
W20250901 01:52:09.782970 440 catalog_manager.cc:4908] TS 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): delete failed for tablet b29e846dfae44f0e9ce3e5fea3763a4f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b29e846dfae44f0e9ce3e5fea3763a4f already in progress: creating tablet
I20250901 01:52:09.784396 1150 consensus_queue.cc:260] T 8d5b10ede9c84be098795c0a787e5a78 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:09.784749 455 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_b27678f366d74143b716da42e7ca32c3" } modify_external_catalogs: true
I20250901 01:52:09.784943 455 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_b27678f366d74143b716da42e7ca32c3" } modify_external_catalogs: true
I20250901 01:52:09.789983 1241 raft_consensus.cc:491] T 8f2e702121c845e0a09a254bbb81c142 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:09.790249 1241 raft_consensus.cc:513] T 8f2e702121c845e0a09a254bbb81c142 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:09.791127 1241 leader_election.cc:290] T 8f2e702121c845e0a09a254bbb81c142 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:09.792138 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f2e702121c845e0a09a254bbb81c142" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:09.792359 1057 raft_consensus.cc:2466] T 8f2e702121c845e0a09a254bbb81c142 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00c282f779c1477d920f24dc0dc16628 in term 0.
I20250901 01:52:09.793097 859 leader_election.cc:304] T 8f2e702121c845e0a09a254bbb81c142 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:09.793809 1241 raft_consensus.cc:2802] T 8f2e702121c845e0a09a254bbb81c142 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:09.793962 1241 raft_consensus.cc:491] T 8f2e702121c845e0a09a254bbb81c142 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:09.794121 1241 raft_consensus.cc:3058] T 8f2e702121c845e0a09a254bbb81c142 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:09.796460 1158 raft_consensus.cc:491] T 288e8029382f4154bc1e2103b3035a60 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:09.796607 1158 raft_consensus.cc:513] T 288e8029382f4154bc1e2103b3035a60 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:09.796686 447 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 3eaf8baac9f1485f944cd9e74b8b5569 (table default.Table_947ec0e1a97a40b59be1e234bcaf4e0d [id=5d14706774d345c48ca3575feabb17b0]). Expected version 1 got 0
I20250901 01:52:09.797176 1158 leader_election.cc:290] T 288e8029382f4154bc1e2103b3035a60 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:09.797029 447 catalog_manager.cc:5582] T b11e8f77243e4764b96548d1132b8e5a P 00c282f779c1477d920f24dc0dc16628 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00c282f779c1477d920f24dc0dc16628 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "00c282f779c1477d920f24dc0dc16628" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: UNKNOWN } } }
I20250901 01:52:09.797566 447 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet b11e8f77243e4764b96548d1132b8e5a (table default.Table_16b3ac9fb2304c8095e0270038501db0 [id=a39e8b171a874df1890d072732e55df9]). Expected version 1 got 0
I20250901 01:52:09.797632 1241 raft_consensus.cc:513] T 8f2e702121c845e0a09a254bbb81c142 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:09.798060 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "288e8029382f4154bc1e2103b3035a60" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:09.798524 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 81fa34655a71439a85b8a932b7603d7d
W20250901 01:52:09.798499 859 leader_election.cc:343] T 288e8029382f4154bc1e2103b3035a60 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:09.798374 1241 leader_election.cc:290] T 8f2e702121c845e0a09a254bbb81c142 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:09.798707 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f2e702121c845e0a09a254bbb81c142" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58"
I20250901 01:52:09.798894 1057 raft_consensus.cc:3058] T 8f2e702121c845e0a09a254bbb81c142 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:09.799280 1150 ts_tablet_manager.cc:1428] T 8d5b10ede9c84be098795c0a787e5a78 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.019s user 0.001s sys 0.002s
I20250901 01:52:09.800381 902 tablet_service.cc:1515] Processing DeleteTablet for tablet 81fa34655a71439a85b8a932b7603d7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:09 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:09.800704 1164 tablet_replica.cc:331] T 81fa34655a71439a85b8a932b7603d7d P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:09.800904 1164 raft_consensus.cc:2241] T 81fa34655a71439a85b8a932b7603d7d P 00c282f779c1477d920f24dc0dc16628 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:09.800940 449 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 31390f38258c465cb1798615a1fcc790
I20250901 01:52:09.801347 1164 pending_rounds.cc:70] T 81fa34655a71439a85b8a932b7603d7d P 00c282f779c1477d920f24dc0dc16628: Trying to abort 1 pending ops.
I20250901 01:52:09.801553 1164 pending_rounds.cc:77] T 81fa34655a71439a85b8a932b7603d7d P 00c282f779c1477d920f24dc0dc16628: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7195408503720812544 op_type: NO_OP noop_request { }
I20250901 01:52:09.801661 1057 raft_consensus.cc:2466] T 8f2e702121c845e0a09a254bbb81c142 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c282f779c1477d920f24dc0dc16628 in term 1.
I20250901 01:52:09.801734 1164 raft_consensus.cc:2887] T 81fa34655a71439a85b8a932b7603d7d P 00c282f779c1477d920f24dc0dc16628 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250901 01:52:09.801859 1164 raft_consensus.cc:2270] T 81fa34655a71439a85b8a932b7603d7d P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:09.801980 1176 raft_consensus.cc:491] T c31993d35207443288835f90091f73cf P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:09.802114 1176 raft_consensus.cc:513] T c31993d35207443288835f90091f73cf P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:09.802690 1176 leader_election.cc:290] T c31993d35207443288835f90091f73cf P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:09.803177 1164 ts_tablet_manager.cc:1905] T 81fa34655a71439a85b8a932b7603d7d P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:09.804221 1150 tablet_bootstrap.cc:492] T 97e40c318f64455d91b0e4f8866990ce P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:09.805027 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet 81fa34655a71439a85b8a932b7603d7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:09 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:09.805377 1166 tablet_replica.cc:331] T 81fa34655a71439a85b8a932b7603d7d P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:09.805608 1166 raft_consensus.cc:2241] T 81fa34655a71439a85b8a932b7603d7d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:09.805827 1166 raft_consensus.cc:2270] T 81fa34655a71439a85b8a932b7603d7d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:09.806478 1166 ts_tablet_manager.cc:1905] T 81fa34655a71439a85b8a932b7603d7d P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:09.807888 1150 tablet_bootstrap.cc:654] T 97e40c318f64455d91b0e4f8866990ce P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:09.808780 859 leader_election.cc:304] T 8f2e702121c845e0a09a254bbb81c142 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
W20250901 01:52:09.810225 992 leader_election.cc:343] T c31993d35207443288835f90091f73cf P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: c31993d35207443288835f90091f73cf
I20250901 01:52:09.811151 1164 ts_tablet_manager.cc:1918] T 81fa34655a71439a85b8a932b7603d7d P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:09.812534 1164 log.cc:1199] T 81fa34655a71439a85b8a932b7603d7d P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/81fa34655a71439a85b8a932b7603d7d
I20250901 01:52:09.813136 1164 ts_tablet_manager.cc:1939] T 81fa34655a71439a85b8a932b7603d7d P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:09.817555 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_063de0c73851415196ba2e77742edee4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:09.815578 1150 tablet_bootstrap.cc:492] T 97e40c318f64455d91b0e4f8866990ce P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:09.816550 1241 raft_consensus.cc:2802] T 8f2e702121c845e0a09a254bbb81c142 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election won for term 1
W20250901 01:52:09.818387 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_063de0c73851415196ba2e77742edee4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:09.819322 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 81fa34655a71439a85b8a932b7603d7d (table default.Table_196019dc3a1d4cbea8d5a87f139c899d [id=928ead2844c646a881f5f327cca0d3b9]) successfully deleted
I20250901 01:52:09.820178 1241 raft_consensus.cc:695] T 8f2e702121c845e0a09a254bbb81c142 P 00c282f779c1477d920f24dc0dc16628 [term 1 LEADER]: Becoming Leader. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Running, Role: LEADER
I20250901 01:52:09.820597 1241 consensus_queue.cc:237] T 8f2e702121c845e0a09a254bbb81c142 P 00c282f779c1477d920f24dc0dc16628 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:09.823006 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_edc9e269bf86407799be609875a22b0f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:09.823658 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_edc9e269bf86407799be609875a22b0f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:09.825074 1150 ts_tablet_manager.cc:1397] T 97e40c318f64455d91b0e4f8866990ce P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.001s
I20250901 01:52:09.826277 1150 raft_consensus.cc:357] T 97e40c318f64455d91b0e4f8866990ce P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:09.826547 1150 raft_consensus.cc:383] T 97e40c318f64455d91b0e4f8866990ce P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:09.826680 1150 raft_consensus.cc:738] T 97e40c318f64455d91b0e4f8866990ce P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:09.826956 1150 consensus_queue.cc:260] T 97e40c318f64455d91b0e4f8866990ce P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:09.830786 1166 ts_tablet_manager.cc:1918] T 81fa34655a71439a85b8a932b7603d7d P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:09.834583 1150 ts_tablet_manager.cc:1428] T 97e40c318f64455d91b0e4f8866990ce P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20250901 01:52:09.834636 758 tablet_service.cc:1515] Processing DeleteTablet for tablet 81fa34655a71439a85b8a932b7603d7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:09 UTC) from {username='slave'} at 127.0.0.1:36178
I20250901 01:52:09.835052 1163 tablet_replica.cc:331] T 81fa34655a71439a85b8a932b7603d7d P 0cf599f74322478d91475df234e58965: stopping tablet replica
I20250901 01:52:09.835325 1163 raft_consensus.cc:2241] T 81fa34655a71439a85b8a932b7603d7d P 0cf599f74322478d91475df234e58965 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:09.835618 1163 raft_consensus.cc:2270] T 81fa34655a71439a85b8a932b7603d7d P 0cf599f74322478d91475df234e58965 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:09.836728 1163 ts_tablet_manager.cc:1905] T 81fa34655a71439a85b8a932b7603d7d P 0cf599f74322478d91475df234e58965: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:09.838025 1150 tablet_bootstrap.cc:492] T 7a8a3aca289c4735abeb9c95c414ab76 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:09.840551 1150 tablet_bootstrap.cc:654] T 7a8a3aca289c4735abeb9c95c414ab76 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:09.845783 1163 ts_tablet_manager.cc:1918] T 81fa34655a71439a85b8a932b7603d7d P 0cf599f74322478d91475df234e58965: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:09.848876 1166 log.cc:1199] T 81fa34655a71439a85b8a932b7603d7d P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/81fa34655a71439a85b8a932b7603d7d
I20250901 01:52:09.849687 1166 ts_tablet_manager.cc:1939] T 81fa34655a71439a85b8a932b7603d7d P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:09.852134 1150 tablet_bootstrap.cc:492] T 7a8a3aca289c4735abeb9c95c414ab76 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:09.852393 1150 ts_tablet_manager.cc:1397] T 7a8a3aca289c4735abeb9c95c414ab76 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20250901 01:52:09.852362 1163 log.cc:1199] T 81fa34655a71439a85b8a932b7603d7d P 0cf599f74322478d91475df234e58965: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-0/wal/wals/81fa34655a71439a85b8a932b7603d7d
I20250901 01:52:09.853371 1163 ts_tablet_manager.cc:1939] T 81fa34655a71439a85b8a932b7603d7d P 0cf599f74322478d91475df234e58965: Deleting consensus metadata
I20250901 01:52:09.853644 1150 raft_consensus.cc:357] T 7a8a3aca289c4735abeb9c95c414ab76 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:09.853955 1150 raft_consensus.cc:383] T 7a8a3aca289c4735abeb9c95c414ab76 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:09.854066 1150 raft_consensus.cc:738] T 7a8a3aca289c4735abeb9c95c414ab76 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:09.859279 440 catalog_manager.cc:4928] TS 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): tablet 81fa34655a71439a85b8a932b7603d7d (table default.Table_196019dc3a1d4cbea8d5a87f139c899d [id=928ead2844c646a881f5f327cca0d3b9]) successfully deleted
I20250901 01:52:09.859836 1150 consensus_queue.cc:260] T 7a8a3aca289c4735abeb9c95c414ab76 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:09.863176 1150 ts_tablet_manager.cc:1428] T 7a8a3aca289c4735abeb9c95c414ab76 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20250901 01:52:09.866333 1150 tablet_bootstrap.cc:492] T 4f8e3fcaa19740d397f4d118ac4cbdb0 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:09.868656 1150 tablet_bootstrap.cc:654] T 4f8e3fcaa19740d397f4d118ac4cbdb0 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:09.869154 441 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 81fa34655a71439a85b8a932b7603d7d (table default.Table_196019dc3a1d4cbea8d5a87f139c899d [id=928ead2844c646a881f5f327cca0d3b9]) successfully deleted
I20250901 01:52:09.877781 1150 tablet_bootstrap.cc:492] T 4f8e3fcaa19740d397f4d118ac4cbdb0 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:09.877914 1150 ts_tablet_manager.cc:1397] T 4f8e3fcaa19740d397f4d118ac4cbdb0 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20250901 01:52:09.878638 1150 raft_consensus.cc:357] T 4f8e3fcaa19740d397f4d118ac4cbdb0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:09.878813 1150 raft_consensus.cc:383] T 4f8e3fcaa19740d397f4d118ac4cbdb0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:09.878947 1150 raft_consensus.cc:738] T 4f8e3fcaa19740d397f4d118ac4cbdb0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:09.879242 1150 consensus_queue.cc:260] T 4f8e3fcaa19740d397f4d118ac4cbdb0 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:09.886236 1150 ts_tablet_manager.cc:1428] T 4f8e3fcaa19740d397f4d118ac4cbdb0 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
W20250901 01:52:09.887684 454 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 85322a864d154f299f2d670fdd2a7bfb has no consensus state
I20250901 01:52:09.888016 454 master_service.cc:948] ReplaceTablet: received request to replace tablet 8e6e96b4de214cadba1ccd19390cffb0 from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:09.888245 1150 tablet_bootstrap.cc:492] T 8e6e96b4de214cadba1ccd19390cffb0 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:09.889394 446 master_service.cc:948] ReplaceTablet: received request to replace tablet 2a0d125423a043dcb11e67345599e468 from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:09.889879 1150 tablet_bootstrap.cc:654] T 8e6e96b4de214cadba1ccd19390cffb0 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:09.890339 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet c31993d35207443288835f90091f73cf with delete_type TABLET_DATA_DELETED (replaced by tablet 4cdd369214ef4f5ebabddbf28b2ae3f7) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:09.898447 1166 tablet_replica.cc:331] T c31993d35207443288835f90091f73cf P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:09.899037 1166 raft_consensus.cc:2241] T c31993d35207443288835f90091f73cf P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:09.899627 1166 raft_consensus.cc:2270] T c31993d35207443288835f90091f73cf P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:09.902247 1150 tablet_bootstrap.cc:492] T 8e6e96b4de214cadba1ccd19390cffb0 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:09.902413 1150 ts_tablet_manager.cc:1397] T 8e6e96b4de214cadba1ccd19390cffb0 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20250901 01:52:09.903157 1150 raft_consensus.cc:357] T 8e6e96b4de214cadba1ccd19390cffb0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:09.903362 1150 raft_consensus.cc:383] T 8e6e96b4de214cadba1ccd19390cffb0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:09.903368 1166 ts_tablet_manager.cc:1905] T c31993d35207443288835f90091f73cf P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:09.903426 1150 raft_consensus.cc:738] T 8e6e96b4de214cadba1ccd19390cffb0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:09.909169 1151 tablet_bootstrap.cc:492] T 288e8029382f4154bc1e2103b3035a60 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:09.909336 1151 ts_tablet_manager.cc:1397] T 288e8029382f4154bc1e2103b3035a60 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.169s user 0.017s sys 0.057s
I20250901 01:52:09.911697 1151 raft_consensus.cc:357] T 288e8029382f4154bc1e2103b3035a60 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:09.911924 1151 raft_consensus.cc:383] T 288e8029382f4154bc1e2103b3035a60 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:09.912014 1151 raft_consensus.cc:738] T 288e8029382f4154bc1e2103b3035a60 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:09.913486 1151 consensus_queue.cc:260] T 288e8029382f4154bc1e2103b3035a60 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:09.915009 1150 consensus_queue.cc:260] T 8e6e96b4de214cadba1ccd19390cffb0 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:09.915537 1150 ts_tablet_manager.cc:1428] T 8e6e96b4de214cadba1ccd19390cffb0 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20250901 01:52:09.916918 1151 ts_tablet_manager.cc:1428] T 288e8029382f4154bc1e2103b3035a60 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.007s user 0.001s sys 0.002s
W20250901 01:52:09.917025 448 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 22691943a134400c8df4617240e5d3ef has no consensus state
I20250901 01:52:09.919440 1151 tablet_bootstrap.cc:492] T 061b760d8e6647c6ab0b5d538840a230 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:09.919534 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_1ddaf993eaa54728ad30f6001b6f9890"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:09.920150 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1ddaf993eaa54728ad30f6001b6f9890 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:09.922736 1150 tablet_bootstrap.cc:492] T 24ddc13d1bb04db4a7850acbff091577 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:09.922981 1166 ts_tablet_manager.cc:1918] T c31993d35207443288835f90091f73cf P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:09.923127 1166 log.cc:1199] T c31993d35207443288835f90091f73cf P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/c31993d35207443288835f90091f73cf
I20250901 01:52:09.923512 1166 ts_tablet_manager.cc:1939] T c31993d35207443288835f90091f73cf P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:09.924264 441 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet c31993d35207443288835f90091f73cf (table default.Table_cf1c9f5fc5104126b163225918707284 [id=00ae7e46c65d4a56b1ffa193b853421a]) successfully deleted
I20250901 01:52:09.924480 1150 tablet_bootstrap.cc:654] T 24ddc13d1bb04db4a7850acbff091577 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:09.926178 1151 tablet_bootstrap.cc:654] T 061b760d8e6647c6ab0b5d538840a230 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:09.934458 450 master_service.cc:948] ReplaceTablet: received request to replace tablet a4f5685260304d589dd467953d9bebfc from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:09.941236 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_d61fc9783e7b45e49afb36280adcbd86"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:09.942009 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d61fc9783e7b45e49afb36280adcbd86 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:09.945245 1150 tablet_bootstrap.cc:492] T 24ddc13d1bb04db4a7850acbff091577 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:09.945399 1150 ts_tablet_manager.cc:1397] T 24ddc13d1bb04db4a7850acbff091577 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.008s
I20250901 01:52:09.946210 1150 raft_consensus.cc:357] T 24ddc13d1bb04db4a7850acbff091577 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } }
I20250901 01:52:09.946393 1150 raft_consensus.cc:383] T 24ddc13d1bb04db4a7850acbff091577 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:09.946470 1150 raft_consensus.cc:738] T 24ddc13d1bb04db4a7850acbff091577 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:09.948418 1150 consensus_queue.cc:260] T 24ddc13d1bb04db4a7850acbff091577 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } }
I20250901 01:52:09.954206 1150 ts_tablet_manager.cc:1428] T 24ddc13d1bb04db4a7850acbff091577 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20250901 01:52:09.960525 1150 tablet_bootstrap.cc:492] T 4d333b670df84f308f0cbf6da636b768 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:09.962404 1150 tablet_bootstrap.cc:654] T 4d333b670df84f308f0cbf6da636b768 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:09.964386 1241 raft_consensus.cc:491] T bf302e76746b4f7bbe4469914f83720f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:09.964543 1241 raft_consensus.cc:513] T bf302e76746b4f7bbe4469914f83720f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:09.965206 1241 leader_election.cc:290] T bf302e76746b4f7bbe4469914f83720f P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:09.965802 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf302e76746b4f7bbe4469914f83720f" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:09.966485 859 leader_election.cc:343] T bf302e76746b4f7bbe4469914f83720f P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:09.968685 1150 tablet_bootstrap.cc:492] T 4d333b670df84f308f0cbf6da636b768 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:09.968850 1150 ts_tablet_manager.cc:1397] T 4d333b670df84f308f0cbf6da636b768 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.001s
I20250901 01:52:09.969604 1150 raft_consensus.cc:357] T 4d333b670df84f308f0cbf6da636b768 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } }
I20250901 01:52:09.969802 1150 raft_consensus.cc:383] T 4d333b670df84f308f0cbf6da636b768 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:09.969872 1150 raft_consensus.cc:738] T 4d333b670df84f308f0cbf6da636b768 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:09.970100 1150 consensus_queue.cc:260] T 4d333b670df84f308f0cbf6da636b768 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } }
I20250901 01:52:09.976744 1150 ts_tablet_manager.cc:1428] T 4d333b670df84f308f0cbf6da636b768 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20250901 01:52:09.980474 902 tablet_service.cc:1515] Processing DeleteTablet for tablet 8f2e702121c845e0a09a254bbb81c142 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:08 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:09.980818 1164 tablet_replica.cc:331] T 8f2e702121c845e0a09a254bbb81c142 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:09.980957 900 tablet_service.cc:1515] Processing DeleteTablet for tablet 81fa34655a71439a85b8a932b7603d7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:09 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:09.981053 1164 raft_consensus.cc:2241] T 8f2e702121c845e0a09a254bbb81c142 P 00c282f779c1477d920f24dc0dc16628 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:09.981354 1164 pending_rounds.cc:70] T 8f2e702121c845e0a09a254bbb81c142 P 00c282f779c1477d920f24dc0dc16628: Trying to abort 1 pending ops.
I20250901 01:52:09.981431 1164 pending_rounds.cc:77] T 8f2e702121c845e0a09a254bbb81c142 P 00c282f779c1477d920f24dc0dc16628: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7195408506148225024 op_type: NO_OP noop_request { }
I20250901 01:52:09.981556 1164 raft_consensus.cc:2887] T 8f2e702121c845e0a09a254bbb81c142 P 00c282f779c1477d920f24dc0dc16628 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250901 01:52:09.981668 1164 raft_consensus.cc:2270] T 8f2e702121c845e0a09a254bbb81c142 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:09.981818 1150 tablet_bootstrap.cc:492] T a4f5685260304d589dd467953d9bebfc P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:09.980845 901 tablet_service.cc:1515] Processing DeleteTablet for tablet e8621a0c15a94ed7bd63b035ba2fded7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:09 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:09.982501 1164 ts_tablet_manager.cc:1905] T 8f2e702121c845e0a09a254bbb81c142 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:09.982775 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_6aa17d35edb04da19ea5d0ce969b34e2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:09.983503 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6aa17d35edb04da19ea5d0ce969b34e2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:09.983639 1150 tablet_bootstrap.cc:654] T a4f5685260304d589dd467953d9bebfc P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:09.989051 1150 tablet_bootstrap.cc:492] T a4f5685260304d589dd467953d9bebfc P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:09.989207 1150 ts_tablet_manager.cc:1397] T a4f5685260304d589dd467953d9bebfc P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20250901 01:52:09.989979 1150 raft_consensus.cc:357] T a4f5685260304d589dd467953d9bebfc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } }
I20250901 01:52:09.990149 1150 raft_consensus.cc:383] T a4f5685260304d589dd467953d9bebfc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:09.990223 1150 raft_consensus.cc:738] T a4f5685260304d589dd467953d9bebfc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:09.990432 1150 consensus_queue.cc:260] T a4f5685260304d589dd467953d9bebfc P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } }
I20250901 01:52:09.997005 1150 ts_tablet_manager.cc:1428] T a4f5685260304d589dd467953d9bebfc P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250901 01:52:10.000294 1150 tablet_bootstrap.cc:492] T 4cdd369214ef4f5ebabddbf28b2ae3f7 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:10.002758 1164 ts_tablet_manager.cc:1918] T 8f2e702121c845e0a09a254bbb81c142 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:10.002946 1164 log.cc:1199] T 8f2e702121c845e0a09a254bbb81c142 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/8f2e702121c845e0a09a254bbb81c142
I20250901 01:52:10.003465 1164 ts_tablet_manager.cc:1939] T 8f2e702121c845e0a09a254bbb81c142 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:10.004155 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 8f2e702121c845e0a09a254bbb81c142 (table default.Table_3cdfd4047a394877b110a5f00a15c062 [id=512e7d06c7254d889809df25c29fc4c2]) successfully deleted
I20250901 01:52:10.004307 1150 tablet_bootstrap.cc:654] T 4cdd369214ef4f5ebabddbf28b2ae3f7 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:10.010689 1150 tablet_bootstrap.cc:492] T 4cdd369214ef4f5ebabddbf28b2ae3f7 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:10.010845 1150 ts_tablet_manager.cc:1397] T 4cdd369214ef4f5ebabddbf28b2ae3f7 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.002s
I20250901 01:52:10.011516 1150 raft_consensus.cc:357] T 4cdd369214ef4f5ebabddbf28b2ae3f7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:10.011703 1150 raft_consensus.cc:383] T 4cdd369214ef4f5ebabddbf28b2ae3f7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:10.011775 1150 raft_consensus.cc:738] T 4cdd369214ef4f5ebabddbf28b2ae3f7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:10.011991 1150 consensus_queue.cc:260] T 4cdd369214ef4f5ebabddbf28b2ae3f7 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
W20250901 01:52:10.025198 440 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 81fa34655a71439a85b8a932b7603d7d because the tablet was not found. No further retry: Not found: Tablet not found: 81fa34655a71439a85b8a932b7603d7d
W20250901 01:52:10.025923 440 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet e8621a0c15a94ed7bd63b035ba2fded7 because the tablet was not found. No further retry: Not found: Tablet not found: e8621a0c15a94ed7bd63b035ba2fded7
I20250901 01:52:10.034216 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_12450bd55edc4f7d96797422cfae2c50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:10.035022 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_12450bd55edc4f7d96797422cfae2c50 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:10.048600 1151 tablet_bootstrap.cc:492] T 061b760d8e6647c6ab0b5d538840a230 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:10.048835 1151 ts_tablet_manager.cc:1397] T 061b760d8e6647c6ab0b5d538840a230 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.129s user 0.019s sys 0.042s
I20250901 01:52:10.049892 1151 raft_consensus.cc:357] T 061b760d8e6647c6ab0b5d538840a230 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:10.050202 1151 raft_consensus.cc:383] T 061b760d8e6647c6ab0b5d538840a230 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:10.050295 1151 raft_consensus.cc:738] T 061b760d8e6647c6ab0b5d538840a230 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:10.051693 454 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_87bd0229edd44590b0b77f0b9dea7b1f" } new_table_name: "default.Table_1c3a507d45d64e389a6c07341b00b9fc" modify_external_catalogs: true
I20250901 01:52:10.052270 1151 consensus_queue.cc:260] T 061b760d8e6647c6ab0b5d538840a230 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:10.056046 1151 ts_tablet_manager.cc:1428] T 061b760d8e6647c6ab0b5d538840a230 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250901 01:52:10.060140 1151 tablet_bootstrap.cc:492] T 116f65a42ab34bf6b8f8113cb7afdc21 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:10.062455 1151 tablet_bootstrap.cc:654] T 116f65a42ab34bf6b8f8113cb7afdc21 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:10.067551 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet a4f5685260304d589dd467953d9bebfc
I20250901 01:52:10.068133 450 catalog_manager.cc:6652] ReplaceTablet: tablet a4f5685260304d589dd467953d9bebfc deleted and replaced by tablet 1f18420261ae41459e1f25b927bd84a9
I20250901 01:52:10.069197 1150 ts_tablet_manager.cc:1428] T 4cdd369214ef4f5ebabddbf28b2ae3f7 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.058s user 0.001s sys 0.001s
I20250901 01:52:10.069541 1150 tablet_bootstrap.cc:492] T 342b7fde3ede494b9eec6e361db255dc P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:10.073163 1241 raft_consensus.cc:491] T 97e40c318f64455d91b0e4f8866990ce P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:10.073333 1241 raft_consensus.cc:513] T 97e40c318f64455d91b0e4f8866990ce P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:10.073845 901 tablet_service.cc:1515] Processing DeleteTablet for tablet a4f5685260304d589dd467953d9bebfc with delete_type TABLET_DATA_DELETED (replaced by tablet 1f18420261ae41459e1f25b927bd84a9) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:10.074033 1241 leader_election.cc:290] T 97e40c318f64455d91b0e4f8866990ce P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:10.074129 1164 tablet_replica.cc:331] T a4f5685260304d589dd467953d9bebfc P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:10.074326 1164 raft_consensus.cc:2241] T a4f5685260304d589dd467953d9bebfc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:10.074512 1164 raft_consensus.cc:2270] T a4f5685260304d589dd467953d9bebfc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:10.074491 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet a4f5685260304d589dd467953d9bebfc with delete_type TABLET_DATA_DELETED (replaced by tablet 1f18420261ae41459e1f25b927bd84a9) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:10.074983 758 tablet_service.cc:1515] Processing DeleteTablet for tablet a4f5685260304d589dd467953d9bebfc with delete_type TABLET_DATA_DELETED (replaced by tablet 1f18420261ae41459e1f25b927bd84a9) from {username='slave'} at 127.0.0.1:36178
I20250901 01:52:10.075745 1164 ts_tablet_manager.cc:1905] T a4f5685260304d589dd467953d9bebfc P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:10.076622 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet a4f5685260304d589dd467953d9bebfc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a4f5685260304d589dd467953d9bebfc already in progress: creating tablet
I20250901 01:52:10.076588 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "97e40c318f64455d91b0e4f8866990ce" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:10.076644 1241 raft_consensus.cc:491] T c297e8bf11ff467a854eab41117ac376 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:10.076774 1241 raft_consensus.cc:513] T c297e8bf11ff467a854eab41117ac376 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:10.077298 859 leader_election.cc:343] T 97e40c318f64455d91b0e4f8866990ce P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:10.077418 1241 leader_election.cc:290] T c297e8bf11ff467a854eab41117ac376 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:10.078080 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c297e8bf11ff467a854eab41117ac376" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:10.078820 440 catalog_manager.cc:4908] TS 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): delete failed for tablet a4f5685260304d589dd467953d9bebfc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a4f5685260304d589dd467953d9bebfc already in progress: creating tablet
W20250901 01:52:10.079363 859 leader_election.cc:343] T c297e8bf11ff467a854eab41117ac376 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:10.082691 1164 ts_tablet_manager.cc:1918] T a4f5685260304d589dd467953d9bebfc P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:10.082885 1164 log.cc:1199] T a4f5685260304d589dd467953d9bebfc P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/a4f5685260304d589dd467953d9bebfc
I20250901 01:52:10.083456 1164 ts_tablet_manager.cc:1939] T a4f5685260304d589dd467953d9bebfc P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:10.084982 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet a4f5685260304d589dd467953d9bebfc (table default.Table_a2bf411424c74109a7c2baa28a4db8b9 [id=7c904bfb6cc94518873d6806d1fda600]) successfully deleted
I20250901 01:52:10.087808 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_409dad98d7eb455a8e16edfb4acedb34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:10.088646 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_409dad98d7eb455a8e16edfb4acedb34 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:10.089959 1150 tablet_bootstrap.cc:654] T 342b7fde3ede494b9eec6e361db255dc P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:10.095638 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_87c07d8f93414f3aaf9b47d8f7bbd24b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:10.096369 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_87c07d8f93414f3aaf9b47d8f7bbd24b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:10.098479 447 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_aa569ededbee4b1bb3c3a2f20f9748cb" } new_table_name: "default.Table_a3a2b89da62b445e83cd3c88fd4fc775" modify_external_catalogs: true
I20250901 01:52:10.111163 449 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_f9876cefc9844c14a66a3f7c233d2f0c" } new_table_name: "default.Table_57db59e62bcb45a8b2827445e56dc7de" modify_external_catalogs: true
I20250901 01:52:10.114259 1241 consensus_queue.cc:1035] T b11e8f77243e4764b96548d1132b8e5a P 00c282f779c1477d920f24dc0dc16628 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:10.121263 1241 raft_consensus.cc:491] T c445c300d71643609071f0f36528a9c4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:10.121471 1241 raft_consensus.cc:513] T c445c300d71643609071f0f36528a9c4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:10.122346 1241 leader_election.cc:290] T c445c300d71643609071f0f36528a9c4 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:10.125377 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c445c300d71643609071f0f36528a9c4" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:10.126124 859 leader_election.cc:343] T c445c300d71643609071f0f36528a9c4 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
W20250901 01:52:10.130782 454 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7cc1efcb6bff4aba8015d2d3b7f4c923 has no consensus state
I20250901 01:52:10.132709 454 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_64b711e409d7434c8b176bf1a3ef8c86" } modify_external_catalogs: true
I20250901 01:52:10.132951 454 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_64b711e409d7434c8b176bf1a3ef8c86" } modify_external_catalogs: true
I20250901 01:52:10.137079 1151 tablet_bootstrap.cc:492] T 116f65a42ab34bf6b8f8113cb7afdc21 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:10.137229 1151 ts_tablet_manager.cc:1397] T 116f65a42ab34bf6b8f8113cb7afdc21 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.077s user 0.021s sys 0.006s
W20250901 01:52:10.138067 439 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 22691943a134400c8df4617240e5d3ef (table default.Table_dd24f0428569419fad1bbe8a7f9f08c1 [id=5a33cfdf60684867bb5f36d84ee80468], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 22691943a134400c8df4617240e5d3ef has no consensus state
I20250901 01:52:10.138051 1151 raft_consensus.cc:357] T 116f65a42ab34bf6b8f8113cb7afdc21 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:10.138226 1151 raft_consensus.cc:383] T 116f65a42ab34bf6b8f8113cb7afdc21 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:10.138306 1151 raft_consensus.cc:738] T 116f65a42ab34bf6b8f8113cb7afdc21 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:10.139161 1151 consensus_queue.cc:260] T 116f65a42ab34bf6b8f8113cb7afdc21 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:10.146251 1151 ts_tablet_manager.cc:1428] T 116f65a42ab34bf6b8f8113cb7afdc21 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20250901 01:52:10.148386 1151 tablet_bootstrap.cc:492] T c297e8bf11ff467a854eab41117ac376 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:10.155129 448 catalog_manager.cc:5582] T 3eaf8baac9f1485f944cd9e74b8b5569 P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } }
I20250901 01:52:10.155582 448 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 3eaf8baac9f1485f944cd9e74b8b5569 (table default.Table_947ec0e1a97a40b59be1e234bcaf4e0d [id=5d14706774d345c48ca3575feabb17b0]). Expected version 1 got 0
I20250901 01:52:10.156095 1151 tablet_bootstrap.cc:654] T c297e8bf11ff467a854eab41117ac376 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:10.159761 452 master_service.cc:948] ReplaceTablet: received request to replace tablet c297e8bf11ff467a854eab41117ac376 from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:10.162447 758 tablet_service.cc:1468] Processing CreateTablet for tablet 360755d30b5d47ba98c9c7e23ff5ab7e (DEFAULT_TABLE table=default.Table_64b711e409d7434c8b176bf1a3ef8c86 [id=a762733ef34449a5a7a3898464cd8045]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:10.163021 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 360755d30b5d47ba98c9c7e23ff5ab7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:10.164665 1158 raft_consensus.cc:491] T 8e6e96b4de214cadba1ccd19390cffb0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:10.164891 1158 raft_consensus.cc:513] T 8e6e96b4de214cadba1ccd19390cffb0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:10.165607 1158 leader_election.cc:290] T 8e6e96b4de214cadba1ccd19390cffb0 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:10.169207 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e6e96b4de214cadba1ccd19390cffb0" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:10.170032 859 leader_election.cc:343] T 8e6e96b4de214cadba1ccd19390cffb0 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:10.170225 1032 tablet_service.cc:1468] Processing CreateTablet for tablet 360755d30b5d47ba98c9c7e23ff5ab7e (DEFAULT_TABLE table=default.Table_64b711e409d7434c8b176bf1a3ef8c86 [id=a762733ef34449a5a7a3898464cd8045]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:10.170665 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 360755d30b5d47ba98c9c7e23ff5ab7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:10.171025 1151 tablet_bootstrap.cc:492] T c297e8bf11ff467a854eab41117ac376 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:10.171197 1151 ts_tablet_manager.cc:1397] T c297e8bf11ff467a854eab41117ac376 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.023s user 0.000s sys 0.008s
I20250901 01:52:10.172101 1151 raft_consensus.cc:357] T c297e8bf11ff467a854eab41117ac376 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:10.172319 1151 raft_consensus.cc:383] T c297e8bf11ff467a854eab41117ac376 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:10.172387 1151 raft_consensus.cc:738] T c297e8bf11ff467a854eab41117ac376 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:10.173076 1151 consensus_queue.cc:260] T c297e8bf11ff467a854eab41117ac376 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:10.174985 1151 ts_tablet_manager.cc:1428] T c297e8bf11ff467a854eab41117ac376 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250901 01:52:10.175344 1151 tablet_bootstrap.cc:492] T c445c300d71643609071f0f36528a9c4 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:10.177188 1151 tablet_bootstrap.cc:654] T c445c300d71643609071f0f36528a9c4 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:10.180191 901 tablet_service.cc:1468] Processing CreateTablet for tablet 360755d30b5d47ba98c9c7e23ff5ab7e (DEFAULT_TABLE table=default.Table_64b711e409d7434c8b176bf1a3ef8c86 [id=a762733ef34449a5a7a3898464cd8045]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:10.180742 901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 360755d30b5d47ba98c9c7e23ff5ab7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:10.181005 1176 raft_consensus.cc:491] T c297e8bf11ff467a854eab41117ac376 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:10.181213 750 tablet_service.cc:1468] Processing CreateTablet for tablet 2685fb7258264463b5f1ec99cd7d67ad (DEFAULT_TABLE table=default.Table_dcacb44eed4241d3acbae153ebaad663 [id=0241aa3ab1984557a151461005c6b97c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:10.181763 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2685fb7258264463b5f1ec99cd7d67ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:10.186964 900 tablet_service.cc:1468] Processing CreateTablet for tablet 2685fb7258264463b5f1ec99cd7d67ad (DEFAULT_TABLE table=default.Table_dcacb44eed4241d3acbae153ebaad663 [id=0241aa3ab1984557a151461005c6b97c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:10.187542 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2685fb7258264463b5f1ec99cd7d67ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:10.194365 755 tablet_service.cc:1468] Processing CreateTablet for tablet 6c15cd14831f422eb615038603c79ed0 (DEFAULT_TABLE table=default.Table_45ba5a130a824accac159d040d1b6a9d [id=b9d81ddd38fe47838877b010be2b40d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:10.194979 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c15cd14831f422eb615038603c79ed0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:10.200925 1176 raft_consensus.cc:513] T c297e8bf11ff467a854eab41117ac376 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:10.201869 1176 leader_election.cc:290] T c297e8bf11ff467a854eab41117ac376 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:10.204169 1034 tablet_service.cc:1468] Processing CreateTablet for tablet 2685fb7258264463b5f1ec99cd7d67ad (DEFAULT_TABLE table=default.Table_dcacb44eed4241d3acbae153ebaad663 [id=0241aa3ab1984557a151461005c6b97c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:10.204770 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2685fb7258264463b5f1ec99cd7d67ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:10.209169 924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c297e8bf11ff467a854eab41117ac376" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:10.209450 924 raft_consensus.cc:2466] T c297e8bf11ff467a854eab41117ac376 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:10.210129 992 leader_election.cc:304] T c297e8bf11ff467a854eab41117ac376 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:10.210561 1176 raft_consensus.cc:2802] T c297e8bf11ff467a854eab41117ac376 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:10.210678 1176 raft_consensus.cc:491] T c297e8bf11ff467a854eab41117ac376 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:10.210808 1176 raft_consensus.cc:3058] T c297e8bf11ff467a854eab41117ac376 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:10.214530 1176 raft_consensus.cc:513] T c297e8bf11ff467a854eab41117ac376 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:10.219414 1151 tablet_bootstrap.cc:492] T c445c300d71643609071f0f36528a9c4 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:10.219617 1151 ts_tablet_manager.cc:1397] T c445c300d71643609071f0f36528a9c4 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.044s user 0.000s sys 0.003s
I20250901 01:52:10.220633 1151 raft_consensus.cc:357] T c445c300d71643609071f0f36528a9c4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:10.220870 1151 raft_consensus.cc:383] T c445c300d71643609071f0f36528a9c4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:10.220964 1151 raft_consensus.cc:738] T c445c300d71643609071f0f36528a9c4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:10.221334 1151 consensus_queue.cc:260] T c445c300d71643609071f0f36528a9c4 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:10.222170 1151 ts_tablet_manager.cc:1428] T c445c300d71643609071f0f36528a9c4 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20250901 01:52:10.222605 1151 tablet_bootstrap.cc:492] T 77405097e68344628f4eda667afaed21 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:10.224484 1151 tablet_bootstrap.cc:654] T 77405097e68344628f4eda667afaed21 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:10.244297 1176 leader_election.cc:290] T c297e8bf11ff467a854eab41117ac376 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:10.244491 924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c297e8bf11ff467a854eab41117ac376" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:10.244751 924 raft_consensus.cc:3058] T c297e8bf11ff467a854eab41117ac376 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:10.248088 924 raft_consensus.cc:2466] T c297e8bf11ff467a854eab41117ac376 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:10.248771 992 leader_election.cc:304] T c297e8bf11ff467a854eab41117ac376 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:10.255081 901 tablet_service.cc:1468] Processing CreateTablet for tablet 6c15cd14831f422eb615038603c79ed0 (DEFAULT_TABLE table=default.Table_45ba5a130a824accac159d040d1b6a9d [id=b9d81ddd38fe47838877b010be2b40d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:10.255654 901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c15cd14831f422eb615038603c79ed0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:10.255970 753 tablet_service.cc:1468] Processing CreateTablet for tablet 0e698923812841b7a5d37d57da103c5c (DEFAULT_TABLE table=default.Table_68682089b6364db98892ca5d82ed1322 [id=83600209eb4649ec9f7842507aa8a1ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:10.256601 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e698923812841b7a5d37d57da103c5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:10.262660 1031 tablet_service.cc:1468] Processing CreateTablet for tablet 0e698923812841b7a5d37d57da103c5c (DEFAULT_TABLE table=default.Table_68682089b6364db98892ca5d82ed1322 [id=83600209eb4649ec9f7842507aa8a1ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:10.263170 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e698923812841b7a5d37d57da103c5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:10.266845 1176 raft_consensus.cc:2802] T c297e8bf11ff467a854eab41117ac376 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:10.267570 1176 raft_consensus.cc:695] T c297e8bf11ff467a854eab41117ac376 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:10.267884 1176 consensus_queue.cc:237] T c297e8bf11ff467a854eab41117ac376 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:10.268934 702 kernel_stack_watchdog.cc:198] Thread 1149 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 261ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fe78f073980 (unknown) at ??:0
@ 0x8921b8 __interceptor_pthread_create at ??:?
@ 0x7fe78bc459c1 kudu::Thread::StartThread() at ??:0
@ 0xa63f97 kudu::Thread::Create() at ??:?
@ 0x7fe78bc6580b kudu::ThreadPool::CreateThread() at ??:0
@ 0x7fe78bc6122d kudu::ThreadPool::DoSubmit() at ??:0
@ 0x7fe78bc6655f kudu::ThreadPool::Submit() at ??:0
@ 0x7fe79701088d kudu::log::SegmentAllocator::AsyncAllocateSegmentUnlocked() at ??:0
@ 0x7fe79700f87f kudu::log::SegmentAllocator::AllocateSegmentAndRollOver() at ??:0
@ 0x7fe79700f43e kudu::log::SegmentAllocator::Init() at ??:0
@ 0x7fe7970174b5 kudu::log::Log::Init() at ??:0
@ 0x7fe797016811 kudu::log::Log::Open() at ??:0
@ 0x7fe79895a5e1 kudu::tablet::TabletBootstrap::OpenNewLog() at ??:0
@ 0x7fe79895694c kudu::tablet::TabletBootstrap::RunBootstrap() at ??:0
@ 0x7fe7989550cf kudu::tablet::TabletBootstrap::Bootstrap() at ??:0
@ 0x7fe7989549ad kudu::tablet::BootstrapTablet() at ??:0
I20250901 01:52:10.272257 1032 tablet_service.cc:1468] Processing CreateTablet for tablet 6c15cd14831f422eb615038603c79ed0 (DEFAULT_TABLE table=default.Table_45ba5a130a824accac159d040d1b6a9d [id=b9d81ddd38fe47838877b010be2b40d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:10.272733 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c15cd14831f422eb615038603c79ed0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:10.280427 901 tablet_service.cc:1468] Processing CreateTablet for tablet 0e698923812841b7a5d37d57da103c5c (DEFAULT_TABLE table=default.Table_68682089b6364db98892ca5d82ed1322 [id=83600209eb4649ec9f7842507aa8a1ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:10.281096 901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e698923812841b7a5d37d57da103c5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:10.282078 902 tablet_service.cc:1468] Processing CreateTablet for tablet 85322a864d154f299f2d670fdd2a7bfb (DEFAULT_TABLE table=default.Table_a8b4fe01e7c54c3e95663bced8b3bbe4 [id=1d8f34cc14cb4e90a562ff4424b8577a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:10.282493 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85322a864d154f299f2d670fdd2a7bfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:10.307823 1244 raft_consensus.cc:491] T c445c300d71643609071f0f36528a9c4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:10.308032 1244 raft_consensus.cc:513] T c445c300d71643609071f0f36528a9c4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:10.308939 1244 leader_election.cc:290] T c445c300d71643609071f0f36528a9c4 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:10.309918 924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c445c300d71643609071f0f36528a9c4" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:10.310168 924 raft_consensus.cc:2466] T c445c300d71643609071f0f36528a9c4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:10.310814 992 leader_election.cc:304] T c445c300d71643609071f0f36528a9c4 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:10.311126 1244 raft_consensus.cc:2802] T c445c300d71643609071f0f36528a9c4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:10.311221 1244 raft_consensus.cc:491] T c445c300d71643609071f0f36528a9c4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:10.311292 1244 raft_consensus.cc:3058] T c445c300d71643609071f0f36528a9c4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:10.314920 1244 raft_consensus.cc:513] T c445c300d71643609071f0f36528a9c4 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:10.315807 1244 leader_election.cc:290] T c445c300d71643609071f0f36528a9c4 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:10.318210 924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c445c300d71643609071f0f36528a9c4" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:10.318528 924 raft_consensus.cc:3058] T c445c300d71643609071f0f36528a9c4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:10.322527 924 raft_consensus.cc:2466] T c445c300d71643609071f0f36528a9c4 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:10.323124 992 leader_election.cc:304] T c445c300d71643609071f0f36528a9c4 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:10.323486 1244 raft_consensus.cc:2802] T c445c300d71643609071f0f36528a9c4 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:10.331656 1031 tablet_service.cc:1468] Processing CreateTablet for tablet 85322a864d154f299f2d670fdd2a7bfb (DEFAULT_TABLE table=default.Table_a8b4fe01e7c54c3e95663bced8b3bbe4 [id=1d8f34cc14cb4e90a562ff4424b8577a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:10.332139 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85322a864d154f299f2d670fdd2a7bfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:10.334219 758 tablet_service.cc:1468] Processing CreateTablet for tablet 7cc1efcb6bff4aba8015d2d3b7f4c923 (DEFAULT_TABLE table=default.Table_1c3a507d45d64e389a6c07341b00b9fc [id=78b00856a9b747c19f390b4fdae8ddcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:10.331676 750 tablet_service.cc:1468] Processing CreateTablet for tablet 85322a864d154f299f2d670fdd2a7bfb (DEFAULT_TABLE table=default.Table_a8b4fe01e7c54c3e95663bced8b3bbe4 [id=1d8f34cc14cb4e90a562ff4424b8577a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:10.336483 1034 tablet_service.cc:1468] Processing CreateTablet for tablet 7cc1efcb6bff4aba8015d2d3b7f4c923 (DEFAULT_TABLE table=default.Table_1c3a507d45d64e389a6c07341b00b9fc [id=78b00856a9b747c19f390b4fdae8ddcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:10.338675 450 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_68682089b6364db98892ca5d82ed1322" } modify_external_catalogs: true
I20250901 01:52:10.338907 450 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_68682089b6364db98892ca5d82ed1322" } modify_external_catalogs: true
I20250901 01:52:10.342525 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7cc1efcb6bff4aba8015d2d3b7f4c923. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:10.345817 900 tablet_service.cc:1468] Processing CreateTablet for tablet 7cc1efcb6bff4aba8015d2d3b7f4c923 (DEFAULT_TABLE table=default.Table_1c3a507d45d64e389a6c07341b00b9fc [id=78b00856a9b747c19f390b4fdae8ddcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:10.346366 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7cc1efcb6bff4aba8015d2d3b7f4c923. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:10.334681 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7cc1efcb6bff4aba8015d2d3b7f4c923. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:10.350018 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85322a864d154f299f2d670fdd2a7bfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:10.352859 1150 tablet_bootstrap.cc:492] T 342b7fde3ede494b9eec6e361db255dc P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:10.353034 1150 ts_tablet_manager.cc:1397] T 342b7fde3ede494b9eec6e361db255dc P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.284s user 0.010s sys 0.076s
I20250901 01:52:10.353274 755 tablet_service.cc:1468] Processing CreateTablet for tablet d6a4040747fa4d588756506ee122d2dc (DEFAULT_TABLE table=default.Table_6400e401888045348632f924f636d011 [id=8c1b263824ec4c15a5ec424364c78895]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:10.353818 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6a4040747fa4d588756506ee122d2dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:10.353956 1150 raft_consensus.cc:357] T 342b7fde3ede494b9eec6e361db255dc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } }
I20250901 01:52:10.354133 1150 raft_consensus.cc:383] T 342b7fde3ede494b9eec6e361db255dc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:10.354218 1150 raft_consensus.cc:738] T 342b7fde3ede494b9eec6e361db255dc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:10.354444 1150 consensus_queue.cc:260] T 342b7fde3ede494b9eec6e361db255dc P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } }
I20250901 01:52:10.354792 1244 raft_consensus.cc:695] T c445c300d71643609071f0f36528a9c4 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:10.355101 1244 consensus_queue.cc:237] T c445c300d71643609071f0f36528a9c4 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:10.354964 1150 ts_tablet_manager.cc:1428] T 342b7fde3ede494b9eec6e361db255dc P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250901 01:52:10.360837 1150 tablet_bootstrap.cc:492] T eeae5294bd474f368f5766c260ccf3fe P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:10.363488 1031 tablet_service.cc:1468] Processing CreateTablet for tablet d6a4040747fa4d588756506ee122d2dc (DEFAULT_TABLE table=default.Table_6400e401888045348632f924f636d011 [id=8c1b263824ec4c15a5ec424364c78895]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:10.363946 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6a4040747fa4d588756506ee122d2dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:10.364683 753 tablet_service.cc:1468] Processing CreateTablet for tablet 88a6ab29d00542aba54f0e4185fbf615 (DEFAULT_TABLE table=default.Table_aa569ededbee4b1bb3c3a2f20f9748cb [id=1d0bc46740444f1fbf2129270e806259]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:10.365099 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88a6ab29d00542aba54f0e4185fbf615. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:10.367063 1038 tablet_service.cc:1468] Processing CreateTablet for tablet 88a6ab29d00542aba54f0e4185fbf615 (DEFAULT_TABLE table=default.Table_aa569ededbee4b1bb3c3a2f20f9748cb [id=1d0bc46740444f1fbf2129270e806259]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:10.367449 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88a6ab29d00542aba54f0e4185fbf615. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:10.376309 453 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_e6bab04ba03d4d9fa029e550a887e303" } modify_external_catalogs: true
I20250901 01:52:10.376574 453 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_e6bab04ba03d4d9fa029e550a887e303" } modify_external_catalogs: true
I20250901 01:52:10.388320 751 tablet_service.cc:1468] Processing CreateTablet for tablet 84fe314502444bd88b038b87eead9cb5 (DEFAULT_TABLE table=default.Table_6a5316d194864096bdfdf11e382f69d2 [id=c57384464f9f44f797e98c309b6866f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:10.388952 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84fe314502444bd88b038b87eead9cb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:10.391449 1031 tablet_service.cc:1468] Processing CreateTablet for tablet 84fe314502444bd88b038b87eead9cb5 (DEFAULT_TABLE table=default.Table_6a5316d194864096bdfdf11e382f69d2 [id=c57384464f9f44f797e98c309b6866f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:10.391876 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84fe314502444bd88b038b87eead9cb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:10.392251 760 tablet_service.cc:1468] Processing CreateTablet for tablet 9496ec6021784800aef62996b0dee593 (DEFAULT_TABLE table=default.Table_95c28a4758e148368604152622fe51be [id=bbcca84b801942d094d955c87cafacb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:10.392657 760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9496ec6021784800aef62996b0dee593. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:10.394460 1038 tablet_service.cc:1468] Processing CreateTablet for tablet 9496ec6021784800aef62996b0dee593 (DEFAULT_TABLE table=default.Table_95c28a4758e148368604152622fe51be [id=bbcca84b801942d094d955c87cafacb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:10.394832 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9496ec6021784800aef62996b0dee593. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:10.395390 754 tablet_service.cc:1468] Processing CreateTablet for tablet 756cfe84a517459ca015aecf5cd81571 (DEFAULT_TABLE table=default.Table_b3ec3b285fac447f8c5c4a9ac4196628 [id=f49c96ff2eaa44cc9b754225cb9e6fdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:10.395740 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 756cfe84a517459ca015aecf5cd81571. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:10.398152 752 tablet_service.cc:1468] Processing CreateTablet for tablet 31390f38258c465cb1798615a1fcc790 (DEFAULT_TABLE table=default.Table_1a83c4b1e2734d038d0309401d0ab08e [id=c1902e6ac21b4a8a8af96d647f85e288]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:10.398463 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31390f38258c465cb1798615a1fcc790. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:10.400815 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 756cfe84a517459ca015aecf5cd81571 (DEFAULT_TABLE table=default.Table_b3ec3b285fac447f8c5c4a9ac4196628 [id=f49c96ff2eaa44cc9b754225cb9e6fdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:10.401222 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 756cfe84a517459ca015aecf5cd81571. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:10.410949 901 tablet_service.cc:1468] Processing CreateTablet for tablet 88a6ab29d00542aba54f0e4185fbf615 (DEFAULT_TABLE table=default.Table_aa569ededbee4b1bb3c3a2f20f9748cb [id=1d0bc46740444f1fbf2129270e806259]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:10.438619 1244 raft_consensus.cc:491] T 288e8029382f4154bc1e2103b3035a60 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:10.438870 1244 raft_consensus.cc:513] T 288e8029382f4154bc1e2103b3035a60 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:10.439797 1244 leader_election.cc:290] T 288e8029382f4154bc1e2103b3035a60 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:10.452492 901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88a6ab29d00542aba54f0e4185fbf615. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:10.457409 1158 raft_consensus.cc:491] T 4cdd369214ef4f5ebabddbf28b2ae3f7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:10.460798 895 tablet_service.cc:1468] Processing CreateTablet for tablet 84fe314502444bd88b038b87eead9cb5 (DEFAULT_TABLE table=default.Table_6a5316d194864096bdfdf11e382f69d2 [id=c57384464f9f44f797e98c309b6866f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:10.461069 903 tablet_service.cc:1468] Processing CreateTablet for tablet 9496ec6021784800aef62996b0dee593 (DEFAULT_TABLE table=default.Table_95c28a4758e148368604152622fe51be [id=bbcca84b801942d094d955c87cafacb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:10.461308 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84fe314502444bd88b038b87eead9cb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:10.461594 903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9496ec6021784800aef62996b0dee593. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:10.466913 899 tablet_service.cc:1468] Processing CreateTablet for tablet 756cfe84a517459ca015aecf5cd81571 (DEFAULT_TABLE table=default.Table_b3ec3b285fac447f8c5c4a9ac4196628 [id=f49c96ff2eaa44cc9b754225cb9e6fdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:10.464535 1158 raft_consensus.cc:513] T 4cdd369214ef4f5ebabddbf28b2ae3f7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:10.466640 902 tablet_service.cc:1468] Processing CreateTablet for tablet d6a4040747fa4d588756506ee122d2dc (DEFAULT_TABLE table=default.Table_6400e401888045348632f924f636d011 [id=8c1b263824ec4c15a5ec424364c78895]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:10.471413 1151 tablet_bootstrap.cc:492] T 77405097e68344628f4eda667afaed21 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:10.471661 1151 ts_tablet_manager.cc:1397] T 77405097e68344628f4eda667afaed21 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.249s user 0.022s sys 0.051s
I20250901 01:52:10.472594 1151 raft_consensus.cc:357] T 77405097e68344628f4eda667afaed21 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:10.472852 1151 raft_consensus.cc:383] T 77405097e68344628f4eda667afaed21 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:10.472929 1151 raft_consensus.cc:738] T 77405097e68344628f4eda667afaed21 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:10.472975 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 756cfe84a517459ca015aecf5cd81571. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:10.473330 1151 consensus_queue.cc:260] T 77405097e68344628f4eda667afaed21 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:10.475630 1151 ts_tablet_manager.cc:1428] T 77405097e68344628f4eda667afaed21 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20250901 01:52:10.476912 1151 tablet_bootstrap.cc:492] T 8d5b10ede9c84be098795c0a787e5a78 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:10.483206 897 tablet_service.cc:1468] Processing CreateTablet for tablet 31390f38258c465cb1798615a1fcc790 (DEFAULT_TABLE table=default.Table_1a83c4b1e2734d038d0309401d0ab08e [id=c1902e6ac21b4a8a8af96d647f85e288]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:10.485100 1151 tablet_bootstrap.cc:654] T 8d5b10ede9c84be098795c0a787e5a78 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:10.488293 1158 leader_election.cc:290] T 4cdd369214ef4f5ebabddbf28b2ae3f7 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:10.488709 1158 raft_consensus.cc:491] T 4d333b670df84f308f0cbf6da636b768 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:10.488905 1158 raft_consensus.cc:513] T 4d333b670df84f308f0cbf6da636b768 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } }
I20250901 01:52:10.489342 1151 tablet_bootstrap.cc:492] T 8d5b10ede9c84be098795c0a787e5a78 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:10.489522 1151 ts_tablet_manager.cc:1397] T 8d5b10ede9c84be098795c0a787e5a78 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.002s
I20250901 01:52:10.489807 1158 leader_election.cc:290] T 4d333b670df84f308f0cbf6da636b768 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447), 0cf599f74322478d91475df234e58965 (127.0.104.1:40731)
I20250901 01:52:10.490022 1158 raft_consensus.cc:491] T 77405097e68344628f4eda667afaed21 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:10.490178 1158 raft_consensus.cc:513] T 77405097e68344628f4eda667afaed21 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:10.490605 1151 raft_consensus.cc:357] T 8d5b10ede9c84be098795c0a787e5a78 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:10.490830 1151 raft_consensus.cc:383] T 8d5b10ede9c84be098795c0a787e5a78 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:10.490813 1158 leader_election.cc:290] T 77405097e68344628f4eda667afaed21 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:10.488523 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31390f38258c465cb1798615a1fcc790. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:10.490939 1151 raft_consensus.cc:738] T 8d5b10ede9c84be098795c0a787e5a78 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:10.491218 1151 consensus_queue.cc:260] T 8d5b10ede9c84be098795c0a787e5a78 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:10.492379 924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "288e8029382f4154bc1e2103b3035a60" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:10.492642 924 raft_consensus.cc:2466] T 288e8029382f4154bc1e2103b3035a60 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:10.498412 992 leader_election.cc:304] T 288e8029382f4154bc1e2103b3035a60 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:10.498383 1244 consensus_queue.cc:1035] T ce2a1ba75d674c95bdba9d6f12273bee P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:10.498831 1288 raft_consensus.cc:2802] T 288e8029382f4154bc1e2103b3035a60 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:10.498921 1288 raft_consensus.cc:491] T 288e8029382f4154bc1e2103b3035a60 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:10.499042 1288 raft_consensus.cc:3058] T 288e8029382f4154bc1e2103b3035a60 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:10.502045 1151 ts_tablet_manager.cc:1428] T 8d5b10ede9c84be098795c0a787e5a78 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.012s user 0.001s sys 0.002s
I20250901 01:52:10.502509 1288 raft_consensus.cc:513] T 288e8029382f4154bc1e2103b3035a60 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:10.503273 1151 tablet_bootstrap.cc:492] T 97e40c318f64455d91b0e4f8866990ce P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:10.503473 1288 leader_election.cc:290] T 288e8029382f4154bc1e2103b3035a60 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:10.509146 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6a4040747fa4d588756506ee122d2dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:10.509447 1151 tablet_bootstrap.cc:654] T 97e40c318f64455d91b0e4f8866990ce P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:10.509485 1031 tablet_service.cc:1468] Processing CreateTablet for tablet 31390f38258c465cb1798615a1fcc790 (DEFAULT_TABLE table=default.Table_1a83c4b1e2734d038d0309401d0ab08e [id=c1902e6ac21b4a8a8af96d647f85e288]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:10.509936 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31390f38258c465cb1798615a1fcc790. 1 dirs total, 0 dirs full, 0 dirs failed
W20250901 01:52:10.509891 442 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.60:44231: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20250901 01:52:10.520265 1151 tablet_bootstrap.cc:492] T 97e40c318f64455d91b0e4f8866990ce P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:10.520506 1151 ts_tablet_manager.cc:1397] T 97e40c318f64455d91b0e4f8866990ce P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.002s
I20250901 01:52:10.521528 1031 tablet_service.cc:1468] Processing CreateTablet for tablet 22691943a134400c8df4617240e5d3ef (DEFAULT_TABLE table=default.Table_dd24f0428569419fad1bbe8a7f9f08c1 [id=5a33cfdf60684867bb5f36d84ee80468]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:10.521611 1151 raft_consensus.cc:357] T 97e40c318f64455d91b0e4f8866990ce P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:10.521853 1151 raft_consensus.cc:383] T 97e40c318f64455d91b0e4f8866990ce P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:10.521952 1151 raft_consensus.cc:738] T 97e40c318f64455d91b0e4f8866990ce P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:10.521986 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22691943a134400c8df4617240e5d3ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:10.534009 1151 consensus_queue.cc:260] T 97e40c318f64455d91b0e4f8866990ce P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:10.536259 1151 ts_tablet_manager.cc:1428] T 97e40c318f64455d91b0e4f8866990ce P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20250901 01:52:10.537420 1151 tablet_bootstrap.cc:492] T bf302e76746b4f7bbe4469914f83720f P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:10.542491 1151 tablet_bootstrap.cc:654] T bf302e76746b4f7bbe4469914f83720f P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:10.549892 757 tablet_service.cc:1468] Processing CreateTablet for tablet 22691943a134400c8df4617240e5d3ef (DEFAULT_TABLE table=default.Table_dd24f0428569419fad1bbe8a7f9f08c1 [id=5a33cfdf60684867bb5f36d84ee80468]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:10.550786 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22691943a134400c8df4617240e5d3ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:10.552772 1151 tablet_bootstrap.cc:492] T bf302e76746b4f7bbe4469914f83720f P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:10.552975 1151 ts_tablet_manager.cc:1397] T bf302e76746b4f7bbe4469914f83720f P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.002s
I20250901 01:52:10.554080 1151 raft_consensus.cc:357] T bf302e76746b4f7bbe4469914f83720f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:10.554337 1151 raft_consensus.cc:383] T bf302e76746b4f7bbe4469914f83720f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:10.554456 1151 raft_consensus.cc:738] T bf302e76746b4f7bbe4469914f83720f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:10.562009 1151 consensus_queue.cc:260] T bf302e76746b4f7bbe4469914f83720f P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:10.564703 1151 ts_tablet_manager.cc:1428] T bf302e76746b4f7bbe4469914f83720f P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.012s user 0.003s sys 0.000s
I20250901 01:52:10.566830 1151 tablet_bootstrap.cc:492] T 24ddc13d1bb04db4a7850acbff091577 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:10.569293 1151 tablet_bootstrap.cc:654] T 24ddc13d1bb04db4a7850acbff091577 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:10.635789 442 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.60:44231, user_credentials={real_user=slave}} blocked reactor thread for 125834us
I20250901 01:52:10.641654 1288 raft_consensus.cc:491] T 061b760d8e6647c6ab0b5d538840a230 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:10.641860 1288 raft_consensus.cc:513] T 061b760d8e6647c6ab0b5d538840a230 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:10.642590 1288 leader_election.cc:290] T 061b760d8e6647c6ab0b5d538840a230 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:10.643275 1150 tablet_bootstrap.cc:654] T eeae5294bd474f368f5766c260ccf3fe P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:10.544512 923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "288e8029382f4154bc1e2103b3035a60" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:10.654906 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 2a0d125423a043dcb11e67345599e468
I20250901 01:52:10.655455 455 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:57810 (request call id 114) took 1097 ms. Trace:
I20250901 01:52:10.656224 901 tablet_service.cc:1515] Processing DeleteTablet for tablet 2a0d125423a043dcb11e67345599e468 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:09 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:10.656716 446 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:57810 (request call id 118) took 1022 ms. Trace:
I20250901 01:52:10.656847 446 rpcz_store.cc:276] 0901 01:52:09.633889 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:09.889261 (+255372us) service_pool.cc:225] Handling call
0901 01:52:09.923385 (+ 34124us) spinlock_profiling.cc:243] Waited 33.6 ms on lock 0x616000040f54. stack: 00007fc53edd57d6 00007fc53edd5590 00007fc53ddc83ad 0000000000972d94 0000000000972d5d 00007fc54dba014d 00007fc54dba0243 00007fc54a85f941 00007fc5523eaa5d 00007fc552580bdb 00007fc5493b80e7 00007fc5493b7e15 00007fc545b832ce 00007fc545b821ae 00007fc545b883e5 00007fc545b88d0f
0901 01:52:10.656672 (+733287us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":77366144}
I20250901 01:52:10.655615 455 rpcz_store.cc:276] 0901 01:52:09.557630 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:09.784560 (+226930us) service_pool.cc:225] Handling call
0901 01:52:09.785048 (+ 488us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0901 01:52:09.785203 (+ 155us) catalog_manager.cc:2790] Modifying in-memory table state
0901 01:52:09.785278 (+ 75us) catalog_manager.cc:2799] Locking tablets
0901 01:52:10.152235 (+366957us) catalog_manager.cc:2813] Removing table and tablets from system table
0901 01:52:10.654683 (+502448us) catalog_manager.cc:2831] Removing table from by-name map
0901 01:52:10.654786 (+ 103us) catalog_manager.cc:2847] Committing in-memory state
0901 01:52:10.654830 (+ 44us) catalog_manager.cc:2851] Aborting table tasks
0901 01:52:10.655408 (+ 578us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0901 01:52:10.495675 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:10.496062 (+ 387us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:10.496071 (+ 9us) write_op.cc:435] Acquired schema lock
0901 01:52:10.496076 (+ 5us) tablet.cc:660] Decoding operations
0901 01:52:10.496463 (+ 387us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:10.496485 (+ 22us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:10.496523 (+ 38us) tablet.cc:683] Acquiring locks for 2 operations
0901 01:52:10.496829 (+ 306us) tablet.cc:699] Row locks acquired
0901 01:52:10.496834 (+ 5us) write_op.cc:260] PREPARE: finished
0901 01:52:10.497100 (+ 266us) write_op.cc:270] Start()
0901 01:52:10.508164 (+ 11064us) write_op.cc:276] Timestamp: P: 1756691530496956 usec, L: 0
0901 01:52:10.508170 (+ 6us) op_driver.cc:348] REPLICATION: starting
0901 01:52:10.517875 (+ 9705us) log.cc:844] Serialized 725 byte log entry
0901 01:52:10.653458 (+135583us) op_driver.cc:464] REPLICATION: finished
0901 01:52:10.653707 (+ 249us) write_op.cc:301] APPLY: starting
0901 01:52:10.653725 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:10.653775 (+ 50us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:10.653801 (+ 26us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:10.653910 (+ 109us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:10.653965 (+ 55us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0901 01:52:10.653973 (+ 8us) write_op.cc:312] APPLY: finished
0901 01:52:10.654073 (+ 100us) log.cc:844] Serialized 29 byte log entry
0901 01:52:10.654152 (+ 79us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:10.654178 (+ 26us) write_op.cc:454] Released schema lock
0901 01:52:10.654186 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0901 01:52:10.654262 (+ 76us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:10.654265 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":185,"apply.run_cpu_time_us":550,"apply.run_wall_time_us":549,"num_ops":2,"prepare.queue_time_us":342332,"prepare.run_cpu_time_us":1549,"prepare.run_wall_time_us":22945,"replication_time_us":145244}]]}
I20250901 01:52:10.658021 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet 2a0d125423a043dcb11e67345599e468 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:09 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:10.658458 923 raft_consensus.cc:3058] T 288e8029382f4154bc1e2103b3035a60 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:10.662336 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 2a0d125423a043dcb11e67345599e468 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:09 UTC) from {username='slave'} at 127.0.0.1:36178
I20250901 01:52:10.660480 899 tablet_service.cc:1468] Processing CreateTablet for tablet 22691943a134400c8df4617240e5d3ef (DEFAULT_TABLE table=default.Table_dd24f0428569419fad1bbe8a7f9f08c1 [id=5a33cfdf60684867bb5f36d84ee80468]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:10.663800 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22691943a134400c8df4617240e5d3ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:10.664109 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 360755d30b5d47ba98c9c7e23ff5ab7e
I20250901 01:52:10.665668 1034 tablet_service.cc:1515] Processing DeleteTablet for tablet 360755d30b5d47ba98c9c7e23ff5ab7e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:10 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:10.666442 760 tablet_service.cc:1515] Processing DeleteTablet for tablet 360755d30b5d47ba98c9c7e23ff5ab7e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:10 UTC) from {username='slave'} at 127.0.0.1:36178
I20250901 01:52:10.660522 922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "061b760d8e6647c6ab0b5d538840a230" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:10.668473 922 raft_consensus.cc:2466] T 061b760d8e6647c6ab0b5d538840a230 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:10.669225 992 leader_election.cc:304] T 061b760d8e6647c6ab0b5d538840a230 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:10.669633 1288 raft_consensus.cc:2802] T 061b760d8e6647c6ab0b5d538840a230 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:10.669731 1288 raft_consensus.cc:491] T 061b760d8e6647c6ab0b5d538840a230 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:10.669848 1288 raft_consensus.cc:3058] T 061b760d8e6647c6ab0b5d538840a230 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:10.673524 1288 raft_consensus.cc:513] T 061b760d8e6647c6ab0b5d538840a230 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:10.674914 1288 leader_election.cc:290] T 061b760d8e6647c6ab0b5d538840a230 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:10.684585 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 360755d30b5d47ba98c9c7e23ff5ab7e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:10 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:10.685107 922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "061b760d8e6647c6ab0b5d538840a230" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:10.685369 922 raft_consensus.cc:3058] T 061b760d8e6647c6ab0b5d538840a230 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:10.687032 923 raft_consensus.cc:2466] T 288e8029382f4154bc1e2103b3035a60 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:10.688047 992 leader_election.cc:304] T 288e8029382f4154bc1e2103b3035a60 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:10.688527 1288 raft_consensus.cc:2802] T 288e8029382f4154bc1e2103b3035a60 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:10.689006 1288 raft_consensus.cc:695] T 288e8029382f4154bc1e2103b3035a60 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:10.690241 1288 consensus_queue.cc:237] T 288e8029382f4154bc1e2103b3035a60 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
W20250901 01:52:10.693245 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 2a0d125423a043dcb11e67345599e468 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a0d125423a043dcb11e67345599e468 already in progress: creating tablet
W20250901 01:52:10.693885 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 360755d30b5d47ba98c9c7e23ff5ab7e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 360755d30b5d47ba98c9c7e23ff5ab7e already in progress: creating tablet
W20250901 01:52:10.697021 448 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.3:47125 (request call id 11) took 799 ms (client timeout 1000 ms). Trace:
W20250901 01:52:10.697166 448 rpcz_store.cc:269] 0901 01:52:09.897206 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:09.917260 (+ 20054us) service_pool.cc:225] Handling call
0901 01:52:09.922688 (+ 5428us) spinlock_profiling.cc:243] Waited 4.98 ms on lock 0x616000040f54. stack: 00007fc53edd57d6 00007fc53edd5590 00007fc53ddc83ad 0000000000972d94 0000000000972d5d 00007fc54dba014d 00007fc54dba0243 00007fc54a85f941 00007fc5523eaa5d 00007fc55256e6d8 00007fc5493a8c86 00007fc5493a89b5 00007fc545b832ce 00007fc545b821ae 00007fc545b883e5 00007fc545b88d0f
0901 01:52:10.696969 (+774281us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0901 01:52:10.524588 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:10.524701 (+ 113us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:10.524708 (+ 7us) write_op.cc:435] Acquired schema lock
0901 01:52:10.524710 (+ 2us) tablet.cc:660] Decoding operations
0901 01:52:10.524831 (+ 121us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:10.524845 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:10.524849 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0901 01:52:10.524918 (+ 69us) tablet.cc:699] Row locks acquired
0901 01:52:10.524920 (+ 2us) write_op.cc:260] PREPARE: finished
0901 01:52:10.524962 (+ 42us) write_op.cc:270] Start()
0901 01:52:10.524998 (+ 36us) write_op.cc:276] Timestamp: P: 1756691530524939 usec, L: 0
0901 01:52:10.525002 (+ 4us) op_driver.cc:348] REPLICATION: starting
0901 01:52:10.525192 (+ 190us) log.cc:844] Serialized 527 byte log entry
0901 01:52:10.695688 (+170496us) op_driver.cc:464] REPLICATION: finished
0901 01:52:10.695819 (+ 131us) write_op.cc:301] APPLY: starting
0901 01:52:10.695834 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:10.695878 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:10.695925 (+ 47us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:10.695977 (+ 52us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:10.696007 (+ 30us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0901 01:52:10.696014 (+ 7us) write_op.cc:312] APPLY: finished
0901 01:52:10.696097 (+ 83us) log.cc:844] Serialized 23 byte log entry
0901 01:52:10.696155 (+ 58us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:10.696173 (+ 18us) write_op.cc:454] Released schema lock
0901 01:52:10.696179 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0901 01:52:10.696250 (+ 71us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:10.696253 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":2,"spinlock_wait_cycles":11456896,"child_traces":[["op",{"apply.queue_time_us":80,"apply.run_cpu_time_us":424,"apply.run_wall_time_us":424,"num_ops":1,"prepare.queue_time_us":368450,"prepare.run_cpu_time_us":705,"prepare.run_wall_time_us":2070,"replication_time_us":170656}]]}
I20250901 01:52:10.697934 448 master_service.cc:948] ReplaceTablet: received request to replace tablet 77405097e68344628f4eda667afaed21 from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:10.698380 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 0e698923812841b7a5d37d57da103c5c
I20250901 01:52:10.698683 452 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet c297e8bf11ff467a854eab41117ac376
I20250901 01:52:10.699055 452 catalog_manager.cc:6652] ReplaceTablet: tablet c297e8bf11ff467a854eab41117ac376 deleted and replaced by tablet 186ec810d02e4748aad15228b5866ac1
I20250901 01:52:11.201259 1329 raft_consensus.cc:491] T 00000000000000000000000000000000 P d2ae2cb6af0f4b3a8f47ab1cdd6567d3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e3add7edb97a40ef9f7c6bc011d13227)
I20250901 01:52:11.225450 1329 raft_consensus.cc:513] T 00000000000000000000000000000000 P d2ae2cb6af0f4b3a8f47ab1cdd6567d3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3add7edb97a40ef9f7c6bc011d13227" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 43871 } } peers { permanent_uuid: "d2ae2cb6af0f4b3a8f47ab1cdd6567d3" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42375 } } peers { permanent_uuid: "7ce2288f784044eea010664e3def133f" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 44231 } }
I20250901 01:52:11.226351 1329 leader_election.cc:290] T 00000000000000000000000000000000 P d2ae2cb6af0f4b3a8f47ab1cdd6567d3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e3add7edb97a40ef9f7c6bc011d13227 (127.0.104.62:43871), 7ce2288f784044eea010664e3def133f (127.0.104.60:44231)
I20250901 01:52:10.709952 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 0e698923812841b7a5d37d57da103c5c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:10 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:10.710332 903 tablet_service.cc:1515] Processing DeleteTablet for tablet c297e8bf11ff467a854eab41117ac376 with delete_type TABLET_DATA_DELETED (replaced by tablet 186ec810d02e4748aad15228b5866ac1) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:10.706763 752 tablet_service.cc:1515] Processing DeleteTablet for tablet c297e8bf11ff467a854eab41117ac376 with delete_type TABLET_DATA_DELETED (replaced by tablet 186ec810d02e4748aad15228b5866ac1) from {username='slave'} at 127.0.0.1:36178
I20250901 01:52:10.715745 922 raft_consensus.cc:2466] T 061b760d8e6647c6ab0b5d538840a230 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:10.703611 1034 tablet_service.cc:1515] Processing DeleteTablet for tablet 0e698923812841b7a5d37d57da103c5c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:10 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:10.699520 453 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 342b7fde3ede494b9eec6e361db255dc
I20250901 01:52:11.228433 1306 tablet_replica.cc:331] T c297e8bf11ff467a854eab41117ac376 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:11.228643 1306 raft_consensus.cc:2241] T c297e8bf11ff467a854eab41117ac376 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:11.228902 1306 raft_consensus.cc:2270] T c297e8bf11ff467a854eab41117ac376 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:11.229074 453 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:57810 (request call id 132) took 1226 ms. Trace:
I20250901 01:52:11.229453 1306 ts_tablet_manager.cc:1905] T c297e8bf11ff467a854eab41117ac376 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:11.229204 453 rpcz_store.cc:276] 0901 01:52:10.002475 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:10.374304 (+371829us) service_pool.cc:225] Handling call
0901 01:52:10.376252 (+ 1948us) spinlock_profiling.cc:243] Waited 1.73 ms on lock 0x616000040f54. stack: 00007fc53edd57d6 00007fc53edd5590 00007fc53ddc83ad 0000000000972d94 0000000000972d5d 00007fc54dba014d 00007fc54dba0243 00007fc54a85f941 00007fc5523eaa5d 00007fc552574e86 00007fc5493ad466 00007fc5493ad195 00007fc545b832ce 00007fc545b821ae 00007fc545b883e5 00007fc545b88d0f
0901 01:52:10.376689 (+ 437us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0901 01:52:10.376817 (+ 128us) catalog_manager.cc:2790] Modifying in-memory table state
0901 01:52:10.376881 (+ 64us) catalog_manager.cc:2799] Locking tablets
0901 01:52:10.377020 (+ 139us) catalog_manager.cc:2813] Removing table and tablets from system table
0901 01:52:10.699371 (+322351us) catalog_manager.cc:2831] Removing table from by-name map
0901 01:52:10.699441 (+ 70us) catalog_manager.cc:2847] Committing in-memory state
0901 01:52:10.699479 (+ 38us) catalog_manager.cc:2851] Aborting table tasks
0901 01:52:11.229004 (+529525us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0901 01:52:10.529923 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:10.530009 (+ 86us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:10.530014 (+ 5us) write_op.cc:435] Acquired schema lock
0901 01:52:10.530016 (+ 2us) tablet.cc:660] Decoding operations
0901 01:52:10.530144 (+ 128us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:10.530151 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:10.530156 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0901 01:52:10.530238 (+ 82us) tablet.cc:699] Row locks acquired
0901 01:52:10.530241 (+ 3us) write_op.cc:260] PREPARE: finished
0901 01:52:10.530265 (+ 24us) write_op.cc:270] Start()
0901 01:52:10.530287 (+ 22us) write_op.cc:276] Timestamp: P: 1756691530530246 usec, L: 0
0901 01:52:10.530292 (+ 5us) op_driver.cc:348] REPLICATION: starting
0901 01:52:10.530431 (+ 139us) log.cc:844] Serialized 725 byte log entry
0901 01:52:10.697275 (+166844us) op_driver.cc:464] REPLICATION: finished
0901 01:52:10.697791 (+ 516us) write_op.cc:301] APPLY: starting
0901 01:52:10.697801 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:10.697829 (+ 28us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:10.697847 (+ 18us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:10.697897 (+ 50us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:10.697930 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0901 01:52:10.697937 (+ 7us) write_op.cc:312] APPLY: finished
0901 01:52:10.697989 (+ 52us) log.cc:844] Serialized 29 byte log entry
0901 01:52:10.698026 (+ 37us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:10.698040 (+ 14us) write_op.cc:454] Released schema lock
0901 01:52:10.698044 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":3982592,"child_traces":[["op",{"apply.queue_time_us":484,"apply.run_cpu_time_us":301,"apply.run_wall_time_us":300,"num_ops":2,"prepare.queue_time_us":152109,"prepare.run_cpu_time_us":575,"prepare.run_wall_time_us":1511,"replication_time_us":166968}]]}
I20250901 01:52:10.700250 455 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_16b3ac9fb2304c8095e0270038501db0" } new_table_name: "default.Table_3c7c628f35fb4d02bb79a34995f12938" modify_external_catalogs: true
I20250901 01:52:10.707170 760 tablet_service.cc:1515] Processing DeleteTablet for tablet 0e698923812841b7a5d37d57da103c5c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:10 UTC) from {username='slave'} at 127.0.0.1:36178
I20250901 01:52:10.700850 454 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_1ddaf993eaa54728ad30f6001b6f9890" } new_table_name: "default.Table_420b0979d6fd42da9b320456f4ccc9ce" modify_external_catalogs: true
W20250901 01:52:11.002732 702 kernel_stack_watchdog.cc:198] Thread 716 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 1172ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fe78f073980 (unknown) at ??:0
@ 0x92ce64 __sanitizer::BlockingMutex::Lock() at ??:?
@ 0x932e1b __sanitizer::ThreadRegistry::CreateThread() at ??:?
@ 0x91983b __asan::AsanThread::Create() at ??:?
@ 0x8921ac __interceptor_pthread_create at ??:?
@ 0x7fe78bc459c1 kudu::Thread::StartThread() at ??:0
@ 0xa63f97 kudu::Thread::Create() at ??:?
@ 0x7fe78bc6580b kudu::ThreadPool::CreateThread() at ??:0
@ 0x7fe78bc6122d kudu::ThreadPool::DoSubmit() at ??:0
@ 0x7fe78bc6655f kudu::ThreadPool::Submit() at ??:0
@ 0x7fe792947bb2 kudu::rpc::ReactorThread::StartConnectionNegotiation() at ??:0
@ 0x7fe7929492d1 kudu::rpc::ReactorThread::FindOrStartConnection() at ??:0
@ 0x7fe792948ad2 kudu::rpc::ReactorThread::AssignOutboundCall() at ??:0
@ 0x7fe79295842e kudu::rpc::AssignOutboundCallTask::Run() at ??:0
@ 0x7fe792946bcd kudu::rpc::ReactorThread::AsyncHandler() at ??:0
@ 0x7fe792958f45 ev::base<>::method_thunk<>() at ??:0
I20250901 01:52:10.725494 1150 tablet_bootstrap.cc:492] T eeae5294bd474f368f5766c260ccf3fe P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:11.233760 1150 ts_tablet_manager.cc:1397] T eeae5294bd474f368f5766c260ccf3fe P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.873s user 0.002s sys 0.008s
I20250901 01:52:10.701483 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_de8ae42a0de043129ceab6caec04d192"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:11.234853 1150 raft_consensus.cc:357] T eeae5294bd474f368f5766c260ccf3fe P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } }
W20250901 01:52:11.234953 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_de8ae42a0de043129ceab6caec04d192 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:11.235107 1150 raft_consensus.cc:383] T eeae5294bd474f368f5766c260ccf3fe P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:11.235190 1150 raft_consensus.cc:738] T eeae5294bd474f368f5766c260ccf3fe P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:11.235533 1150 consensus_queue.cc:260] T eeae5294bd474f368f5766c260ccf3fe P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } }
I20250901 01:52:11.236218 1150 ts_tablet_manager.cc:1428] T eeae5294bd474f368f5766c260ccf3fe P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250901 01:52:10.702411 451 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 061b760d8e6647c6ab0b5d538840a230 (table default.Table_fbba493de02249499d26467bf15c3d5f [id=748daa63137c48a5b151bd34dc5b13fe]). Expected version 1 got 0
I20250901 01:52:10.755582 1292 raft_consensus.cc:491] T 24ddc13d1bb04db4a7850acbff091577 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:10.784051 1293 raft_consensus.cc:491] T 8d5b10ede9c84be098795c0a787e5a78 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:11.236536 1292 raft_consensus.cc:513] T 24ddc13d1bb04db4a7850acbff091577 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } }
I20250901 01:52:11.236635 451 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 8d5b10ede9c84be098795c0a787e5a78 (table default.Table_472dac9061664ce0afb468db1d8fa1de [id=017c724f98ec404bab6e882ead9b8541]). Expected version 1 got 0
I20250901 01:52:11.236538 1293 raft_consensus.cc:513] T 8d5b10ede9c84be098795c0a787e5a78 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:11.236986 451 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 97e40c318f64455d91b0e4f8866990ce (table default.Table_0c6a1616a7b64455a3ae44c9983dc6c0 [id=9af016a062a64218ad3b62e131d8b65d]). Expected version 2 got 0
I20250901 01:52:11.237355 451 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 24ddc13d1bb04db4a7850acbff091577 (table default.Table_8c1924d21a91425cbbbd8429d56cf0ab [id=18aecbf3fe4643c4b589ee47706161bd]). Expected version 2 got 0
I20250901 01:52:11.237361 1293 leader_election.cc:290] T 8d5b10ede9c84be098795c0a787e5a78 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:10.704072 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_79c2dbec23ad41159b38d5177072d78c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:11.238546 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_79c2dbec23ad41159b38d5177072d78c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:10.808686 1241 raft_consensus.cc:491] T 7a8a3aca289c4735abeb9c95c414ab76 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:11.238917 1241 raft_consensus.cc:513] T 7a8a3aca289c4735abeb9c95c414ab76 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:11.239624 1241 leader_election.cc:290] T 7a8a3aca289c4735abeb9c95c414ab76 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:11.240001 447 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 88a6ab29d00542aba54f0e4185fbf615 consensus state has no leader
I20250901 01:52:10.849076 1316 tablet.cc:1722] T b11e8f77243e4764b96548d1132b8e5a P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:11.240207 447 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:57810 (request call id 127) took 1378 ms. Trace:
I20250901 01:52:10.873934 1315 raft_consensus.cc:491] T 4f8e3fcaa19740d397f4d118ac4cbdb0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:11.240298 447 rpcz_store.cc:276] 0901 01:52:09.861971 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:10.093409 (+231438us) service_pool.cc:225] Handling call
0901 01:52:10.098395 (+ 4986us) spinlock_profiling.cc:243] Waited 4.7 ms on lock 0x616000040f54. stack: 00007fc53edd57d6 00007fc53edd5590 00007fc53ddc83ad 0000000000972d94 0000000000972d5d 00007fc54dba014d 00007fc54dba0243 00007fc54a85f941 00007fc5523eaa5d 00007fc552576126 00007fc5493af126 00007fc5493aee55 00007fc545b832ce 00007fc545b821ae 00007fc545b883e5 00007fc545b88d0f
0901 01:52:10.098687 (+ 292us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0901 01:52:10.098929 (+ 242us) catalog_manager.cc:3683] Apply alter schema
0901 01:52:10.099125 (+ 196us) catalog_manager.cc:3715] Acquired catalog manager lock
0901 01:52:10.099192 (+ 67us) catalog_manager.cc:3873] Updating metadata on disk
0901 01:52:10.406224 (+307032us) catalog_manager.cc:3904] Committing alterations to in-memory state
0901 01:52:11.240143 (+833919us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0901 01:52:10.378625 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:10.378780 (+ 155us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:10.378786 (+ 6us) write_op.cc:435] Acquired schema lock
0901 01:52:10.378789 (+ 3us) tablet.cc:660] Decoding operations
0901 01:52:10.379024 (+ 235us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:10.379043 (+ 19us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:10.379052 (+ 9us) tablet.cc:683] Acquiring locks for 1 operations
0901 01:52:10.379195 (+ 143us) tablet.cc:699] Row locks acquired
0901 01:52:10.379200 (+ 5us) write_op.cc:260] PREPARE: finished
0901 01:52:10.379243 (+ 43us) write_op.cc:270] Start()
0901 01:52:10.379285 (+ 42us) write_op.cc:276] Timestamp: P: 1756691530379177 usec, L: 0
0901 01:52:10.379290 (+ 5us) op_driver.cc:348] REPLICATION: starting
0901 01:52:10.399853 (+ 20563us) log.cc:844] Serialized 405 byte log entry
0901 01:52:10.403783 (+ 3930us) op_driver.cc:464] REPLICATION: finished
0901 01:52:10.405649 (+ 1866us) write_op.cc:301] APPLY: starting
0901 01:52:10.405670 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:10.405732 (+ 62us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:10.405762 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:10.405826 (+ 64us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:10.405862 (+ 36us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0901 01:52:10.405870 (+ 8us) write_op.cc:312] APPLY: finished
0901 01:52:10.405957 (+ 87us) log.cc:844] Serialized 23 byte log entry
0901 01:52:10.406020 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:10.406040 (+ 20us) write_op.cc:454] Released schema lock
0901 01:52:10.406048 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0901 01:52:10.430298 (+ 24250us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:10.430311 (+ 13us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":10799360,"child_traces":[["op",{"apply.queue_time_us":1786,"apply.run_cpu_time_us":475,"apply.run_wall_time_us":476,"num_ops":1,"prepare.queue_time_us":278936,"prepare.run_cpu_time_us":1059,"prepare.run_wall_time_us":51639,"raft.queue_time_us":97,"raft.run_cpu_time_us":234,"raft.run_wall_time_us":236,"replication_time_us":24456}]]}
I20250901 01:52:11.240916 1315 raft_consensus.cc:513] T 4f8e3fcaa19740d397f4d118ac4cbdb0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:11.241255 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_7bb211d400ca44d48416f325df223fa6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:11.241545 1315 leader_election.cc:290] T 4f8e3fcaa19740d397f4d118ac4cbdb0 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:11.241751 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7bb211d400ca44d48416f325df223fa6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250901 01:52:10.844589 859 consensus_peers.cc:489] T fab3c1a8928a4066a15868ba708267f1 P 00c282f779c1477d920f24dc0dc16628 -> Peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Couldn't send request to peer fd27ab6665304aaea53adbd102dd4f58. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fab3c1a8928a4066a15868ba708267f1. This is attempt 1: this message will repeat every 5th retry.
W20250901 01:52:11.242182 859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:39447, user_credentials={real_user=slave}} blocked reactor thread for 398258us
I20250901 01:52:11.076027 1314 raft_consensus.cc:491] T 342b7fde3ede494b9eec6e361db255dc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:11.242897 1314 raft_consensus.cc:513] T 342b7fde3ede494b9eec6e361db255dc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } }
I20250901 01:52:11.243009 448 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:57810 (request call id 136) took 1113 ms. Trace:
I20250901 01:52:11.243134 448 rpcz_store.cc:276] 0901 01:52:10.129872 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:10.697863 (+567991us) service_pool.cc:225] Handling call
0901 01:52:10.701179 (+ 3316us) spinlock_profiling.cc:243] Waited 2.97 ms on lock 0x616000040f54. stack: 00007fc53edd57d6 00007fc53edd5590 00007fc53ddc83ad 0000000000972d94 0000000000972d5d 00007fc54dba014d 00007fc54dba0243 00007fc54a85f941 00007fc5523eaa5d 00007fc552580bdb 00007fc5493b80e7 00007fc5493b7e15 00007fc545b832ce 00007fc545b821ae 00007fc545b883e5 00007fc545b88d0f
0901 01:52:11.242939 (+541760us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6838656}
I20250901 01:52:11.243534 1314 leader_election.cc:290] T 342b7fde3ede494b9eec6e361db255dc P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447), 0cf599f74322478d91475df234e58965 (127.0.104.1:40731)
I20250901 01:52:10.706449 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet 8f2e702121c845e0a09a254bbb81c142 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:08 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:11.243633 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_25b49da03a1843a4ad60c23d2385c80a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:10.709774 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet c297e8bf11ff467a854eab41117ac376 with delete_type TABLET_DATA_DELETED (replaced by tablet 186ec810d02e4748aad15228b5866ac1) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:10.758841 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4cdd369214ef4f5ebabddbf28b2ae3f7" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:10.759964 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d333b670df84f308f0cbf6da636b768" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:10.760650 1055 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "77405097e68344628f4eda667afaed21" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:11.244185 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_25b49da03a1843a4ad60c23d2385c80a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:11.244335 1055 raft_consensus.cc:2466] T 77405097e68344628f4eda667afaed21 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00c282f779c1477d920f24dc0dc16628 in term 0.
I20250901 01:52:10.843401 1288 raft_consensus.cc:491] T 8d5b10ede9c84be098795c0a787e5a78 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:10.705826 446 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_457273ab524846e6954d3e60f698e178" } modify_external_catalogs: true
I20250901 01:52:11.244637 1288 raft_consensus.cc:513] T 8d5b10ede9c84be098795c0a787e5a78 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:11.244776 446 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_457273ab524846e6954d3e60f698e178" } modify_external_catalogs: true
I20250901 01:52:10.848009 1151 tablet_bootstrap.cc:492] T 24ddc13d1bb04db4a7850acbff091577 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:11.245296 1288 leader_election.cc:290] T 8d5b10ede9c84be098795c0a787e5a78 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:11.245483 1151 ts_tablet_manager.cc:1397] T 24ddc13d1bb04db4a7850acbff091577 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.679s user 0.020s sys 0.073s
I20250901 01:52:11.246493 1151 raft_consensus.cc:357] T 24ddc13d1bb04db4a7850acbff091577 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } }
I20250901 01:52:11.246732 1151 raft_consensus.cc:383] T 24ddc13d1bb04db4a7850acbff091577 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:11.246812 1151 raft_consensus.cc:738] T 24ddc13d1bb04db4a7850acbff091577 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:11.247089 1151 consensus_queue.cc:260] T 24ddc13d1bb04db4a7850acbff091577 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } }
I20250901 01:52:11.247633 1151 ts_tablet_manager.cc:1428] T 24ddc13d1bb04db4a7850acbff091577 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250901 01:52:11.248036 1151 tablet_bootstrap.cc:492] T 8e6e96b4de214cadba1ccd19390cffb0 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
W20250901 01:52:10.757104 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 2a0d125423a043dcb11e67345599e468 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a0d125423a043dcb11e67345599e468 already in progress: creating tablet
W20250901 01:52:11.248564 440 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.2:40561, user_credentials={real_user=slave}} blocked reactor thread for 491525us
W20250901 01:52:11.249832 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 360755d30b5d47ba98c9c7e23ff5ab7e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 360755d30b5d47ba98c9c7e23ff5ab7e already in progress: creating tablet
I20250901 01:52:11.250099 760 tablet_service.cc:1515] Processing DeleteTablet for tablet 342b7fde3ede494b9eec6e361db255dc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:10 UTC) from {username='slave'} at 127.0.0.1:36178
I20250901 01:52:10.872148 1176 raft_consensus.cc:491] T 97e40c318f64455d91b0e4f8866990ce P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250901 01:52:11.226433 439 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 97e40c318f64455d91b0e4f8866990ce (table default.Table_0c6a1616a7b64455a3ae44c9983dc6c0 [id=9af016a062a64218ad3b62e131d8b65d], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 97e40c318f64455d91b0e4f8866990ce consensus state has no leader
I20250901 01:52:11.250460 1176 raft_consensus.cc:513] T 97e40c318f64455d91b0e4f8866990ce P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:10.944433 1244 raft_consensus.cc:491] T bf302e76746b4f7bbe4469914f83720f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:11.250799 1244 raft_consensus.cc:513] T bf302e76746b4f7bbe4469914f83720f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:11.251253 1176 leader_election.cc:290] T 97e40c318f64455d91b0e4f8866990ce P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:11.251571 1244 leader_election.cc:290] T bf302e76746b4f7bbe4469914f83720f P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:11.009145 1321 consensus_queue.cc:1035] T 3eaf8baac9f1485f944cd9e74b8b5569 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:11.046952 1324 raft_consensus.cc:491] T 116f65a42ab34bf6b8f8113cb7afdc21 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:11.252350 1324 raft_consensus.cc:513] T 116f65a42ab34bf6b8f8113cb7afdc21 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:11.091058 1323 consensus_queue.cc:1035] T c297e8bf11ff467a854eab41117ac376 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:11.253053 1324 leader_election.cc:290] T 116f65a42ab34bf6b8f8113cb7afdc21 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:11.227428 465 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d2ae2cb6af0f4b3a8f47ab1cdd6567d3" candidate_term: 3 candidate_status { last_received { term: 2 index: 142 } } ignore_live_leader: false dest_uuid: "e3add7edb97a40ef9f7c6bc011d13227" is_pre_election: true
I20250901 01:52:11.255139 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 342b7fde3ede494b9eec6e361db255dc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:10 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:11.259145 1151 tablet_bootstrap.cc:654] T 8e6e96b4de214cadba1ccd19390cffb0 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:11.232975 992 leader_election.cc:304] T 061b760d8e6647c6ab0b5d538840a230 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:11.276223 1324 raft_consensus.cc:2802] T 061b760d8e6647c6ab0b5d538840a230 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:11.276453 1324 raft_consensus.cc:695] T 061b760d8e6647c6ab0b5d538840a230 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:11.276774 1324 consensus_queue.cc:237] T 061b760d8e6647c6ab0b5d538840a230 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:11.277074 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet 342b7fde3ede494b9eec6e361db255dc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:10 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:11.279865 1054 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f8e3fcaa19740d397f4d118ac4cbdb0" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:11.280689 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ddc13d1bb04db4a7850acbff091577" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:11.280934 1056 raft_consensus.cc:2466] T 24ddc13d1bb04db4a7850acbff091577 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00c282f779c1477d920f24dc0dc16628 in term 0.
I20250901 01:52:11.281311 1058 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a8a3aca289c4735abeb9c95c414ab76" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:11.282167 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8d5b10ede9c84be098795c0a787e5a78" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:11.282402 1057 raft_consensus.cc:2466] T 8d5b10ede9c84be098795c0a787e5a78 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00c282f779c1477d920f24dc0dc16628 in term 0.
W20250901 01:52:11.285375 859 leader_election.cc:343] T 4cdd369214ef4f5ebabddbf28b2ae3f7 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
W20250901 01:52:11.285674 859 leader_election.cc:343] T 4d333b670df84f308f0cbf6da636b768 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:11.285846 859 leader_election.cc:304] T 77405097e68344628f4eda667afaed21 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:11.286253 1158 raft_consensus.cc:2802] T 77405097e68344628f4eda667afaed21 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:11.286383 1158 raft_consensus.cc:491] T 77405097e68344628f4eda667afaed21 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:11.286492 1158 raft_consensus.cc:3058] T 77405097e68344628f4eda667afaed21 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:11.289592 1292 leader_election.cc:290] T 24ddc13d1bb04db4a7850acbff091577 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447), 0cf599f74322478d91475df234e58965 (127.0.104.1:40731)
I20250901 01:52:11.291270 924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "97e40c318f64455d91b0e4f8866990ce" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:11.291527 924 raft_consensus.cc:2466] T 97e40c318f64455d91b0e4f8866990ce P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:11.291965 923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf302e76746b4f7bbe4469914f83720f" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:11.292155 923 raft_consensus.cc:2466] T bf302e76746b4f7bbe4469914f83720f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:11.293035 922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8d5b10ede9c84be098795c0a787e5a78" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:11.293217 922 raft_consensus.cc:2466] T 8d5b10ede9c84be098795c0a787e5a78 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:11.294520 1151 tablet_bootstrap.cc:492] T 8e6e96b4de214cadba1ccd19390cffb0 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:11.294711 1151 ts_tablet_manager.cc:1397] T 8e6e96b4de214cadba1ccd19390cffb0 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.006s
I20250901 01:52:11.295725 1151 raft_consensus.cc:357] T 8e6e96b4de214cadba1ccd19390cffb0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:11.295944 1151 raft_consensus.cc:383] T 8e6e96b4de214cadba1ccd19390cffb0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:11.296036 1151 raft_consensus.cc:738] T 8e6e96b4de214cadba1ccd19390cffb0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:11.297780 1332 tablet_replica.cc:331] T 8f2e702121c845e0a09a254bbb81c142 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:11.298081 1332 raft_consensus.cc:2241] T 8f2e702121c845e0a09a254bbb81c142 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:11.298398 1332 raft_consensus.cc:2270] T 8f2e702121c845e0a09a254bbb81c142 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:11.299206 1332 ts_tablet_manager.cc:1905] T 8f2e702121c845e0a09a254bbb81c142 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:11.305402 513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.60:44231: connect: Connection refused (error 111) [suppressed 4 similar messages]
I20250901 01:52:11.296269 1151 consensus_queue.cc:260] T 8e6e96b4de214cadba1ccd19390cffb0 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:11.306753 1151 ts_tablet_manager.cc:1428] T 8e6e96b4de214cadba1ccd19390cffb0 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.012s user 0.003s sys 0.000s
W20250901 01:52:11.307030 513 leader_election.cc:336] T 00000000000000000000000000000000 P d2ae2cb6af0f4b3a8f47ab1cdd6567d3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7ce2288f784044eea010664e3def133f (127.0.104.60:44231): Network error: Client connection negotiation failed: client connection to 127.0.104.60:44231: connect: Connection refused (error 111)
I20250901 01:52:11.307148 1151 tablet_bootstrap.cc:492] T 4f8e3fcaa19740d397f4d118ac4cbdb0 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:11.307188 513 leader_election.cc:304] T 00000000000000000000000000000000 P d2ae2cb6af0f4b3a8f47ab1cdd6567d3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2ae2cb6af0f4b3a8f47ab1cdd6567d3; no voters: 7ce2288f784044eea010664e3def133f, e3add7edb97a40ef9f7c6bc011d13227
I20250901 01:52:11.307457 1329 raft_consensus.cc:2747] T 00000000000000000000000000000000 P d2ae2cb6af0f4b3a8f47ab1cdd6567d3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250901 01:52:11.292830 920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "116f65a42ab34bf6b8f8113cb7afdc21" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:11.307693 920 raft_consensus.cc:2466] T 116f65a42ab34bf6b8f8113cb7afdc21 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:11.315294 1151 tablet_bootstrap.cc:654] T 4f8e3fcaa19740d397f4d118ac4cbdb0 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:11.315665 992 leader_election.cc:304] T 97e40c318f64455d91b0e4f8866990ce P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:11.316020 992 leader_election.cc:304] T bf302e76746b4f7bbe4469914f83720f P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:11.316270 1321 raft_consensus.cc:2802] T 97e40c318f64455d91b0e4f8866990ce P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:11.316422 1321 raft_consensus.cc:491] T 97e40c318f64455d91b0e4f8866990ce P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:11.316514 1321 raft_consensus.cc:3058] T 97e40c318f64455d91b0e4f8866990ce P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
W20250901 01:52:11.320693 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 0e698923812841b7a5d37d57da103c5c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0e698923812841b7a5d37d57da103c5c already in progress: creating tablet
I20250901 01:52:11.321061 1288 raft_consensus.cc:2802] T bf302e76746b4f7bbe4469914f83720f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:11.321187 1288 raft_consensus.cc:491] T bf302e76746b4f7bbe4469914f83720f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:11.321288 1288 raft_consensus.cc:3058] T bf302e76746b4f7bbe4469914f83720f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:11.320138 1321 raft_consensus.cc:513] T 97e40c318f64455d91b0e4f8866990ce P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:11.325385 1321 leader_election.cc:290] T 97e40c318f64455d91b0e4f8866990ce P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:11.325838 859 leader_election.cc:343] T 4f8e3fcaa19740d397f4d118ac4cbdb0 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:11.326467 859 leader_election.cc:304] T 24ddc13d1bb04db4a7850acbff091577 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
W20250901 01:52:11.326859 859 leader_election.cc:343] T 7a8a3aca289c4735abeb9c95c414ab76 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:11.327072 859 leader_election.cc:304] T 8d5b10ede9c84be098795c0a787e5a78 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:11.327380 1292 raft_consensus.cc:2802] T 24ddc13d1bb04db4a7850acbff091577 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:11.327505 1292 raft_consensus.cc:491] T 24ddc13d1bb04db4a7850acbff091577 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:11.327606 1292 raft_consensus.cc:3058] T 24ddc13d1bb04db4a7850acbff091577 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:11.331107 1292 raft_consensus.cc:513] T 24ddc13d1bb04db4a7850acbff091577 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } }
I20250901 01:52:11.332243 1314 raft_consensus.cc:2802] T 8d5b10ede9c84be098795c0a787e5a78 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:11.332393 1314 raft_consensus.cc:491] T 8d5b10ede9c84be098795c0a787e5a78 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:11.332511 1314 raft_consensus.cc:3058] T 8d5b10ede9c84be098795c0a787e5a78 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:11.334300 1151 tablet_bootstrap.cc:492] T 4f8e3fcaa19740d397f4d118ac4cbdb0 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:11.334548 1151 ts_tablet_manager.cc:1397] T 4f8e3fcaa19740d397f4d118ac4cbdb0 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.027s user 0.000s sys 0.007s
I20250901 01:52:11.335727 1151 raft_consensus.cc:357] T 4f8e3fcaa19740d397f4d118ac4cbdb0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:11.336023 1151 raft_consensus.cc:383] T 4f8e3fcaa19740d397f4d118ac4cbdb0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:11.336122 1151 raft_consensus.cc:738] T 4f8e3fcaa19740d397f4d118ac4cbdb0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:11.336058 1314 raft_consensus.cc:513] T 8d5b10ede9c84be098795c0a787e5a78 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:11.339665 1151 consensus_queue.cc:260] T 4f8e3fcaa19740d397f4d118ac4cbdb0 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:11.340010 1150 tablet_bootstrap.cc:492] T 447ffe73741640ceb73f35676232a094 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:11.340235 1151 ts_tablet_manager.cc:1428] T 4f8e3fcaa19740d397f4d118ac4cbdb0 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20250901 01:52:11.340605 1151 tablet_bootstrap.cc:492] T 7a8a3aca289c4735abeb9c95c414ab76 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:11.342289 1150 tablet_bootstrap.cc:654] T 447ffe73741640ceb73f35676232a094 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:11.342739 1151 tablet_bootstrap.cc:654] T 7a8a3aca289c4735abeb9c95c414ab76 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:11.342937 1332 ts_tablet_manager.cc:1918] T 8f2e702121c845e0a09a254bbb81c142 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:11.343035 1332 log.cc:1199] T 8f2e702121c845e0a09a254bbb81c142 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/8f2e702121c845e0a09a254bbb81c142
I20250901 01:52:11.343390 1332 ts_tablet_manager.cc:1939] T 8f2e702121c845e0a09a254bbb81c142 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:11.344234 1332 tablet_replica.cc:331] T c297e8bf11ff467a854eab41117ac376 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:11.344446 1332 raft_consensus.cc:2241] T c297e8bf11ff467a854eab41117ac376 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:11.344663 1332 pending_rounds.cc:70] T c297e8bf11ff467a854eab41117ac376 P fd27ab6665304aaea53adbd102dd4f58: Trying to abort 1 pending ops.
I20250901 01:52:11.344727 1332 pending_rounds.cc:77] T c297e8bf11ff467a854eab41117ac376 P fd27ab6665304aaea53adbd102dd4f58: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7195408507979464704 op_type: NO_OP noop_request { }
I20250901 01:52:11.344834 1332 raft_consensus.cc:2887] T c297e8bf11ff467a854eab41117ac376 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250901 01:52:11.344940 1332 raft_consensus.cc:2270] T c297e8bf11ff467a854eab41117ac376 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:11.345458 1332 ts_tablet_manager.cc:1905] T c297e8bf11ff467a854eab41117ac376 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:11.346974 992 leader_election.cc:304] T 8d5b10ede9c84be098795c0a787e5a78 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:11.306831 1158 raft_consensus.cc:513] T 77405097e68344628f4eda667afaed21 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:11.348304 1158 leader_election.cc:290] T 77405097e68344628f4eda667afaed21 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:11.353736 1292 leader_election.cc:290] T 24ddc13d1bb04db4a7850acbff091577 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: Requested vote from peers fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447), 0cf599f74322478d91475df234e58965 (127.0.104.1:40731)
I20250901 01:52:11.354168 1323 raft_consensus.cc:2802] T 8d5b10ede9c84be098795c0a787e5a78 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:11.354312 1323 raft_consensus.cc:491] T 8d5b10ede9c84be098795c0a787e5a78 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:11.354439 1323 raft_consensus.cc:3058] T 8d5b10ede9c84be098795c0a787e5a78 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:11.357043 992 leader_election.cc:304] T 116f65a42ab34bf6b8f8113cb7afdc21 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:11.358963 1323 raft_consensus.cc:513] T 8d5b10ede9c84be098795c0a787e5a78 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:11.359745 1323 leader_election.cc:290] T 8d5b10ede9c84be098795c0a787e5a78 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:11.361459 1288 raft_consensus.cc:513] T bf302e76746b4f7bbe4469914f83720f P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:11.362109 1288 leader_election.cc:290] T bf302e76746b4f7bbe4469914f83720f P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:11.362593 1244 raft_consensus.cc:2802] T 116f65a42ab34bf6b8f8113cb7afdc21 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:11.362702 1244 raft_consensus.cc:491] T 116f65a42ab34bf6b8f8113cb7afdc21 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:11.362856 1244 raft_consensus.cc:3058] T 116f65a42ab34bf6b8f8113cb7afdc21 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:11.364197 1151 tablet_bootstrap.cc:492] T 7a8a3aca289c4735abeb9c95c414ab76 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:11.364380 1151 ts_tablet_manager.cc:1397] T 7a8a3aca289c4735abeb9c95c414ab76 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.001s
I20250901 01:52:11.366394 1151 raft_consensus.cc:357] T 7a8a3aca289c4735abeb9c95c414ab76 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:11.366508 1244 raft_consensus.cc:513] T 116f65a42ab34bf6b8f8113cb7afdc21 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:11.366632 1151 raft_consensus.cc:383] T 7a8a3aca289c4735abeb9c95c414ab76 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:11.366747 1151 raft_consensus.cc:738] T 7a8a3aca289c4735abeb9c95c414ab76 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:11.367841 1314 leader_election.cc:290] T 8d5b10ede9c84be098795c0a787e5a78 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:11.369045 1244 leader_election.cc:290] T 116f65a42ab34bf6b8f8113cb7afdc21 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:11.370800 1151 consensus_queue.cc:260] T 7a8a3aca289c4735abeb9c95c414ab76 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:11.393272 441 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 8f2e702121c845e0a09a254bbb81c142 (table default.Table_3cdfd4047a394877b110a5f00a15c062 [id=512e7d06c7254d889809df25c29fc4c2]) successfully deleted
W20250901 01:52:11.788077 441 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.3:39447, user_credentials={real_user=slave}} blocked reactor thread for 394783us
W20250901 01:52:11.667954 1255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 01:52:11.790406 1255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 01:52:11.790618 1255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250901 01:52:11.547974 760 tablet_service.cc:1468] Processing CreateTablet for tablet f65ec84958eb488d8b0f812b59b5b3d5 (DEFAULT_TABLE table=default.Table_87c07d8f93414f3aaf9b47d8f7bbd24b [id=d1cd5288671540c0a95f7a037363014e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:11.793026 760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f65ec84958eb488d8b0f812b59b5b3d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:11.391729 920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "97e40c318f64455d91b0e4f8866990ce" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:11.796636 920 raft_consensus.cc:3058] T 97e40c318f64455d91b0e4f8866990ce P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:11.371585 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "342b7fde3ede494b9eec6e361db255dc" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:11.521122 454 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4410da9e95454ecf95faf80221b0a71c has no consensus state
I20250901 01:52:11.801093 454 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:57810 (request call id 135) took 1681 ms. Trace:
I20250901 01:52:11.801254 454 rpcz_store.cc:276] 0901 01:52:10.119323 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:10.664838 (+545515us) service_pool.cc:225] Handling call
0901 01:52:10.700785 (+ 35947us) spinlock_profiling.cc:243] Waited 35.6 ms on lock 0x616000040f54. stack: 00007fc53edd57d6 00007fc53edd5590 00007fc53ddc83ad 0000000000972d94 0000000000972d5d 00007fc54dba014d 00007fc54dba0243 00007fc54a85f941 00007fc5523eaa5d 00007fc552576126 00007fc5493af126 00007fc5493aee55 00007fc545b832ce 00007fc545b821ae 00007fc545b883e5 00007fc545b88d0f
0901 01:52:11.233251 (+532466us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0901 01:52:11.233482 (+ 231us) catalog_manager.cc:3683] Apply alter schema
0901 01:52:11.233681 (+ 199us) catalog_manager.cc:3715] Acquired catalog manager lock
0901 01:52:11.233786 (+ 105us) catalog_manager.cc:3873] Updating metadata on disk
0901 01:52:11.520851 (+287065us) catalog_manager.cc:3904] Committing alterations to in-memory state
0901 01:52:11.800992 (+280141us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0901 01:52:11.469333 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:11.469541 (+ 208us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:11.469552 (+ 11us) write_op.cc:435] Acquired schema lock
0901 01:52:11.469557 (+ 5us) tablet.cc:660] Decoding operations
0901 01:52:11.470804 (+ 1247us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:11.470824 (+ 20us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:11.470834 (+ 10us) tablet.cc:683] Acquiring locks for 1 operations
0901 01:52:11.470911 (+ 77us) tablet.cc:699] Row locks acquired
0901 01:52:11.470914 (+ 3us) write_op.cc:260] PREPARE: finished
0901 01:52:11.470963 (+ 49us) write_op.cc:270] Start()
0901 01:52:11.471007 (+ 44us) write_op.cc:276] Timestamp: P: 1756691531470900 usec, L: 0
0901 01:52:11.471011 (+ 4us) op_driver.cc:348] REPLICATION: starting
0901 01:52:11.481547 (+ 10536us) log.cc:844] Serialized 405 byte log entry
0901 01:52:11.507267 (+ 25720us) op_driver.cc:464] REPLICATION: finished
0901 01:52:11.507883 (+ 616us) write_op.cc:301] APPLY: starting
0901 01:52:11.507898 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:11.507933 (+ 35us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:11.507988 (+ 55us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:11.508039 (+ 51us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:11.508069 (+ 30us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0901 01:52:11.508076 (+ 7us) write_op.cc:312] APPLY: finished
0901 01:52:11.508169 (+ 93us) log.cc:844] Serialized 23 byte log entry
0901 01:52:11.520721 (+ 12552us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:11.520741 (+ 20us) write_op.cc:454] Released schema lock
0901 01:52:11.520749 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0901 01:52:11.520838 (+ 89us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:11.520841 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":81905408,"child_traces":[["op",{"apply.queue_time_us":555,"apply.run_cpu_time_us":498,"apply.run_wall_time_us":12946,"num_ops":1,"prepare.queue_time_us":234950,"prepare.run_cpu_time_us":3369,"prepare.run_wall_time_us":25048,"raft.queue_time_us":12964,"raft.run_cpu_time_us":9,"raft.run_wall_time_us":9,"replication_time_us":36225,"thread_start_us":13179,"threads_started":2}]]}
I20250901 01:52:11.806794 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 1f18420261ae41459e1f25b927bd84a9 (DEFAULT_TABLE table=default.Table_a2bf411424c74109a7c2baa28a4db8b9 [id=7c904bfb6cc94518873d6806d1fda600]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:11.807371 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f18420261ae41459e1f25b927bd84a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:11.548431 752 tablet_service.cc:1468] Processing CreateTablet for tablet 920925ee9b014e14a79ceb7796ff7828 (DEFAULT_TABLE table=default.Table_6aa17d35edb04da19ea5d0ce969b34e2 [id=8199118617cb4707bc3ae9b1363750f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:11.393952 922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8d5b10ede9c84be098795c0a787e5a78" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:11.810106 922 raft_consensus.cc:2391] T 8d5b10ede9c84be098795c0a787e5a78 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate fd27ab6665304aaea53adbd102dd4f58 in current term 1: Already voted for candidate 00c282f779c1477d920f24dc0dc16628 in this term.
I20250901 01:52:11.375622 1332 ts_tablet_manager.cc:1918] T c297e8bf11ff467a854eab41117ac376 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:11.810650 1332 log.cc:1199] T c297e8bf11ff467a854eab41117ac376 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/c297e8bf11ff467a854eab41117ac376
I20250901 01:52:11.811271 1332 ts_tablet_manager.cc:1939] T c297e8bf11ff467a854eab41117ac376 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:11.811754 1332 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44246 (request call id 96) took 1108 ms. Trace:
I20250901 01:52:11.811935 1332 rpcz_store.cc:276] 0901 01:52:10.703404 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:10.709663 (+ 6259us) service_pool.cc:225] Handling call
0901 01:52:11.344173 (+634510us) ts_tablet_manager.cc:1145] Deleting tablet c297e8bf11ff467a854eab41117ac376
0901 01:52:11.344181 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:11.358212 (+ 14031us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:11.375442 (+ 17230us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:11.811701 (+436259us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":100334}
I20250901 01:52:11.523470 450 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:57810 (request call id 137) took 1385 ms. Trace:
I20250901 01:52:11.812786 450 rpcz_store.cc:276] 0901 01:52:10.138310 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:10.698964 (+560654us) service_pool.cc:225] Handling call
0901 01:52:10.701409 (+ 2445us) spinlock_profiling.cc:243] Waited 2.15 ms on lock 0x616000040f54. stack: 00007fc53edd57d6 00007fc53edd5590 00007fc53ddc83ad 0000000000972d94 0000000000972d5d 00007fc54dba014d 00007fc54dba0243 00007fc54a85f941 00007fc5523eaa5d 00007fc552573e93 00007fc5493ab7a6 00007fc5493ab4d5 00007fc545b832ce 00007fc545b821ae 00007fc545b883e5 00007fc545b88d0f
0901 01:52:11.235096 (+533687us) catalog_manager.cc:2076] Acquired catalog manager lock
0901 01:52:11.235742 (+ 646us) catalog_manager.cc:2122] Created new table and tablet info
0901 01:52:11.523138 (+287396us) catalog_manager.cc:2182] Wrote table and tablets to system table
0901 01:52:11.523266 (+ 128us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0901 01:52:11.523399 (+ 133us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0901 01:52:11.494431 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:11.494570 (+ 139us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:11.494577 (+ 7us) write_op.cc:435] Acquired schema lock
0901 01:52:11.494580 (+ 3us) tablet.cc:660] Decoding operations
0901 01:52:11.494695 (+ 115us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:11.494707 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:11.494712 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0901 01:52:11.494795 (+ 83us) tablet.cc:699] Row locks acquired
0901 01:52:11.494798 (+ 3us) write_op.cc:260] PREPARE: finished
0901 01:52:11.494834 (+ 36us) write_op.cc:270] Start()
0901 01:52:11.494869 (+ 35us) write_op.cc:276] Timestamp: P: 1756691531494815 usec, L: 0
0901 01:52:11.494872 (+ 3us) op_driver.cc:348] REPLICATION: starting
0901 01:52:11.508757 (+ 13885us) log.cc:844] Serialized 433 byte log entry
0901 01:52:11.509405 (+ 648us) spinlock_profiling.cc:243] Waited 13.7 ms on lock 0x616000040f54. stack: 00007fc53edd57d6 00007fc53edd5590 00007fc53ddc83ad 0000000000972d94 0000000000972d5d 000000000097272a 00007fc54a831d62 00007fc54bbda2d5 00007fc54bbd8df9 00007fc54bbe1220 00007fc54bbe102d 0000000000a774dc 00007fc53ee1bb5f 00007fc53ee1ee2b 00007fc53ee1eca8 0000000000a774dc
0901 01:52:11.522243 (+ 12838us) op_driver.cc:464] REPLICATION: finished
0901 01:52:11.522411 (+ 168us) write_op.cc:301] APPLY: starting
0901 01:52:11.522428 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:11.522497 (+ 69us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:11.522502 (+ 5us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:11.522650 (+ 148us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:11.522694 (+ 44us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0901 01:52:11.522705 (+ 11us) write_op.cc:312] APPLY: finished
0901 01:52:11.522786 (+ 81us) log.cc:844] Serialized 29 byte log entry
0901 01:52:11.523035 (+ 249us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:11.523056 (+ 21us) write_op.cc:454] Released schema lock
0901 01:52:11.523062 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0901 01:52:11.524279 (+ 1217us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:11.524285 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":4940032,"child_traces":[["op",{"apply.queue_time_us":97,"apply.run_cpu_time_us":561,"apply.run_wall_time_us":1219,"num_ops":2,"prepare.queue_time_us":258069,"prepare.run_cpu_time_us":1452,"prepare.run_wall_time_us":15052,"raft.queue_time_us":1878,"raft.run_cpu_time_us":253,"raft.run_wall_time_us":663,"replication_time_us":27340,"spinlock_wait_cycles":31538432}]]}
I20250901 01:52:11.814009 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_c2829420d635461ea141ba803c71bdf9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:11.815402 859 leader_election.cc:343] T 342b7fde3ede494b9eec6e361db255dc P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:11.815976 1034 tablet_service.cc:1468] Processing CreateTablet for tablet f65ec84958eb488d8b0f812b59b5b3d5 (DEFAULT_TABLE table=default.Table_87c07d8f93414f3aaf9b47d8f7bbd24b [id=d1cd5288671540c0a95f7a037363014e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:11.816509 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f65ec84958eb488d8b0f812b59b5b3d5. 1 dirs total, 0 dirs full, 0 dirs failed
#0 0x9f3833 in base::subtle::NoBarrier_Load(int const volatile*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/gutil/atomicops-internals-x86.h:200:10
I20250901 01:52:11.820770 1032 tablet_service.cc:1468] Processing CreateTablet for tablet 920925ee9b014e14a79ceb7796ff7828 (DEFAULT_TABLE table=default.Table_6aa17d35edb04da19ea5d0ce969b34e2 [id=8199118617cb4707bc3ae9b1363750f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:11.821270 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 920925ee9b014e14a79ceb7796ff7828. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:11.824898 1029 tablet_service.cc:1468] Processing CreateTablet for tablet ceb724d859d24d569598e18389dae3b3 (DEFAULT_TABLE table=default.Table_063de0c73851415196ba2e77742edee4 [id=696debd71f87437a880d4ad17fdf5ad8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:11.825430 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ceb724d859d24d569598e18389dae3b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:11.828810 1038 tablet_service.cc:1468] Processing CreateTablet for tablet a5f4b0a2d72a4b8095ae4af1fe7c0c5d (DEFAULT_TABLE table=default.Table_457273ab524846e6954d3e60f698e178 [id=a2546d228d4d42bb9829c56443a4c867]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:11.829324 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5f4b0a2d72a4b8095ae4af1fe7c0c5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:11.833377 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 34042ea5c1b645969c1de41fe587c343 (DEFAULT_TABLE table=default.Table_edc9e269bf86407799be609875a22b0f [id=edbc04078c904f29a16a1bbb354dad92]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:11.833925 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34042ea5c1b645969c1de41fe587c343. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:11.836762 1037 tablet_service.cc:1468] Processing CreateTablet for tablet c27bb23c150f4565a1d0ba8f231cc386 (DEFAULT_TABLE table=default.Table_e066e678c3a8406583dbe851d3424bac [id=f70c2bc07ea245bea6293c3742dfcf94]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:11.837255 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c27bb23c150f4565a1d0ba8f231cc386. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:11.840783 1036 tablet_service.cc:1468] Processing CreateTablet for tablet 3b18a7084727490380d26b3485e4e799 (DEFAULT_TABLE table=default.Table_05707bc6c5ca46c69e791188350c8615 [id=c1b1e9e3876244f68d31c63ce7bdebf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:11.548835 758 tablet_service.cc:1468] Processing CreateTablet for tablet 4410da9e95454ecf95faf80221b0a71c (DEFAULT_TABLE table=default.Table_420b0979d6fd42da9b320456f4ccc9ce [id=d2004563c4374790954327f05c305cc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:11.841766 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4410da9e95454ecf95faf80221b0a71c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:11.394925 921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf302e76746b4f7bbe4469914f83720f" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:11.398880 919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "116f65a42ab34bf6b8f8113cb7afdc21" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:11.416808 1306 ts_tablet_manager.cc:1918] T c297e8bf11ff467a854eab41117ac376 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:11.844625 921 raft_consensus.cc:3058] T bf302e76746b4f7bbe4469914f83720f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:11.844786 1306 log.cc:1199] T c297e8bf11ff467a854eab41117ac376 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/c297e8bf11ff467a854eab41117ac376
I20250901 01:52:11.845417 1306 ts_tablet_manager.cc:1939] T c297e8bf11ff467a854eab41117ac376 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:11.846144 1306 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52846 (request call id 96) took 1136 ms. Trace:
I20250901 01:52:11.846335 1306 rpcz_store.cc:276] 0901 01:52:10.710142 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:10.710264 (+ 122us) service_pool.cc:225] Handling call
0901 01:52:11.228226 (+517962us) ts_tablet_manager.cc:1145] Deleting tablet c297e8bf11ff467a854eab41117ac376
0901 01:52:11.228240 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:11.349520 (+121280us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:11.416618 (+ 67098us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:11.846080 (+429462us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":213}
I20250901 01:52:11.847095 1306 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52846 (request call id 95) took 1138 ms. Trace:
I20250901 01:52:11.847267 1306 rpcz_store.cc:276] 0901 01:52:10.708843 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:10.709072 (+ 229us) service_pool.cc:225] Handling call
0901 01:52:11.846887 (+1137815us) ts_tablet_manager.cc:1145] Deleting tablet 0e698923812841b7a5d37d57da103c5c
0901 01:52:11.846900 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:11.847046 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21852,"tablet-delete.queue_time_us":618869}
I20250901 01:52:11.847681 1306 tablet_replica.cc:331] T 342b7fde3ede494b9eec6e361db255dc P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:11.847934 1306 raft_consensus.cc:2241] T 342b7fde3ede494b9eec6e361db255dc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:11.442636 1150 tablet_bootstrap.cc:492] T 447ffe73741640ceb73f35676232a094 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:11.848292 1150 ts_tablet_manager.cc:1397] T 447ffe73741640ceb73f35676232a094 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.508s user 0.009s sys 0.000s
I20250901 01:52:11.381325 1058 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ddc13d1bb04db4a7850acbff091577" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58"
I20250901 01:52:11.388134 1151 ts_tablet_manager.cc:1428] T 7a8a3aca289c4735abeb9c95c414ab76 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.023s user 0.002s sys 0.000s
I20250901 01:52:11.524475 452 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:57810 (request call id 138) took 1379 ms. Trace:
I20250901 01:52:11.849527 1150 raft_consensus.cc:357] T 447ffe73741640ceb73f35676232a094 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } }
I20250901 01:52:11.849759 1150 raft_consensus.cc:383] T 447ffe73741640ceb73f35676232a094 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:11.849860 1150 raft_consensus.cc:738] T 447ffe73741640ceb73f35676232a094 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:11.850145 1150 consensus_queue.cc:260] T 447ffe73741640ceb73f35676232a094 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } }
I20250901 01:52:11.549178 754 tablet_service.cc:1468] Processing CreateTablet for tablet ceb724d859d24d569598e18389dae3b3 (DEFAULT_TABLE table=default.Table_063de0c73851415196ba2e77742edee4 [id=696debd71f87437a880d4ad17fdf5ad8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:11.850849 1150 ts_tablet_manager.cc:1428] T 447ffe73741640ceb73f35676232a094 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250901 01:52:11.851151 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ceb724d859d24d569598e18389dae3b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:11.851239 1150 tablet_bootstrap.cc:492] T f0affc7a78634607852a48458412761d P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:11.551939 895 tablet_service.cc:1468] Processing CreateTablet for tablet f65ec84958eb488d8b0f812b59b5b3d5 (DEFAULT_TABLE table=default.Table_87c07d8f93414f3aaf9b47d8f7bbd24b [id=d1cd5288671540c0a95f7a037363014e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:11.852803 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f65ec84958eb488d8b0f812b59b5b3d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:11.538323 447 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:57810 (request call id 139) took 1322 ms. Trace:
I20250901 01:52:11.549505 751 tablet_service.cc:1468] Processing CreateTablet for tablet a5f4b0a2d72a4b8095ae4af1fe7c0c5d (DEFAULT_TABLE table=default.Table_457273ab524846e6954d3e60f698e178 [id=a2546d228d4d42bb9829c56443a4c867]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:11.390194 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "77405097e68344628f4eda667afaed21" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58"
I20250901 01:52:11.856894 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5f4b0a2d72a4b8095ae4af1fe7c0c5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:11.391028 1054 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8d5b10ede9c84be098795c0a787e5a78" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58"
I20250901 01:52:11.552314 903 tablet_service.cc:1468] Processing CreateTablet for tablet 920925ee9b014e14a79ceb7796ff7828 (DEFAULT_TABLE table=default.Table_6aa17d35edb04da19ea5d0ce969b34e2 [id=8199118617cb4707bc3ae9b1363750f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:11.411746 1321 consensus_queue.cc:1035] T c445c300d71643609071f0f36528a9c4 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:11.860790 903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 920925ee9b014e14a79ceb7796ff7828. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:11.439080 1337 tablet.cc:1722] T b11e8f77243e4764b96548d1132b8e5a P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:11.446493 1324 raft_consensus.cc:491] T 7a8a3aca289c4735abeb9c95c414ab76 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:11.574940 1176 consensus_queue.cc:1035] T 288e8029382f4154bc1e2103b3035a60 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:11.844655 919 raft_consensus.cc:3058] T 116f65a42ab34bf6b8f8113cb7afdc21 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:11.733621 1244 raft_consensus.cc:491] T 8e6e96b4de214cadba1ccd19390cffb0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:11.554526 453 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 061b760d8e6647c6ab0b5d538840a230 (table default.Table_fbba493de02249499d26467bf15c3d5f [id=748daa63137c48a5b151bd34dc5b13fe]). Expected version 1 got 0
I20250901 01:52:11.560469 448 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:57810 (request call id 140) took 1325 ms. Trace:
I20250901 01:52:11.561041 446 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet a5f4b0a2d72a4b8095ae4af1fe7c0c5d
I20250901 01:52:11.549875 753 tablet_service.cc:1468] Processing CreateTablet for tablet 34042ea5c1b645969c1de41fe587c343 (DEFAULT_TABLE table=default.Table_edc9e269bf86407799be609875a22b0f [id=edbc04078c904f29a16a1bbb354dad92]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:11.865108 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34042ea5c1b645969c1de41fe587c343. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:11.867660 1306 raft_consensus.cc:2270] T 342b7fde3ede494b9eec6e361db255dc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:11.552690 902 tablet_service.cc:1468] Processing CreateTablet for tablet 4410da9e95454ecf95faf80221b0a71c (DEFAULT_TABLE table=default.Table_420b0979d6fd42da9b320456f4ccc9ce [id=d2004563c4374790954327f05c305cc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:11.868702 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4410da9e95454ecf95faf80221b0a71c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:11.869175 919 raft_consensus.cc:2466] T 116f65a42ab34bf6b8f8113cb7afdc21 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:11.550784 755 tablet_service.cc:1468] Processing CreateTablet for tablet c27bb23c150f4565a1d0ba8f231cc386 (DEFAULT_TABLE table=default.Table_e066e678c3a8406583dbe851d3424bac [id=f70c2bc07ea245bea6293c3742dfcf94]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:11.557358 750 tablet_service.cc:1468] Processing CreateTablet for tablet 3b18a7084727490380d26b3485e4e799 (DEFAULT_TABLE table=default.Table_05707bc6c5ca46c69e791188350c8615 [id=c1b1e9e3876244f68d31c63ce7bdebf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:11.553011 899 tablet_service.cc:1468] Processing CreateTablet for tablet ceb724d859d24d569598e18389dae3b3 (DEFAULT_TABLE table=default.Table_063de0c73851415196ba2e77742edee4 [id=696debd71f87437a880d4ad17fdf5ad8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:11.872705 1306 ts_tablet_manager.cc:1905] T 342b7fde3ede494b9eec6e361db255dc P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:11.872841 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b18a7084727490380d26b3485e4e799. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:11.873011 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ceb724d859d24d569598e18389dae3b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:11.873883 1150 tablet_bootstrap.cc:654] T f0affc7a78634607852a48458412761d P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:11.553371 901 tablet_service.cc:1468] Processing CreateTablet for tablet a5f4b0a2d72a4b8095ae4af1fe7c0c5d (DEFAULT_TABLE table=default.Table_457273ab524846e6954d3e60f698e178 [id=a2546d228d4d42bb9829c56443a4c867]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:11.558189 762 tablet_service.cc:1468] Processing CreateTablet for tablet 2f28451a3ecc481a958588466f7fec77 (DEFAULT_TABLE table=default.Table_12450bd55edc4f7d96797422cfae2c50 [id=a919133c60784a4ca1bec96b7f410029]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:11.876762 762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f28451a3ecc481a958588466f7fec77. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:11.876812 901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5f4b0a2d72a4b8095ae4af1fe7c0c5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:11.553655 1293 raft_consensus.cc:491] T eeae5294bd474f368f5766c260ccf3fe P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:11.559950 761 tablet_service.cc:1468] Processing CreateTablet for tablet 58c0e7a2e90e48528bc2bc7ec889f608 (DEFAULT_TABLE table=default.Table_409dad98d7eb455a8e16edfb4acedb34 [id=6f29cdc9974049368af2ae10beadb772]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:11.880600 1293 raft_consensus.cc:513] T eeae5294bd474f368f5766c260ccf3fe P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } }
I20250901 01:52:11.880795 761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58c0e7a2e90e48528bc2bc7ec889f608. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:11.881572 1293 leader_election.cc:290] T eeae5294bd474f368f5766c260ccf3fe P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447), 0cf599f74322478d91475df234e58965 (127.0.104.1:40731)
I20250901 01:52:11.553745 900 tablet_service.cc:1468] Processing CreateTablet for tablet 34042ea5c1b645969c1de41fe587c343 (DEFAULT_TABLE table=default.Table_edc9e269bf86407799be609875a22b0f [id=edbc04078c904f29a16a1bbb354dad92]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:11.882103 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34042ea5c1b645969c1de41fe587c343. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:11.562480 759 tablet_service.cc:1468] Processing CreateTablet for tablet 1f18420261ae41459e1f25b927bd84a9 (DEFAULT_TABLE table=default.Table_a2bf411424c74109a7c2baa28a4db8b9 [id=7c904bfb6cc94518873d6806d1fda600]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:11.555725 897 tablet_service.cc:1468] Processing CreateTablet for tablet c27bb23c150f4565a1d0ba8f231cc386 (DEFAULT_TABLE table=default.Table_e066e678c3a8406583dbe851d3424bac [id=f70c2bc07ea245bea6293c3742dfcf94]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:11.884752 759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f18420261ae41459e1f25b927bd84a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:11.885110 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c27bb23c150f4565a1d0ba8f231cc386. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:11.887905 1306 ts_tablet_manager.cc:1918] T 342b7fde3ede494b9eec6e361db255dc P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:11.564862 749 tablet_service.cc:1468] Processing CreateTablet for tablet 7442c6ff699141e1aff208ebdc9d192a (DEFAULT_TABLE table=default.Table_d61fc9783e7b45e49afb36280adcbd86 [id=0bc0d862df03433dacaf6af87a8390cd]), partition=RANGE (key) PARTITION UNBOUNDED
W20250901 01:52:11.620041 702 kernel_stack_watchdog.cc:198] Thread 1149 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 1828ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fe78f073980 (unknown) at ??:0
@ 0x92cb07 __sanitizer::internal_sched_yield() at ??:?
@ 0x8921bd __interceptor_pthread_create at ??:?
@ 0x7fe78bc459c1 kudu::Thread::StartThread() at ??:0
@ 0xa63f97 kudu::Thread::Create() at ??:?
@ 0x7fe78bc6580b kudu::ThreadPool::CreateThread() at ??:0
@ 0x7fe78bc6122d kudu::ThreadPool::DoSubmit() at ??:0
@ 0x7fe78bc6655f kudu::ThreadPool::Submit() at ??:0
@ 0x7fe79701088d kudu::log::SegmentAllocator::AsyncAllocateSegmentUnlocked() at ??:0
@ 0x7fe79700f87f kudu::log::SegmentAllocator::AllocateSegmentAndRollOver() at ??:0
@ 0x7fe79700f43e kudu::log::SegmentAllocator::Init() at ??:0
@ 0x7fe7970174b5 kudu::log::Log::Init() at ??:0
@ 0x7fe797016811 kudu::log::Log::Open() at ??:0
@ 0x7fe79895a5e1 kudu::tablet::TabletBootstrap::OpenNewLog() at ??:0
@ 0x7fe79895694c kudu::tablet::TabletBootstrap::RunBootstrap() at ??:0
@ 0x7fe7989550cf kudu::tablet::TabletBootstrap::Bootstrap() at ??:0
I20250901 01:52:11.558065 898 tablet_service.cc:1468] Processing CreateTablet for tablet 3b18a7084727490380d26b3485e4e799 (DEFAULT_TABLE table=default.Table_05707bc6c5ca46c69e791188350c8615 [id=c1b1e9e3876244f68d31c63ce7bdebf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:11.888823 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7442c6ff699141e1aff208ebdc9d192a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:11.888855 1306 log.cc:1199] T 342b7fde3ede494b9eec6e361db255dc P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/342b7fde3ede494b9eec6e361db255dc
I20250901 01:52:11.889004 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b18a7084727490380d26b3485e4e799. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:11.889278 1306 ts_tablet_manager.cc:1939] T 342b7fde3ede494b9eec6e361db255dc P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:11.559345 896 tablet_service.cc:1468] Processing CreateTablet for tablet 2f28451a3ecc481a958588466f7fec77 (DEFAULT_TABLE table=default.Table_12450bd55edc4f7d96797422cfae2c50 [id=a919133c60784a4ca1bec96b7f410029]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:11.892879 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f28451a3ecc481a958588466f7fec77. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:11.893141 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c27bb23c150f4565a1d0ba8f231cc386. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:11.893311 1150 tablet_bootstrap.cc:492] T f0affc7a78634607852a48458412761d P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:11.893473 1150 ts_tablet_manager.cc:1397] T f0affc7a78634607852a48458412761d P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.042s user 0.005s sys 0.001s
I20250901 01:52:11.894376 1150 raft_consensus.cc:357] T f0affc7a78634607852a48458412761d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } }
I20250901 01:52:11.894590 1150 raft_consensus.cc:383] T f0affc7a78634607852a48458412761d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:11.898871 1150 raft_consensus.cc:738] T f0affc7a78634607852a48458412761d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:11.899142 1150 consensus_queue.cc:260] T f0affc7a78634607852a48458412761d P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } }
I20250901 01:52:11.899703 1150 ts_tablet_manager.cc:1428] T f0affc7a78634607852a48458412761d P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250901 01:52:11.563505 894 tablet_service.cc:1468] Processing CreateTablet for tablet 1f18420261ae41459e1f25b927bd84a9 (DEFAULT_TABLE table=default.Table_a2bf411424c74109a7c2baa28a4db8b9 [id=7c904bfb6cc94518873d6806d1fda600]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:11.897003 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 920925ee9b014e14a79ceb7796ff7828. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:11.900096 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f18420261ae41459e1f25b927bd84a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:11.566082 893 tablet_service.cc:1468] Processing CreateTablet for tablet 7442c6ff699141e1aff208ebdc9d192a (DEFAULT_TABLE table=default.Table_d61fc9783e7b45e49afb36280adcbd86 [id=0bc0d862df03433dacaf6af87a8390cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:11.904778 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7442c6ff699141e1aff208ebdc9d192a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:11.561563 904 tablet_service.cc:1468] Processing CreateTablet for tablet 58c0e7a2e90e48528bc2bc7ec889f608 (DEFAULT_TABLE table=default.Table_409dad98d7eb455a8e16edfb4acedb34 [id=6f29cdc9974049368af2ae10beadb772]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:11.910429 904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58c0e7a2e90e48528bc2bc7ec889f608. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:11.931325 1150 tablet_bootstrap.cc:492] T 2a0d125423a043dcb11e67345599e468 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:11.935292 1150 tablet_bootstrap.cc:654] T 2a0d125423a043dcb11e67345599e468 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:11.937189 920 raft_consensus.cc:2466] T 97e40c318f64455d91b0e4f8866990ce P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:11.949144 921 raft_consensus.cc:2466] T bf302e76746b4f7bbe4469914f83720f P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
#1 0x7fe7a0c94e41 in kudu::rw_semaphore::unlock() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/rw_semaphore.h:148:5
I20250901 01:52:11.967667 1324 raft_consensus.cc:513] T 7a8a3aca289c4735abeb9c95c414ab76 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:11.968437 1056 raft_consensus.cc:3058] T 77405097e68344628f4eda667afaed21 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:11.970986 1244 raft_consensus.cc:513] T 8e6e96b4de214cadba1ccd19390cffb0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:11.971683 1244 leader_election.cc:290] T 8e6e96b4de214cadba1ccd19390cffb0 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:11.971823 994 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250901 01:52:11.972510 1151 ts_tablet_manager.cc:1460] T 7a8a3aca289c4735abeb9c95c414ab76 P fd27ab6665304aaea53adbd102dd4f58: Tablet startup took 577ms
I20250901 01:52:11.972872 1031 tablet_service.cc:1468] Processing CreateTablet for tablet 4410da9e95454ecf95faf80221b0a71c (DEFAULT_TABLE table=default.Table_420b0979d6fd42da9b320456f4ccc9ce [id=d2004563c4374790954327f05c305cc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:11.979982 1058 raft_consensus.cc:3058] T 24ddc13d1bb04db4a7850acbff091577 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:11.980777 1176 raft_consensus.cc:491] T 4f8e3fcaa19740d397f4d118ac4cbdb0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:11.980926 1176 raft_consensus.cc:513] T 4f8e3fcaa19740d397f4d118ac4cbdb0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:11.981585 1176 leader_election.cc:290] T 4f8e3fcaa19740d397f4d118ac4cbdb0 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:11.982431 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b18a7084727490380d26b3485e4e799. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:11.983222 1324 leader_election.cc:290] T 7a8a3aca289c4735abeb9c95c414ab76 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:11.983405 1054 raft_consensus.cc:2391] T 8d5b10ede9c84be098795c0a787e5a78 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 00c282f779c1477d920f24dc0dc16628 in current term 1: Already voted for candidate fd27ab6665304aaea53adbd102dd4f58 in this term.
I20250901 01:52:11.983471 992 leader_election.cc:304] T 116f65a42ab34bf6b8f8113cb7afdc21 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:11.984045 992 leader_election.cc:304] T 97e40c318f64455d91b0e4f8866990ce P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:11.984097 1324 raft_consensus.cc:2802] T 116f65a42ab34bf6b8f8113cb7afdc21 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:11.984278 1324 raft_consensus.cc:695] T 116f65a42ab34bf6b8f8113cb7afdc21 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:11.984453 992 leader_election.cc:304] T bf302e76746b4f7bbe4469914f83720f P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:11.984557 1324 consensus_queue.cc:237] T 116f65a42ab34bf6b8f8113cb7afdc21 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:11.985072 1035 tablet_service.cc:1468] Processing CreateTablet for tablet 2f28451a3ecc481a958588466f7fec77 (DEFAULT_TABLE table=default.Table_12450bd55edc4f7d96797422cfae2c50 [id=a919133c60784a4ca1bec96b7f410029]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:11.985430 1321 raft_consensus.cc:2802] T 97e40c318f64455d91b0e4f8866990ce P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:11.985566 1321 raft_consensus.cc:695] T 97e40c318f64455d91b0e4f8866990ce P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:11.985852 1244 raft_consensus.cc:2802] T bf302e76746b4f7bbe4469914f83720f P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:11.985814 1321 consensus_queue.cc:237] T 97e40c318f64455d91b0e4f8866990ce P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:11.986485 1033 tablet_service.cc:1468] Processing CreateTablet for tablet 58c0e7a2e90e48528bc2bc7ec889f608 (DEFAULT_TABLE table=default.Table_409dad98d7eb455a8e16edfb4acedb34 [id=6f29cdc9974049368af2ae10beadb772]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:11.987900 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4410da9e95454ecf95faf80221b0a71c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:11.990499 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58c0e7a2e90e48528bc2bc7ec889f608. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:11.990401 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 7442c6ff699141e1aff208ebdc9d192a (DEFAULT_TABLE table=default.Table_d61fc9783e7b45e49afb36280adcbd86 [id=0bc0d862df03433dacaf6af87a8390cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:11.990692 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eeae5294bd474f368f5766c260ccf3fe" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:11.991248 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f28451a3ecc481a958588466f7fec77. 1 dirs total, 0 dirs full, 0 dirs failed
W20250901 01:52:11.987735 1151 ts_tablet_manager.cc:1462] T 7a8a3aca289c4735abeb9c95c414ab76 P fd27ab6665304aaea53adbd102dd4f58: Trace:
0901 01:52:08.897018 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:08.927064 (+ 30046us) service_pool.cc:225] Handling call
0901 01:52:08.927557 (+ 493us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0901 01:52:08.927615 (+ 58us) ts_tablet_manager.cc:676] Creating new metadata...
0901 01:52:08.931001 (+ 3386us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:08.952353 (+ 21352us) tablet_replica.cc:206] Creating consensus instance
0901 01:52:08.952600 (+ 247us) inbound_call.cc:173] Queueing success response
0901 01:52:11.340440 (+2387840us) ts_tablet_manager.cc:1373] Bootstrapping tablet
0901 01:52:11.365356 (+ 24916us) ts_tablet_manager.cc:1429] Starting tablet replica
0901 01:52:11.365639 (+ 283us) tablet_replica.cc:260] Starting instrumentation
0901 01:52:11.365843 (+ 204us) tablet_replica.cc:279] Starting consensus
Metrics: {"mutex_wait_us":16386,"raft.queue_time_us":17208,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":124,"tablet-open.queue_time_us":2387880}
I20250901 01:52:11.997562 1151 tablet_bootstrap.cc:492] T a4f5685260304d589dd467953d9bebfc P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:11.998653 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7442c6ff699141e1aff208ebdc9d192a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:12.005976 448 rpcz_store.cc:276] 0901 01:52:10.235054 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:11.243371 (+1008317us) service_pool.cc:225] Handling call
0901 01:52:11.244292 (+ 921us) catalog_manager.cc:2076] Acquired catalog manager lock
0901 01:52:11.253805 (+ 9513us) catalog_manager.cc:2122] Created new table and tablet info
0901 01:52:11.560169 (+306364us) catalog_manager.cc:2182] Wrote table and tablets to system table
0901 01:52:11.560286 (+ 117us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0901 01:52:11.560388 (+ 102us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0901 01:52:11.544927 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:11.545092 (+ 165us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:11.545100 (+ 8us) write_op.cc:435] Acquired schema lock
0901 01:52:11.545105 (+ 5us) tablet.cc:660] Decoding operations
0901 01:52:11.545277 (+ 172us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:11.545294 (+ 17us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:11.545302 (+ 8us) tablet.cc:683] Acquiring locks for 2 operations
0901 01:52:11.545422 (+ 120us) tablet.cc:699] Row locks acquired
0901 01:52:11.545426 (+ 4us) write_op.cc:260] PREPARE: finished
0901 01:52:11.545473 (+ 47us) write_op.cc:270] Start()
0901 01:52:11.545514 (+ 41us) write_op.cc:276] Timestamp: P: 1756691531545445 usec, L: 0
0901 01:52:11.545520 (+ 6us) op_driver.cc:348] REPLICATION: starting
0901 01:52:11.545802 (+ 282us) log.cc:844] Serialized 433 byte log entry
0901 01:52:11.559256 (+ 13454us) op_driver.cc:464] REPLICATION: finished
0901 01:52:11.559515 (+ 259us) write_op.cc:301] APPLY: starting
0901 01:52:11.559537 (+ 22us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:11.559599 (+ 62us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:11.559649 (+ 50us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:11.559790 (+ 141us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:11.559854 (+ 64us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0901 01:52:11.559865 (+ 11us) write_op.cc:312] APPLY: finished
0901 01:52:11.559965 (+ 100us) log.cc:844] Serialized 29 byte log entry
0901 01:52:11.560041 (+ 76us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:11.560065 (+ 24us) write_op.cc:454] Released schema lock
0901 01:52:11.560074 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0901 01:52:11.560173 (+ 99us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:11.560179 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":194,"apply.run_cpu_time_us":648,"apply.run_wall_time_us":648,"num_ops":2,"prepare.queue_time_us":290498,"prepare.run_cpu_time_us":1007,"prepare.run_wall_time_us":2560,"raft.queue_time_us":1513,"raft.run_cpu_time_us":7,"raft.run_wall_time_us":7,"replication_time_us":13702}]]}
I20250901 01:52:12.008985 453 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet b11e8f77243e4764b96548d1132b8e5a (table default.Table_16b3ac9fb2304c8095e0270038501db0 [id=a39e8b171a874df1890d072732e55df9]). Expected version 1 got 0
W20250901 01:52:12.009582 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c2829420d635461ea141ba803c71bdf9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:12.011595 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_b582d9a2134b4184a195cfe0a8335a33"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:12.008203 447 rpcz_store.cc:276] 0901 01:52:10.215969 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:11.241038 (+1025069us) service_pool.cc:225] Handling call
0901 01:52:11.241844 (+ 806us) catalog_manager.cc:2076] Acquired catalog manager lock
0901 01:52:11.242291 (+ 447us) catalog_manager.cc:2122] Created new table and tablet info
0901 01:52:11.538006 (+295715us) catalog_manager.cc:2182] Wrote table and tablets to system table
0901 01:52:11.538136 (+ 130us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0901 01:52:11.538252 (+ 116us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0901 01:52:11.521035 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:11.521155 (+ 120us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:11.521159 (+ 4us) write_op.cc:435] Acquired schema lock
0901 01:52:11.521162 (+ 3us) tablet.cc:660] Decoding operations
0901 01:52:11.521246 (+ 84us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:11.521255 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:11.521260 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0901 01:52:11.521328 (+ 68us) tablet.cc:699] Row locks acquired
0901 01:52:11.521331 (+ 3us) write_op.cc:260] PREPARE: finished
0901 01:52:11.521393 (+ 62us) write_op.cc:270] Start()
0901 01:52:11.521421 (+ 28us) write_op.cc:276] Timestamp: P: 1756691531521371 usec, L: 0
0901 01:52:11.521425 (+ 4us) op_driver.cc:348] REPLICATION: starting
0901 01:52:11.521602 (+ 177us) log.cc:844] Serialized 433 byte log entry
0901 01:52:11.537054 (+ 15452us) op_driver.cc:464] REPLICATION: finished
0901 01:52:11.537215 (+ 161us) write_op.cc:301] APPLY: starting
0901 01:52:11.537234 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:11.537283 (+ 49us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:11.537315 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:11.537451 (+ 136us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:11.537489 (+ 38us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0901 01:52:11.537496 (+ 7us) write_op.cc:312] APPLY: finished
0901 01:52:11.537598 (+ 102us) log.cc:844] Serialized 29 byte log entry
0901 01:52:11.537887 (+ 289us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:11.537910 (+ 23us) write_op.cc:454] Released schema lock
0901 01:52:11.537917 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0901 01:52:11.538520 (+ 603us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:11.538527 (+ 7us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":47,"apply.run_cpu_time_us":611,"apply.run_wall_time_us":1316,"num_ops":2,"prepare.queue_time_us":278301,"prepare.run_cpu_time_us":666,"prepare.run_wall_time_us":1638,"raft.queue_time_us":30,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":4,"replication_time_us":15584}]]}
I20250901 01:52:12.013383 453 catalog_manager.cc:5582] T c445c300d71643609071f0f36528a9c4 P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: UNKNOWN } } }
I20250901 01:52:12.014802 453 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 97e40c318f64455d91b0e4f8866990ce (table default.Table_0c6a1616a7b64455a3ae44c9983dc6c0 [id=9af016a062a64218ad3b62e131d8b65d]). Expected version 2 got 0
I20250901 01:52:12.015307 441 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet c297e8bf11ff467a854eab41117ac376 (table default.Table_9d0d7a93ff15424092012d4d681b8ae0 [id=aad143eea5c2439299aad87723399c58]) successfully deleted
I20250901 01:52:12.015574 453 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 8d5b10ede9c84be098795c0a787e5a78 (table default.Table_472dac9061664ce0afb468db1d8fa1de [id=017c724f98ec404bab6e882ead9b8541]). Expected version 1 got 0
W20250901 01:52:12.016647 449 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 116f65a42ab34bf6b8f8113cb7afdc21 consensus state has no leader
I20250901 01:52:12.019186 449 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:57810 (request call id 128) took 2131 ms. Trace:
I20250901 01:52:12.022924 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_c1959bda12f14bc9995d832476abf670"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:12.025939 441 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.3:39447, user_credentials={real_user=slave}} blocked reactor thread for 107583us
W20250901 01:52:12.027150 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 342b7fde3ede494b9eec6e361db255dc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 342b7fde3ede494b9eec6e361db255dc already in progress: creating tablet
I20250901 01:52:12.019309 449 rpcz_store.cc:276] 0901 01:52:09.884985 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:10.109711 (+224726us) service_pool.cc:225] Handling call
0901 01:52:10.111073 (+ 1362us) spinlock_profiling.cc:243] Waited 1.02 ms on lock 0x616000040f54. stack: 00007fc53edd57d6 00007fc53edd5590 00007fc53ddc83ad 0000000000972d94 0000000000972d5d 00007fc54dba014d 00007fc54dba0243 00007fc54a85f941 00007fc5523eaa5d 00007fc552576126 00007fc5493af126 00007fc5493aee55 00007fc545b832ce 00007fc545b821ae 00007fc545b883e5 00007fc545b88d0f
0901 01:52:10.111438 (+ 365us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0901 01:52:10.111693 (+ 255us) catalog_manager.cc:3683] Apply alter schema
0901 01:52:10.111919 (+ 226us) catalog_manager.cc:3715] Acquired catalog manager lock
0901 01:52:10.112020 (+ 101us) catalog_manager.cc:3873] Updating metadata on disk
0901 01:52:10.436503 (+324483us) catalog_manager.cc:3904] Committing alterations to in-memory state
0901 01:52:12.016794 (+1580291us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0901 01:52:10.430438 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:10.430656 (+ 218us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:10.430664 (+ 8us) write_op.cc:435] Acquired schema lock
0901 01:52:10.430668 (+ 4us) tablet.cc:660] Decoding operations
0901 01:52:10.430846 (+ 178us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:10.430865 (+ 19us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:10.430874 (+ 9us) tablet.cc:683] Acquiring locks for 1 operations
0901 01:52:10.430964 (+ 90us) tablet.cc:699] Row locks acquired
0901 01:52:10.430968 (+ 4us) write_op.cc:260] PREPARE: finished
0901 01:52:10.431010 (+ 42us) write_op.cc:270] Start()
0901 01:52:10.431056 (+ 46us) write_op.cc:276] Timestamp: P: 1756691530430940 usec, L: 0
0901 01:52:10.431061 (+ 5us) op_driver.cc:348] REPLICATION: starting
0901 01:52:10.431294 (+ 233us) log.cc:844] Serialized 405 byte log entry
0901 01:52:10.435608 (+ 4314us) op_driver.cc:464] REPLICATION: finished
0901 01:52:10.435885 (+ 277us) write_op.cc:301] APPLY: starting
0901 01:52:10.435908 (+ 23us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:10.435965 (+ 57us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:10.436008 (+ 43us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:10.436073 (+ 65us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:10.436119 (+ 46us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0901 01:52:10.436131 (+ 12us) write_op.cc:312] APPLY: finished
0901 01:52:10.436237 (+ 106us) log.cc:844] Serialized 23 byte log entry
0901 01:52:10.436351 (+ 114us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:10.436379 (+ 28us) write_op.cc:454] Released schema lock
0901 01:52:10.436389 (+ 10us) write_op.cc:341] FINISH: Updating metrics
0901 01:52:10.495266 (+ 58877us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:10.495279 (+ 13us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":2341120,"child_traces":[["op",{"apply.queue_time_us":217,"apply.run_cpu_time_us":598,"apply.run_wall_time_us":598,"num_ops":1,"prepare.queue_time_us":317818,"prepare.run_cpu_time_us":1080,"prepare.run_wall_time_us":64775,"raft.queue_time_us":63,"raft.run_cpu_time_us":300,"raft.run_wall_time_us":301,"replication_time_us":4500}]]}
I20250901 01:52:12.030602 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_cb6a9252a90d4184b917db4c7edc06b1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:12.010373 452 rpcz_store.cc:276] 0901 01:52:10.145313 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:10.699171 (+553858us) service_pool.cc:225] Handling call
0901 01:52:10.703996 (+ 4825us) spinlock_profiling.cc:243] Waited 4.52 ms on lock 0x616000040f54. stack: 00007fc53edd57d6 00007fc53edd5590 00007fc53ddc83ad 0000000000972d94 0000000000972d5d 00007fc54dba014d 00007fc54dba0243 00007fc54a85f941 00007fc5523eaa5d 00007fc552573e93 00007fc5493ab7a6 00007fc5493ab4d5 00007fc545b832ce 00007fc545b821ae 00007fc545b883e5 00007fc545b88d0f
0901 01:52:11.238697 (+534701us) catalog_manager.cc:2076] Acquired catalog manager lock
0901 01:52:11.239366 (+ 669us) catalog_manager.cc:2122] Created new table and tablet info
0901 01:52:11.524209 (+284843us) catalog_manager.cc:2182] Wrote table and tablets to system table
0901 01:52:11.524309 (+ 100us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0901 01:52:11.524430 (+ 121us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0901 01:52:11.509503 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:11.509633 (+ 130us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:11.509639 (+ 6us) write_op.cc:435] Acquired schema lock
0901 01:52:11.509641 (+ 2us) tablet.cc:660] Decoding operations
0901 01:52:11.509747 (+ 106us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:11.509757 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:11.509762 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0901 01:52:11.509836 (+ 74us) tablet.cc:699] Row locks acquired
0901 01:52:11.509838 (+ 2us) write_op.cc:260] PREPARE: finished
0901 01:52:11.509870 (+ 32us) write_op.cc:270] Start()
0901 01:52:11.509910 (+ 40us) write_op.cc:276] Timestamp: P: 1756691531509850 usec, L: 0
0901 01:52:11.509913 (+ 3us) op_driver.cc:348] REPLICATION: starting
0901 01:52:11.510038 (+ 125us) log.cc:844] Serialized 433 byte log entry
0901 01:52:11.522306 (+ 12268us) op_driver.cc:464] REPLICATION: finished
0901 01:52:11.523787 (+ 1481us) write_op.cc:301] APPLY: starting
0901 01:52:11.523799 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:11.523859 (+ 60us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:11.523885 (+ 26us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:11.523968 (+ 83us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:11.524000 (+ 32us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0901 01:52:11.524006 (+ 6us) write_op.cc:312] APPLY: finished
0901 01:52:11.524068 (+ 62us) log.cc:844] Serialized 29 byte log entry
0901 01:52:11.524109 (+ 41us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:11.524124 (+ 15us) write_op.cc:454] Released schema lock
0901 01:52:11.524130 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0901 01:52:11.524200 (+ 70us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:11.524204 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":10398080,"child_traces":[["op",{"apply.queue_time_us":1440,"apply.run_cpu_time_us":406,"apply.run_wall_time_us":406,"num_ops":2,"prepare.queue_time_us":269688,"prepare.run_cpu_time_us":650,"prepare.run_wall_time_us":11501,"raft.queue_time_us":1543,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":3,"replication_time_us":12377}]]}
I20250901 01:52:12.031744 452 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_7a731ba38b4048009b421d8da7f58089" } new_table_name: "default.Table_d4d399ae87234df7a710191d0ecf02d0" modify_external_catalogs: true
I20250901 01:52:12.009258 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet c297e8bf11ff467a854eab41117ac376 (table default.Table_9d0d7a93ff15424092012d4d681b8ae0 [id=aad143eea5c2439299aad87723399c58]) successfully deleted
I20250901 01:52:12.041114 455 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:57810 (request call id 134) took 1945 ms. Trace:
I20250901 01:52:12.041236 455 rpcz_store.cc:276] 0901 01:52:10.094607 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:10.657122 (+562515us) service_pool.cc:225] Handling call
0901 01:52:10.700204 (+ 43082us) spinlock_profiling.cc:243] Waited 42.8 ms on lock 0x616000040f54. stack: 00007fc53edd57d6 00007fc53edd5590 00007fc53ddc83ad 0000000000972d94 0000000000972d5d 00007fc54dba014d 00007fc54dba0243 00007fc54a85f941 00007fc5523eaa5d 00007fc552576126 00007fc5493af126 00007fc5493aee55 00007fc545b832ce 00007fc545b821ae 00007fc545b883e5 00007fc545b88d0f
0901 01:52:11.231696 (+531492us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0901 01:52:11.232057 (+ 361us) catalog_manager.cc:3683] Apply alter schema
0901 01:52:11.232330 (+ 273us) catalog_manager.cc:3715] Acquired catalog manager lock
0901 01:52:11.232406 (+ 76us) catalog_manager.cc:3873] Updating metadata on disk
0901 01:52:11.509437 (+277031us) catalog_manager.cc:3904] Committing alterations to in-memory state
0901 01:52:12.039556 (+530119us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0901 01:52:11.448687 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:11.448910 (+ 223us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:11.448918 (+ 8us) write_op.cc:435] Acquired schema lock
0901 01:52:11.448920 (+ 2us) tablet.cc:660] Decoding operations
0901 01:52:11.449052 (+ 132us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:11.449067 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:11.449072 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0901 01:52:11.449146 (+ 74us) tablet.cc:699] Row locks acquired
0901 01:52:11.449149 (+ 3us) write_op.cc:260] PREPARE: finished
0901 01:52:11.449196 (+ 47us) write_op.cc:270] Start()
0901 01:52:11.449238 (+ 42us) write_op.cc:276] Timestamp: P: 1756691531449135 usec, L: 0
0901 01:52:11.449243 (+ 5us) op_driver.cc:348] REPLICATION: starting
0901 01:52:11.449738 (+ 495us) log.cc:844] Serialized 405 byte log entry
0901 01:52:11.494512 (+ 44774us) op_driver.cc:464] REPLICATION: finished
0901 01:52:11.503156 (+ 8644us) write_op.cc:301] APPLY: starting
0901 01:52:11.503179 (+ 23us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:11.503221 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:11.503300 (+ 79us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:11.503389 (+ 89us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:11.503429 (+ 40us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0901 01:52:11.503438 (+ 9us) write_op.cc:312] APPLY: finished
0901 01:52:11.503529 (+ 91us) log.cc:844] Serialized 23 byte log entry
0901 01:52:11.509292 (+ 5763us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:11.509316 (+ 24us) write_op.cc:454] Released schema lock
0901 01:52:11.509325 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0901 01:52:11.509436 (+ 111us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:11.509440 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":98483712,"child_traces":[["op",{"apply.queue_time_us":8542,"apply.run_cpu_time_us":624,"apply.run_wall_time_us":6283,"num_ops":1,"prepare.queue_time_us":211825,"prepare.run_cpu_time_us":1129,"prepare.run_wall_time_us":20598,"raft.queue_time_us":78,"raft.run_cpu_time_us":308,"raft.run_wall_time_us":309,"replication_time_us":45223,"thread_start_us":220914,"threads_started":2}]]}
I20250901 01:52:12.042400 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_7183dbd577624461af0f861fa4f63d6d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:12.043004 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7183dbd577624461af0f861fa4f63d6d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:12.043365 446 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:57810 (request call id 133) took 2037 ms. Trace:
I20250901 01:52:12.043479 446 rpcz_store.cc:276] 0901 01:52:10.002609 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:10.657114 (+654505us) service_pool.cc:225] Handling call
0901 01:52:10.705761 (+ 48647us) spinlock_profiling.cc:243] Waited 48.3 ms on lock 0x616000040f54. stack: 00007fc53edd57d6 00007fc53edd5590 00007fc53ddc83ad 0000000000972d94 0000000000972d5d 00007fc54dba014d 00007fc54dba0243 00007fc54a85f941 00007fc5523eaa5d 00007fc552574e86 00007fc5493ad466 00007fc5493ad195 00007fc545b832ce 00007fc545b821ae 00007fc545b883e5 00007fc545b88d0f
0901 01:52:11.244952 (+539191us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0901 01:52:11.245132 (+ 180us) catalog_manager.cc:2790] Modifying in-memory table state
0901 01:52:11.245237 (+ 105us) catalog_manager.cc:2799] Locking tablets
0901 01:52:11.540056 (+294819us) catalog_manager.cc:2813] Removing table and tablets from system table
0901 01:52:11.560795 (+ 20739us) catalog_manager.cc:2831] Removing table from by-name map
0901 01:52:11.560895 (+ 100us) catalog_manager.cc:2847] Committing in-memory state
0901 01:52:11.560963 (+ 68us) catalog_manager.cc:2851] Aborting table tasks
0901 01:52:12.039836 (+478873us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0901 01:52:11.547583 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:11.547727 (+ 144us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:11.547734 (+ 7us) write_op.cc:435] Acquired schema lock
0901 01:52:11.547738 (+ 4us) tablet.cc:660] Decoding operations
0901 01:52:11.547894 (+ 156us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:11.547907 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:11.547915 (+ 8us) tablet.cc:683] Acquiring locks for 2 operations
0901 01:52:11.551752 (+ 3837us) tablet.cc:699] Row locks acquired
0901 01:52:11.551757 (+ 5us) write_op.cc:260] PREPARE: finished
0901 01:52:11.551811 (+ 54us) write_op.cc:270] Start()
0901 01:52:11.551856 (+ 45us) write_op.cc:276] Timestamp: P: 1756691531551781 usec, L: 0
0901 01:52:11.551862 (+ 6us) op_driver.cc:348] REPLICATION: starting
0901 01:52:11.552119 (+ 257us) log.cc:844] Serialized 725 byte log entry
0901 01:52:11.559356 (+ 7237us) op_driver.cc:464] REPLICATION: finished
0901 01:52:11.560274 (+ 918us) write_op.cc:301] APPLY: starting
0901 01:52:11.560287 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:11.560335 (+ 48us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:11.560388 (+ 53us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:11.560488 (+ 100us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:11.560527 (+ 39us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0901 01:52:11.560538 (+ 11us) write_op.cc:312] APPLY: finished
0901 01:52:11.560608 (+ 70us) log.cc:844] Serialized 29 byte log entry
0901 01:52:11.560674 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:11.560694 (+ 20us) write_op.cc:454] Released schema lock
0901 01:52:11.560701 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0901 01:52:11.561282 (+ 581us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:11.561289 (+ 7us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":111083520,"child_traces":[["op",{"apply.queue_time_us":859,"apply.run_cpu_time_us":522,"apply.run_wall_time_us":974,"num_ops":2,"prepare.queue_time_us":6748,"prepare.run_cpu_time_us":1001,"prepare.run_wall_time_us":5277,"raft.queue_time_us":584,"raft.run_cpu_time_us":6,"raft.run_wall_time_us":7,"replication_time_us":7458}]]}
W20250901 01:52:12.045488 440 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.2:40561, user_credentials={real_user=slave}} blocked reactor thread for 142466us
W20250901 01:52:12.045745 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 0e698923812841b7a5d37d57da103c5c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0e698923812841b7a5d37d57da103c5c already in progress: creating tablet
W20250901 01:52:12.046101 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b582d9a2134b4184a195cfe0a8335a33 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250901 01:52:12.046229 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c1959bda12f14bc9995d832476abf670 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:12.047516 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 342b7fde3ede494b9eec6e361db255dc (table default.Table_e6bab04ba03d4d9fa029e550a887e303 [id=405abb093fe44077b374dc89acb88328]) successfully deleted
I20250901 01:52:12.051560 903 tablet_service.cc:1515] Processing DeleteTablet for tablet 8e6e96b4de214cadba1ccd19390cffb0 with delete_type TABLET_DATA_DELETED (replaced by tablet 3b18a7084727490380d26b3485e4e799) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:12.051896 893 tablet_service.cc:1515] Processing DeleteTablet for tablet fab3c1a8928a4066a15868ba708267f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:08 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:12.052186 1306 tablet_replica.cc:331] T 8e6e96b4de214cadba1ccd19390cffb0 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:12.052246 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_3c7fe20853f245469347813ac5161b25"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:12.052492 1306 raft_consensus.cc:2241] T 8e6e96b4de214cadba1ccd19390cffb0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:12.052738 1306 raft_consensus.cc:2270] T 8e6e96b4de214cadba1ccd19390cffb0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250901 01:52:12.053013 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3c7fe20853f245469347813ac5161b25 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250901 01:52:12.046350 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cb6a9252a90d4184b917db4c7edc06b1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:12.053536 1306 ts_tablet_manager.cc:1905] T 8e6e96b4de214cadba1ccd19390cffb0 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:12.056037 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 288e8029382f4154bc1e2103b3035a60 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:09 UTC) from {username='slave'} at 127.0.0.1:52846
W20250901 01:52:12.056190 451 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:41695 (request call id 15) took 2075 ms (client timeout 1000 ms). Trace:
I20250901 01:52:12.104660 1151 tablet_bootstrap.cc:654] T a4f5685260304d589dd467953d9bebfc P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:12.111899 1058 raft_consensus.cc:2466] T 24ddc13d1bb04db4a7850acbff091577 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c282f779c1477d920f24dc0dc16628 in term 1.
I20250901 01:52:12.071607 1382 raft_consensus.cc:491] T 00000000000000000000000000000000 P d2ae2cb6af0f4b3a8f47ab1cdd6567d3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e3add7edb97a40ef9f7c6bc011d13227)
I20250901 01:52:12.139047 1382 raft_consensus.cc:513] T 00000000000000000000000000000000 P d2ae2cb6af0f4b3a8f47ab1cdd6567d3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3add7edb97a40ef9f7c6bc011d13227" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 43871 } } peers { permanent_uuid: "d2ae2cb6af0f4b3a8f47ab1cdd6567d3" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42375 } } peers { permanent_uuid: "7ce2288f784044eea010664e3def133f" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 44231 } }
I20250901 01:52:12.139891 1382 leader_election.cc:290] T 00000000000000000000000000000000 P d2ae2cb6af0f4b3a8f47ab1cdd6567d3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e3add7edb97a40ef9f7c6bc011d13227 (127.0.104.62:43871), 7ce2288f784044eea010664e3def133f (127.0.104.60:44231)
I20250901 01:52:12.061116 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 7a8a3aca289c4735abeb9c95c414ab76 with delete_type TABLET_DATA_DELETED (replaced by tablet 85322a864d154f299f2d670fdd2a7bfb) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:12.061841 903 tablet_service.cc:1515] Processing DeleteTablet for tablet c297e8bf11ff467a854eab41117ac376 with delete_type TABLET_DATA_DELETED (replaced by tablet 186ec810d02e4748aad15228b5866ac1) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:12.062326 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 4d333b670df84f308f0cbf6da636b768 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:08 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:12.062913 902 tablet_service.cc:1515] Processing DeleteTablet for tablet 77405097e68344628f4eda667afaed21 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:09 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:12.065907 904 tablet_service.cc:1515] Processing DeleteTablet for tablet 4f8e3fcaa19740d397f4d118ac4cbdb0 with delete_type TABLET_DATA_DELETED (replaced by tablet 88a6ab29d00542aba54f0e4185fbf615) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:12.077279 1306 ts_tablet_manager.cc:1918] T 8e6e96b4de214cadba1ccd19390cffb0 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:12.143081 1306 log.cc:1199] T 8e6e96b4de214cadba1ccd19390cffb0 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/8e6e96b4de214cadba1ccd19390cffb0
I20250901 01:52:12.143533 1306 ts_tablet_manager.cc:1939] T 8e6e96b4de214cadba1ccd19390cffb0 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
#2 0x7fe79898321d in std::unique_lock<kudu::rw_semaphore>::unlock() ../../../include/c++/7.5.0/bits/std_mutex.h:323:17
I20250901 01:52:12.144193 1306 tablet_replica.cc:331] T fab3c1a8928a4066a15868ba708267f1 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:12.144393 1306 raft_consensus.cc:2241] T fab3c1a8928a4066a15868ba708267f1 P 00c282f779c1477d920f24dc0dc16628 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:12.144608 1306 pending_rounds.cc:70] T fab3c1a8928a4066a15868ba708267f1 P 00c282f779c1477d920f24dc0dc16628: Trying to abort 1 pending ops.
I20250901 01:52:12.144698 1306 pending_rounds.cc:77] T fab3c1a8928a4066a15868ba708267f1 P 00c282f779c1477d920f24dc0dc16628: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7195408505234038784 op_type: NO_OP noop_request { }
I20250901 01:52:12.144806 1306 raft_consensus.cc:2887] T fab3c1a8928a4066a15868ba708267f1 P 00c282f779c1477d920f24dc0dc16628 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250901 01:52:12.144934 1306 raft_consensus.cc:2270] T fab3c1a8928a4066a15868ba708267f1 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:12.145514 1306 ts_tablet_manager.cc:1905] T fab3c1a8928a4066a15868ba708267f1 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
#3 0x7fe798983183 in std::unique_lock<kudu::rw_semaphore>::~unique_lock() ../../../include/c++/7.5.0/bits/std_mutex.h:232:4
#4 0x7fe7989776a3 in kudu::tablet::AlterSchemaOpState::~AlterSchemaOpState() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tablet/ops/alter_schema_op.h:47:3
W20250901 01:52:12.056306 451 rpcz_store.cc:269] 0901 01:52:09.979357 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:10.008454 (+ 29097us) service_pool.cc:225] Handling call
0901 01:52:10.010607 (+ 2153us) spinlock_profiling.cc:243] Waited 1.14 ms on lock 0x616000040f54. stack: 00007fc53edd57d6 00007fc53edd5590 00007fc53ddc83ad 0000000000972d94 0000000000972d5d 00007fc54dba014d 00007fc54dba0243 00007fc54a85f941 00007fc5523eaa5d 00007fc55256e6d8 00007fc5493a8c86 00007fc5493a89b5 00007fc545b832ce 00007fc545b821ae 00007fc545b883e5 00007fc545b88d0f
0901 01:52:12.054433 (+2043826us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":16,"spinlock_wait_cycles":2620160}
I20250901 01:52:12.149838 465 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d2ae2cb6af0f4b3a8f47ab1cdd6567d3" candidate_term: 3 candidate_status { last_received { term: 2 index: 150 } } ignore_live_leader: false dest_uuid: "e3add7edb97a40ef9f7c6bc011d13227" is_pre_election: true
I20250901 01:52:12.152487 451 master_service.cc:948] ReplaceTablet: received request to replace tablet 3eaf8baac9f1485f944cd9e74b8b5569 from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:12.152899 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 8e6e96b4de214cadba1ccd19390cffb0 (table default.Table_05707bc6c5ca46c69e791188350c8615 [id=c1b1e9e3876244f68d31c63ce7bdebf1]) successfully deleted
I20250901 01:52:12.116474 1384 tablet.cc:1722] T 3eaf8baac9f1485f944cd9e74b8b5569 P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:12.135617 1056 raft_consensus.cc:2466] T 77405097e68344628f4eda667afaed21 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c282f779c1477d920f24dc0dc16628 in term 1.
I20250901 01:52:12.167194 1306 ts_tablet_manager.cc:1918] T fab3c1a8928a4066a15868ba708267f1 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:12.172277 1306 log.cc:1199] T fab3c1a8928a4066a15868ba708267f1 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/fab3c1a8928a4066a15868ba708267f1
I20250901 01:52:12.092727 897 tablet_service.cc:1515] Processing DeleteTablet for tablet bf302e76746b4f7bbe4469914f83720f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:08 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:12.173556 1306 ts_tablet_manager.cc:1939] T fab3c1a8928a4066a15868ba708267f1 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:12.175276 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet fab3c1a8928a4066a15868ba708267f1 (table default.Table_2a6cc96cb6b44ff59a573b691d22c7da [id=dd30bd2e6bee462894768b9341db0ff2]) successfully deleted
#5 0x7fe798a1d822 in kudu::tablet::AlterSchemaOpState::~AlterSchemaOpState() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tablet/ops/alter_schema_op.h:46:25
I20250901 01:52:12.176716 1306 tablet_replica.cc:331] T 288e8029382f4154bc1e2103b3035a60 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:12.177699 1244 raft_consensus.cc:695] T bf302e76746b4f7bbe4469914f83720f P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
W20250901 01:52:12.178038 992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:40561, user_credentials={real_user=slave}} blocked reactor thread for 193140us
I20250901 01:52:12.177506 1306 raft_consensus.cc:2241] T 288e8029382f4154bc1e2103b3035a60 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:12.179577 1306 pending_rounds.cc:70] T 288e8029382f4154bc1e2103b3035a60 P 00c282f779c1477d920f24dc0dc16628: Trying to abort 1 pending ops.
I20250901 01:52:12.179702 752 tablet_service.cc:1515] Processing DeleteTablet for tablet a5f4b0a2d72a4b8095ae4af1fe7c0c5d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:11 UTC) from {username='slave'} at 127.0.0.1:36178
I20250901 01:52:12.179736 1306 pending_rounds.cc:77] T 288e8029382f4154bc1e2103b3035a60 P 00c282f779c1477d920f24dc0dc16628: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7195408509711110144 op_type: NO_OP noop_request { }
I20250901 01:52:12.180368 1244 consensus_queue.cc:237] T bf302e76746b4f7bbe4469914f83720f P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:12.180792 513 leader_election.cc:336] T 00000000000000000000000000000000 P d2ae2cb6af0f4b3a8f47ab1cdd6567d3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7ce2288f784044eea010664e3def133f (127.0.104.60:44231): Network error: Client connection negotiation failed: client connection to 127.0.104.60:44231: connect: Connection refused (error 111)
I20250901 01:52:12.181928 513 leader_election.cc:304] T 00000000000000000000000000000000 P d2ae2cb6af0f4b3a8f47ab1cdd6567d3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2ae2cb6af0f4b3a8f47ab1cdd6567d3; no voters: 7ce2288f784044eea010664e3def133f, e3add7edb97a40ef9f7c6bc011d13227
I20250901 01:52:12.182274 1382 raft_consensus.cc:2747] T 00000000000000000000000000000000 P d2ae2cb6af0f4b3a8f47ab1cdd6567d3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250901 01:52:12.182665 922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f8e3fcaa19740d397f4d118ac4cbdb0" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:12.182710 1306 raft_consensus.cc:2887] T 288e8029382f4154bc1e2103b3035a60 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250901 01:52:12.182883 922 raft_consensus.cc:2466] T 4f8e3fcaa19740d397f4d118ac4cbdb0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:12.091780 899 tablet_service.cc:1515] Processing DeleteTablet for tablet a5f4b0a2d72a4b8095ae4af1fe7c0c5d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:11 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:12.102576 898 tablet_service.cc:1515] Processing DeleteTablet for tablet 8f2e702121c845e0a09a254bbb81c142 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:08 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:12.183348 922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a8a3aca289c4735abeb9c95c414ab76" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:12.183457 1306 raft_consensus.cc:2270] T 288e8029382f4154bc1e2103b3035a60 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:12.183530 922 raft_consensus.cc:2466] T 7a8a3aca289c4735abeb9c95c414ab76 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:12.184628 1306 ts_tablet_manager.cc:1905] T 288e8029382f4154bc1e2103b3035a60 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:12.185632 859 leader_election.cc:343] T eeae5294bd474f368f5766c260ccf3fe P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:12.187022 1151 tablet_bootstrap.cc:492] T a4f5685260304d589dd467953d9bebfc P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:12.187223 1151 ts_tablet_manager.cc:1397] T a4f5685260304d589dd467953d9bebfc P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.193s user 0.005s sys 0.001s
I20250901 01:52:12.187772 1391 tablet.cc:1722] T 3eaf8baac9f1485f944cd9e74b8b5569 P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:12.188053 1151 raft_consensus.cc:357] T a4f5685260304d589dd467953d9bebfc P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } }
I20250901 01:52:12.188254 1151 raft_consensus.cc:383] T a4f5685260304d589dd467953d9bebfc P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:12.188357 1151 raft_consensus.cc:738] T a4f5685260304d589dd467953d9bebfc P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:12.188679 1151 consensus_queue.cc:260] T a4f5685260304d589dd467953d9bebfc P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } }
I20250901 01:52:12.189270 1151 ts_tablet_manager.cc:1428] T a4f5685260304d589dd467953d9bebfc P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250901 01:52:12.189633 1151 tablet_bootstrap.cc:492] T eeae5294bd474f368f5766c260ccf3fe P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:12.191229 1306 ts_tablet_manager.cc:1918] T 288e8029382f4154bc1e2103b3035a60 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:12.191401 1151 tablet_bootstrap.cc:654] T eeae5294bd474f368f5766c260ccf3fe P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:12.201117 1306 log.cc:1199] T 288e8029382f4154bc1e2103b3035a60 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/288e8029382f4154bc1e2103b3035a60
I20250901 01:52:12.201861 1306 ts_tablet_manager.cc:1939] T 288e8029382f4154bc1e2103b3035a60 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:12.199338 859 leader_election.cc:304] T 24ddc13d1bb04db4a7850acbff091577 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:12.202929 859 leader_election.cc:304] T 77405097e68344628f4eda667afaed21 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:12.203008 1306 tablet_replica.cc:331] T 7a8a3aca289c4735abeb9c95c414ab76 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:12.203235 1306 raft_consensus.cc:2241] T 7a8a3aca289c4735abeb9c95c414ab76 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:12.204053 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 288e8029382f4154bc1e2103b3035a60 (table default.Table_4121e29ecfc848658eb0b8e07b0ff24e [id=cf6f662c33424262ba7ba2bebb719c5c]) successfully deleted
I20250901 01:52:12.203305 1293 raft_consensus.cc:2802] T 24ddc13d1bb04db4a7850acbff091577 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:12.203449 1158 raft_consensus.cc:2802] T 77405097e68344628f4eda667afaed21 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:12.204308 1158 raft_consensus.cc:695] T 77405097e68344628f4eda667afaed21 P 00c282f779c1477d920f24dc0dc16628 [term 1 LEADER]: Becoming Leader. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Running, Role: LEADER
I20250901 01:52:12.204392 1306 raft_consensus.cc:2270] T 7a8a3aca289c4735abeb9c95c414ab76 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:12.204612 1158 consensus_queue.cc:237] T 77405097e68344628f4eda667afaed21 P 00c282f779c1477d920f24dc0dc16628 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:12.204957 1306 ts_tablet_manager.cc:1905] T 7a8a3aca289c4735abeb9c95c414ab76 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:12.205775 1293 raft_consensus.cc:695] T 24ddc13d1bb04db4a7850acbff091577 P 00c282f779c1477d920f24dc0dc16628 [term 1 LEADER]: Becoming Leader. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Running, Role: LEADER
I20250901 01:52:12.206103 1293 consensus_queue.cc:237] T 24ddc13d1bb04db4a7850acbff091577 P 00c282f779c1477d920f24dc0dc16628 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } }
I20250901 01:52:12.222702 1306 ts_tablet_manager.cc:1918] T 7a8a3aca289c4735abeb9c95c414ab76 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:12.229651 1306 log.cc:1199] T 7a8a3aca289c4735abeb9c95c414ab76 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/7a8a3aca289c4735abeb9c95c414ab76
I20250901 01:52:12.230454 1306 ts_tablet_manager.cc:1939] T 7a8a3aca289c4735abeb9c95c414ab76 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:12.231832 1306 tablet_replica.cc:331] T 4d333b670df84f308f0cbf6da636b768 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:12.232357 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 7a8a3aca289c4735abeb9c95c414ab76 (table default.Table_a8b4fe01e7c54c3e95663bced8b3bbe4 [id=1d8f34cc14cb4e90a562ff4424b8577a]) successfully deleted
W20250901 01:52:12.232717 440 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet c297e8bf11ff467a854eab41117ac376 because the tablet was not found. No further retry: Not found: Tablet not found: c297e8bf11ff467a854eab41117ac376
I20250901 01:52:12.234635 1033 tablet_service.cc:1515] Processing DeleteTablet for tablet a5f4b0a2d72a4b8095ae4af1fe7c0c5d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:11 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:12.236897 1306 raft_consensus.cc:2241] T 4d333b670df84f308f0cbf6da636b768 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:12.237466 1306 raft_consensus.cc:2270] T 4d333b670df84f308f0cbf6da636b768 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:12.237695 1151 tablet_bootstrap.cc:492] T eeae5294bd474f368f5766c260ccf3fe P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:12.237900 1151 ts_tablet_manager.cc:1397] T eeae5294bd474f368f5766c260ccf3fe P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.048s user 0.004s sys 0.000s
I20250901 01:52:12.238507 1306 ts_tablet_manager.cc:1905] T 4d333b670df84f308f0cbf6da636b768 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:12.238965 1151 raft_consensus.cc:357] T eeae5294bd474f368f5766c260ccf3fe P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } }
I20250901 01:52:12.239212 1151 raft_consensus.cc:383] T eeae5294bd474f368f5766c260ccf3fe P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:12.239308 1151 raft_consensus.cc:738] T eeae5294bd474f368f5766c260ccf3fe P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:12.239614 1151 consensus_queue.cc:260] T eeae5294bd474f368f5766c260ccf3fe P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } }
I20250901 01:52:12.242689 1151 ts_tablet_manager.cc:1428] T eeae5294bd474f368f5766c260ccf3fe P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250901 01:52:12.250329 1150 tablet_bootstrap.cc:492] T 2a0d125423a043dcb11e67345599e468 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:12.251536 1150 ts_tablet_manager.cc:1397] T 2a0d125423a043dcb11e67345599e468 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.320s user 0.012s sys 0.017s
W20250901 01:52:12.252833 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet a5f4b0a2d72a4b8095ae4af1fe7c0c5d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a5f4b0a2d72a4b8095ae4af1fe7c0c5d already in progress: creating tablet
W20250901 01:52:12.253430 992 leader_election.cc:343] T 8e6e96b4de214cadba1ccd19390cffb0 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 8e6e96b4de214cadba1ccd19390cffb0
I20250901 01:52:12.256464 992 leader_election.cc:304] T 4f8e3fcaa19740d397f4d118ac4cbdb0 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:12.257248 992 leader_election.cc:304] T 7a8a3aca289c4735abeb9c95c414ab76 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:12.257712 1150 raft_consensus.cc:357] T 2a0d125423a043dcb11e67345599e468 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:12.257448 1244 raft_consensus.cc:2802] T 4f8e3fcaa19740d397f4d118ac4cbdb0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20250901 01:52:12.258122 442 consensus_peers.cc:489] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227 -> Peer 7ce2288f784044eea010664e3def133f (127.0.104.60:44231): Couldn't send request to peer 7ce2288f784044eea010664e3def133f. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.60:44231: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250901 01:52:12.258199 1244 raft_consensus.cc:491] T 4f8e3fcaa19740d397f4d118ac4cbdb0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:12.258352 1244 raft_consensus.cc:3058] T 4f8e3fcaa19740d397f4d118ac4cbdb0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:12.261399 1244 raft_consensus.cc:513] T 4f8e3fcaa19740d397f4d118ac4cbdb0 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:12.259924 1306 ts_tablet_manager.cc:1918] T 4d333b670df84f308f0cbf6da636b768 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:12.261982 1150 raft_consensus.cc:383] T 2a0d125423a043dcb11e67345599e468 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:12.262095 1306 log.cc:1199] T 4d333b670df84f308f0cbf6da636b768 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/4d333b670df84f308f0cbf6da636b768
I20250901 01:52:12.262104 1150 raft_consensus.cc:738] T 2a0d125423a043dcb11e67345599e468 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:12.263162 1150 consensus_queue.cc:260] T 2a0d125423a043dcb11e67345599e468 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:12.263811 1150 ts_tablet_manager.cc:1428] T 2a0d125423a043dcb11e67345599e468 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20250901 01:52:12.261822 1377 raft_consensus.cc:2802] T 7a8a3aca289c4735abeb9c95c414ab76 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:12.265370 1377 raft_consensus.cc:491] T 7a8a3aca289c4735abeb9c95c414ab76 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:12.265538 1377 raft_consensus.cc:3058] T 7a8a3aca289c4735abeb9c95c414ab76 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:12.268790 1377 raft_consensus.cc:513] T 7a8a3aca289c4735abeb9c95c414ab76 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:12.270236 1377 leader_election.cc:290] T 7a8a3aca289c4735abeb9c95c414ab76 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:12.264624 992 consensus_peers.cc:489] T 288e8029382f4154bc1e2103b3035a60 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Couldn't send request to peer 00c282f779c1477d920f24dc0dc16628. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 288e8029382f4154bc1e2103b3035a60. This is attempt 1: this message will repeat every 5th retry.
W20250901 01:52:12.271378 453 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.3:47125 (request call id 12) took 1489 ms (client timeout 1000 ms). Trace:
W20250901 01:52:12.271520 453 rpcz_store.cc:269] 0901 01:52:10.782373 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:11.230057 (+447684us) service_pool.cc:225] Handling call
0901 01:52:12.271335 (+1041278us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0901 01:52:12.017001 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:12.017159 (+ 158us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:12.017168 (+ 9us) write_op.cc:435] Acquired schema lock
0901 01:52:12.017172 (+ 4us) tablet.cc:660] Decoding operations
0901 01:52:12.017307 (+ 135us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:12.017322 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:12.017326 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0901 01:52:12.017426 (+ 100us) tablet.cc:699] Row locks acquired
0901 01:52:12.017430 (+ 4us) write_op.cc:260] PREPARE: finished
0901 01:52:12.017477 (+ 47us) write_op.cc:270] Start()
0901 01:52:12.017520 (+ 43us) write_op.cc:276] Timestamp: P: 1756691532017441 usec, L: 0
0901 01:52:12.017526 (+ 6us) op_driver.cc:348] REPLICATION: starting
0901 01:52:12.018074 (+ 548us) log.cc:844] Serialized 527 byte log entry
0901 01:52:12.259216 (+241142us) op_driver.cc:464] REPLICATION: finished
0901 01:52:12.267461 (+ 8245us) write_op.cc:301] APPLY: starting
0901 01:52:12.267483 (+ 22us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:12.267529 (+ 46us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:12.267586 (+ 57us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:12.267663 (+ 77us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:12.267699 (+ 36us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0901 01:52:12.267707 (+ 8us) write_op.cc:312] APPLY: finished
0901 01:52:12.267825 (+ 118us) log.cc:844] Serialized 23 byte log entry
0901 01:52:12.268840 (+ 1015us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:12.268863 (+ 23us) write_op.cc:454] Released schema lock
0901 01:52:12.268872 (+ 9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":10,"child_traces":[["op",{"apply.queue_time_us":8148,"num_ops":1,"prepare.queue_time_us":577,"prepare.run_cpu_time_us":1012,"prepare.run_wall_time_us":1198,"raft.queue_time_us":4060,"raft.run_cpu_time_us":455,"raft.run_wall_time_us":4706,"replication_time_us":241625,"spinlock_wait_cycles":676736,"thread_start_us":8809,"threads_started":1}]]}
I20250901 01:52:12.272015 922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f8e3fcaa19740d397f4d118ac4cbdb0" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:12.272243 922 raft_consensus.cc:3058] T 4f8e3fcaa19740d397f4d118ac4cbdb0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:12.275513 922 raft_consensus.cc:2466] T 4f8e3fcaa19740d397f4d118ac4cbdb0 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
W20250901 01:52:12.277086 441 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 116f65a42ab34bf6b8f8113cb7afdc21 (table default.Table_57db59e62bcb45a8b2827445e56dc7de [id=70e493e2ad6b4c988f1d78784f892a35], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 116f65a42ab34bf6b8f8113cb7afdc21 consensus state has no leader
I20250901 01:52:12.278012 1151 tablet_bootstrap.cc:492] T f0affc7a78634607852a48458412761d P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:12.278090 992 leader_election.cc:304] T 4f8e3fcaa19740d397f4d118ac4cbdb0 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
W20250901 01:52:12.278458 992 leader_election.cc:343] T 7a8a3aca289c4735abeb9c95c414ab76 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 7a8a3aca289c4735abeb9c95c414ab76
I20250901 01:52:12.262792 1306 ts_tablet_manager.cc:1939] T 4d333b670df84f308f0cbf6da636b768 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:12.279021 1377 raft_consensus.cc:2802] T 4f8e3fcaa19740d397f4d118ac4cbdb0 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:12.269685 1244 leader_election.cc:290] T 4f8e3fcaa19740d397f4d118ac4cbdb0 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:12.279876 1306 tablet_replica.cc:331] T 77405097e68344628f4eda667afaed21 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:12.280043 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 4d333b670df84f308f0cbf6da636b768 (table default.Table_84b7ac25a6b94ac0bbf0865c39abac94 [id=c4ac57f676cd4799a98ad922c26ef14b]) successfully deleted
I20250901 01:52:12.280092 1306 raft_consensus.cc:2241] T 77405097e68344628f4eda667afaed21 P 00c282f779c1477d920f24dc0dc16628 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:12.280313 1306 pending_rounds.cc:70] T 77405097e68344628f4eda667afaed21 P 00c282f779c1477d920f24dc0dc16628: Trying to abort 1 pending ops.
I20250901 01:52:12.280395 1306 pending_rounds.cc:77] T 77405097e68344628f4eda667afaed21 P 00c282f779c1477d920f24dc0dc16628: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7195408515912781824 op_type: NO_OP noop_request { }
I20250901 01:52:12.280573 1306 raft_consensus.cc:2887] T 77405097e68344628f4eda667afaed21 P 00c282f779c1477d920f24dc0dc16628 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250901 01:52:12.280735 1306 raft_consensus.cc:2270] T 77405097e68344628f4eda667afaed21 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:12.280930 1033 tablet_service.cc:1515] Processing DeleteTablet for tablet 288e8029382f4154bc1e2103b3035a60 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:09 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:12.281188 1332 tablet_replica.cc:331] T 288e8029382f4154bc1e2103b3035a60 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:12.281482 1332 raft_consensus.cc:2241] T 288e8029382f4154bc1e2103b3035a60 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:12.281774 1306 ts_tablet_manager.cc:1905] T 77405097e68344628f4eda667afaed21 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:12.281843 1332 raft_consensus.cc:2270] T 288e8029382f4154bc1e2103b3035a60 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:12.282325 1033 tablet_service.cc:1515] Processing DeleteTablet for tablet c297e8bf11ff467a854eab41117ac376 with delete_type TABLET_DATA_DELETED (replaced by tablet 186ec810d02e4748aad15228b5866ac1) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:12.282716 1033 tablet_service.cc:1515] Processing DeleteTablet for tablet bf302e76746b4f7bbe4469914f83720f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:08 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:12.282750 1332 ts_tablet_manager.cc:1905] T 288e8029382f4154bc1e2103b3035a60 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:12.284358 1150 tablet_bootstrap.cc:492] T 4fd0e4f9d86c4362af96ce9964b72328 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:12.285440 1033 tablet_service.cc:1515] Processing DeleteTablet for tablet 77405097e68344628f4eda667afaed21 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:09 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:12.288349 1306 ts_tablet_manager.cc:1918] T 77405097e68344628f4eda667afaed21 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:12.288522 1306 log.cc:1199] T 77405097e68344628f4eda667afaed21 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/77405097e68344628f4eda667afaed21
I20250901 01:52:12.288972 1306 ts_tablet_manager.cc:1939] T 77405097e68344628f4eda667afaed21 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:12.289721 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 77405097e68344628f4eda667afaed21 (table default.Table_f8793d5bb6ca480491a9729761f3f61b [id=bc05464fd80c46a5b4ea19d08d7baf49]) successfully deleted
I20250901 01:52:12.290081 1306 tablet_replica.cc:331] T 4f8e3fcaa19740d397f4d118ac4cbdb0 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:12.290331 1306 raft_consensus.cc:2241] T 4f8e3fcaa19740d397f4d118ac4cbdb0 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:12.290599 1306 raft_consensus.cc:2270] T 4f8e3fcaa19740d397f4d118ac4cbdb0 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:12.291177 1306 ts_tablet_manager.cc:1905] T 4f8e3fcaa19740d397f4d118ac4cbdb0 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:12.296017 1150 tablet_bootstrap.cc:654] T 4fd0e4f9d86c4362af96ce9964b72328 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:12.297667 1306 ts_tablet_manager.cc:1918] T 4f8e3fcaa19740d397f4d118ac4cbdb0 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:12.297888 1306 log.cc:1199] T 4f8e3fcaa19740d397f4d118ac4cbdb0 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/4f8e3fcaa19740d397f4d118ac4cbdb0
I20250901 01:52:12.297966 1332 ts_tablet_manager.cc:1918] T 288e8029382f4154bc1e2103b3035a60 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:12.298216 1332 log.cc:1199] T 288e8029382f4154bc1e2103b3035a60 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/288e8029382f4154bc1e2103b3035a60
I20250901 01:52:12.298333 1306 ts_tablet_manager.cc:1939] T 4f8e3fcaa19740d397f4d118ac4cbdb0 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:12.298719 1332 ts_tablet_manager.cc:1939] T 288e8029382f4154bc1e2103b3035a60 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:12.299180 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 4f8e3fcaa19740d397f4d118ac4cbdb0 (table default.Table_a3a2b89da62b445e83cd3c88fd4fc775 [id=1d0bc46740444f1fbf2129270e806259]) successfully deleted
I20250901 01:52:12.299607 1306 tablet_replica.cc:331] T bf302e76746b4f7bbe4469914f83720f P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:12.299598 441 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 288e8029382f4154bc1e2103b3035a60 (table default.Table_4121e29ecfc848658eb0b8e07b0ff24e [id=cf6f662c33424262ba7ba2bebb719c5c]) successfully deleted
I20250901 01:52:12.299759 1332 tablet_replica.cc:331] T bf302e76746b4f7bbe4469914f83720f P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:12.299906 1306 raft_consensus.cc:2241] T bf302e76746b4f7bbe4469914f83720f P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
W20250901 01:52:12.299927 441 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet c297e8bf11ff467a854eab41117ac376 because the tablet was not found. No further retry: Not found: Tablet not found: c297e8bf11ff467a854eab41117ac376
I20250901 01:52:12.299974 1332 raft_consensus.cc:2241] T bf302e76746b4f7bbe4469914f83720f P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:12.300171 1306 raft_consensus.cc:2270] T bf302e76746b4f7bbe4469914f83720f P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:12.300262 1332 pending_rounds.cc:70] T bf302e76746b4f7bbe4469914f83720f P fd27ab6665304aaea53adbd102dd4f58: Trying to abort 1 pending ops.
I20250901 01:52:12.300357 1332 pending_rounds.cc:77] T bf302e76746b4f7bbe4469914f83720f P fd27ab6665304aaea53adbd102dd4f58: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7195408515829166080 op_type: NO_OP noop_request { }
I20250901 01:52:12.301112 1306 ts_tablet_manager.cc:1905] T bf302e76746b4f7bbe4469914f83720f P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:12.304939 450 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:57810 (request call id 141) took 2055 ms. Trace:
I20250901 01:52:12.305071 450 rpcz_store.cc:276] 0901 01:52:10.249781 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:11.813767 (+1563986us) service_pool.cc:225] Handling call
0901 01:52:12.009731 (+195964us) catalog_manager.cc:2076] Acquired catalog manager lock
0901 01:52:12.036629 (+ 26898us) catalog_manager.cc:2122] Created new table and tablet info
0901 01:52:12.304574 (+267945us) catalog_manager.cc:2182] Wrote table and tablets to system table
0901 01:52:12.304725 (+ 151us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0901 01:52:12.304860 (+ 135us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0901 01:52:12.037273 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:12.037435 (+ 162us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:12.037443 (+ 8us) write_op.cc:435] Acquired schema lock
0901 01:52:12.037447 (+ 4us) tablet.cc:660] Decoding operations
0901 01:52:12.037565 (+ 118us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:12.037579 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:12.037587 (+ 8us) tablet.cc:683] Acquiring locks for 2 operations
0901 01:52:12.037701 (+ 114us) tablet.cc:699] Row locks acquired
0901 01:52:12.037705 (+ 4us) write_op.cc:260] PREPARE: finished
0901 01:52:12.037746 (+ 41us) write_op.cc:270] Start()
0901 01:52:12.037786 (+ 40us) write_op.cc:276] Timestamp: P: 1756691532037710 usec, L: 0
0901 01:52:12.037791 (+ 5us) op_driver.cc:348] REPLICATION: starting
0901 01:52:12.037988 (+ 197us) log.cc:844] Serialized 433 byte log entry
0901 01:52:12.298201 (+260213us) op_driver.cc:464] REPLICATION: finished
0901 01:52:12.298608 (+ 407us) write_op.cc:301] APPLY: starting
0901 01:52:12.298629 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:12.298675 (+ 46us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:12.298704 (+ 29us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:12.299359 (+ 655us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:12.299402 (+ 43us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0901 01:52:12.299412 (+ 10us) write_op.cc:312] APPLY: finished
0901 01:52:12.303911 (+ 4499us) log.cc:844] Serialized 29 byte log entry
0901 01:52:12.304430 (+ 519us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:12.304451 (+ 21us) write_op.cc:454] Released schema lock
0901 01:52:12.304459 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":315,"apply.run_cpu_time_us":625,"apply.run_wall_time_us":5936,"num_ops":2,"prepare.queue_time_us":93,"prepare.run_cpu_time_us":795,"prepare.run_wall_time_us":796,"replication_time_us":260375,"threads_started":1}]]}
W20250901 01:52:12.307155 452 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4fd0e4f9d86c4362af96ce9964b72328 consensus state has no leader
I20250901 01:52:12.307330 452 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:57810 (request call id 145) took 1974 ms. Trace:
I20250901 01:52:12.307439 452 rpcz_store.cc:276] 0901 01:52:10.332726 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:12.031569 (+1698843us) service_pool.cc:225] Handling call
0901 01:52:12.031915 (+ 346us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0901 01:52:12.032164 (+ 249us) catalog_manager.cc:3683] Apply alter schema
0901 01:52:12.032378 (+ 214us) catalog_manager.cc:3715] Acquired catalog manager lock
0901 01:52:12.032501 (+ 123us) catalog_manager.cc:3873] Updating metadata on disk
0901 01:52:12.306878 (+274377us) catalog_manager.cc:3904] Committing alterations to in-memory state
0901 01:52:12.307294 (+ 416us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0901 01:52:12.033374 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:12.033517 (+ 143us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:12.033526 (+ 9us) write_op.cc:435] Acquired schema lock
0901 01:52:12.033530 (+ 4us) tablet.cc:660] Decoding operations
0901 01:52:12.033662 (+ 132us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:12.033678 (+ 16us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:12.033703 (+ 25us) tablet.cc:683] Acquiring locks for 1 operations
0901 01:52:12.033782 (+ 79us) tablet.cc:699] Row locks acquired
0901 01:52:12.033787 (+ 5us) write_op.cc:260] PREPARE: finished
0901 01:52:12.033833 (+ 46us) write_op.cc:270] Start()
0901 01:52:12.033879 (+ 46us) write_op.cc:276] Timestamp: P: 1756691532033795 usec, L: 0
0901 01:52:12.033885 (+ 6us) op_driver.cc:348] REPLICATION: starting
0901 01:52:12.034114 (+ 229us) log.cc:844] Serialized 405 byte log entry
0901 01:52:12.294095 (+259981us) op_driver.cc:464] REPLICATION: finished
0901 01:52:12.296878 (+ 2783us) write_op.cc:301] APPLY: starting
0901 01:52:12.296903 (+ 25us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:12.296958 (+ 55us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:12.297024 (+ 66us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:12.297097 (+ 73us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:12.297135 (+ 38us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0901 01:52:12.297143 (+ 8us) write_op.cc:312] APPLY: finished
0901 01:52:12.305817 (+ 8674us) log.cc:844] Serialized 23 byte log entry
0901 01:52:12.306492 (+ 675us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:12.306509 (+ 17us) write_op.cc:454] Released schema lock
0901 01:52:12.306515 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":2687,"num_ops":1,"prepare.queue_time_us":178,"prepare.run_cpu_time_us":837,"prepare.run_wall_time_us":839,"replication_time_us":260171,"thread_start_us":3923,"threads_started":1}]]}
I20250901 01:52:12.308214 452 master_service.cc:948] ReplaceTablet: received request to replace tablet 6c15cd14831f422eb615038603c79ed0 from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:12.309701 1306 ts_tablet_manager.cc:1918] T bf302e76746b4f7bbe4469914f83720f P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:12.310111 1306 log.cc:1199] T bf302e76746b4f7bbe4469914f83720f P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/bf302e76746b4f7bbe4469914f83720f
I20250901 01:52:12.310726 1306 ts_tablet_manager.cc:1939] T bf302e76746b4f7bbe4469914f83720f P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:12.311530 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet bf302e76746b4f7bbe4469914f83720f (table default.Table_3063e0e5f956483db9161270ce39f4b9 [id=c393985e34824ce49877085a5f739e8f]) successfully deleted
W20250901 01:52:12.312597 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet a5f4b0a2d72a4b8095ae4af1fe7c0c5d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a5f4b0a2d72a4b8095ae4af1fe7c0c5d already in progress: creating tablet
W20250901 01:52:12.313247 440 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 8f2e702121c845e0a09a254bbb81c142 because the tablet was not found. No further retry: Not found: Tablet not found: 8f2e702121c845e0a09a254bbb81c142
I20250901 01:52:12.314210 1391 tablet.cc:1722] T b11e8f77243e4764b96548d1132b8e5a P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250901 01:52:12.315778 1150 tablet_bootstrap.cc:492] T 4fd0e4f9d86c4362af96ce9964b72328 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:12.316002 1150 ts_tablet_manager.cc:1397] T 4fd0e4f9d86c4362af96ce9964b72328 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.002s
I20250901 01:52:12.311932 1377 raft_consensus.cc:695] T 4f8e3fcaa19740d397f4d118ac4cbdb0 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:12.316373 1377 consensus_queue.cc:237] T 4f8e3fcaa19740d397f4d118ac4cbdb0 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:12.316967 1150 raft_consensus.cc:357] T 4fd0e4f9d86c4362af96ce9964b72328 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:12.317183 1150 raft_consensus.cc:383] T 4fd0e4f9d86c4362af96ce9964b72328 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:12.317265 1150 raft_consensus.cc:738] T 4fd0e4f9d86c4362af96ce9964b72328 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:12.315177 1337 tablet.cc:1722] T b11e8f77243e4764b96548d1132b8e5a P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250901 01:52:12.318451 1293 raft_consensus.cc:491] T f0affc7a78634607852a48458412761d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:12.318675 1293 raft_consensus.cc:513] T f0affc7a78634607852a48458412761d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } }
I20250901 01:52:12.319344 1293 leader_election.cc:290] T f0affc7a78634607852a48458412761d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447), 0cf599f74322478d91475df234e58965 (127.0.104.1:40731)
I20250901 01:52:12.320212 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0affc7a78634607852a48458412761d" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:12.321027 455 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:57810 (request call id 146) took 1812 ms. Trace:
W20250901 01:52:12.321024 859 leader_election.cc:343] T f0affc7a78634607852a48458412761d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:12.321159 455 rpcz_store.cc:276] 0901 01:52:10.508646 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:12.042085 (+1533439us) service_pool.cc:225] Handling call
0901 01:52:12.057014 (+ 14929us) catalog_manager.cc:2076] Acquired catalog manager lock
0901 01:52:12.057660 (+ 646us) catalog_manager.cc:2122] Created new table and tablet info
0901 01:52:12.320701 (+263041us) catalog_manager.cc:2182] Wrote table and tablets to system table
0901 01:52:12.320862 (+ 161us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0901 01:52:12.320960 (+ 98us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0901 01:52:12.059733 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:12.059854 (+ 121us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:12.059862 (+ 8us) write_op.cc:435] Acquired schema lock
0901 01:52:12.059865 (+ 3us) tablet.cc:660] Decoding operations
0901 01:52:12.059972 (+ 107us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:12.059984 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:12.059988 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0901 01:52:12.060069 (+ 81us) tablet.cc:699] Row locks acquired
0901 01:52:12.060072 (+ 3us) write_op.cc:260] PREPARE: finished
0901 01:52:12.060109 (+ 37us) write_op.cc:270] Start()
0901 01:52:12.060144 (+ 35us) write_op.cc:276] Timestamp: P: 1756691532060073 usec, L: 0
0901 01:52:12.060147 (+ 3us) op_driver.cc:348] REPLICATION: starting
0901 01:52:12.063260 (+ 3113us) log.cc:844] Serialized 433 byte log entry
0901 01:52:12.319676 (+256416us) op_driver.cc:464] REPLICATION: finished
0901 01:52:12.319934 (+ 258us) write_op.cc:301] APPLY: starting
0901 01:52:12.319952 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:12.320001 (+ 49us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:12.320034 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:12.320146 (+ 112us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:12.320180 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0901 01:52:12.320187 (+ 7us) write_op.cc:312] APPLY: finished
0901 01:52:12.320272 (+ 85us) log.cc:844] Serialized 29 byte log entry
0901 01:52:12.320573 (+ 301us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:12.320596 (+ 23us) write_op.cc:454] Released schema lock
0901 01:52:12.320606 (+ 10us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":192,"num_ops":2,"prepare.queue_time_us":1478,"prepare.run_cpu_time_us":732,"prepare.run_wall_time_us":3629,"replication_time_us":253944}]]}
I20250901 01:52:12.322314 1374 raft_consensus.cc:491] T eeae5294bd474f368f5766c260ccf3fe P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:12.322464 1374 raft_consensus.cc:513] T eeae5294bd474f368f5766c260ccf3fe P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } }
I20250901 01:52:12.323253 1374 leader_election.cc:290] T eeae5294bd474f368f5766c260ccf3fe P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561), 0cf599f74322478d91475df234e58965 (127.0.104.1:40731)
I20250901 01:52:12.323946 922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eeae5294bd474f368f5766c260ccf3fe" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:12.324105 448 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:57810 (request call id 142) took 2027 ms. Trace:
I20250901 01:52:12.324124 922 raft_consensus.cc:2466] T eeae5294bd474f368f5766c260ccf3fe P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:12.324209 448 rpcz_store.cc:276] 0901 01:52:10.296258 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:12.006682 (+1710424us) service_pool.cc:225] Handling call
0901 01:52:12.058263 (+ 51581us) catalog_manager.cc:2076] Acquired catalog manager lock
0901 01:52:12.058678 (+ 415us) catalog_manager.cc:2122] Created new table and tablet info
0901 01:52:12.323824 (+265146us) catalog_manager.cc:2182] Wrote table and tablets to system table
0901 01:52:12.323947 (+ 123us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0901 01:52:12.324037 (+ 90us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0901 01:52:12.063415 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:12.066647 (+ 3232us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:12.066658 (+ 11us) write_op.cc:435] Acquired schema lock
0901 01:52:12.066662 (+ 4us) tablet.cc:660] Decoding operations
0901 01:52:12.066809 (+ 147us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:12.066826 (+ 17us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:12.066834 (+ 8us) tablet.cc:683] Acquiring locks for 2 operations
0901 01:52:12.066941 (+ 107us) tablet.cc:699] Row locks acquired
0901 01:52:12.066945 (+ 4us) write_op.cc:260] PREPARE: finished
0901 01:52:12.066991 (+ 46us) write_op.cc:270] Start()
0901 01:52:12.067037 (+ 46us) write_op.cc:276] Timestamp: P: 1756691532066951 usec, L: 0
0901 01:52:12.067043 (+ 6us) op_driver.cc:348] REPLICATION: starting
0901 01:52:12.067268 (+ 225us) log.cc:844] Serialized 433 byte log entry
0901 01:52:12.319790 (+252522us) op_driver.cc:464] REPLICATION: finished
0901 01:52:12.322713 (+ 2923us) write_op.cc:301] APPLY: starting
0901 01:52:12.322726 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:12.322763 (+ 37us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:12.322785 (+ 22us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:12.322897 (+ 112us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:12.322929 (+ 32us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0901 01:52:12.322936 (+ 7us) write_op.cc:312] APPLY: finished
0901 01:52:12.323007 (+ 71us) log.cc:844] Serialized 29 byte log entry
0901 01:52:12.323726 (+ 719us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:12.323742 (+ 16us) write_op.cc:454] Released schema lock
0901 01:52:12.323748 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0901 01:52:12.323818 (+ 70us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:12.323822 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":2886,"apply.run_cpu_time_us":439,"apply.run_wall_time_us":1099,"num_ops":2,"prepare.queue_time_us":4384,"prepare.run_cpu_time_us":896,"prepare.run_wall_time_us":3940,"replication_time_us":252725}]]}
I20250901 01:52:12.324704 447 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:57810 (request call id 143) took 2028 ms. Trace:
I20250901 01:52:12.324796 447 rpcz_store.cc:276] 0901 01:52:10.296476 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:12.013149 (+1716673us) service_pool.cc:225] Handling call
0901 01:52:12.059033 (+ 45884us) catalog_manager.cc:2076] Acquired catalog manager lock
0901 01:52:12.059404 (+ 371us) catalog_manager.cc:2122] Created new table and tablet info
0901 01:52:12.324364 (+264960us) catalog_manager.cc:2182] Wrote table and tablets to system table
0901 01:52:12.324538 (+ 174us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0901 01:52:12.324625 (+ 87us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0901 01:52:12.067421 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:12.067568 (+ 147us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:12.067575 (+ 7us) write_op.cc:435] Acquired schema lock
0901 01:52:12.067579 (+ 4us) tablet.cc:660] Decoding operations
0901 01:52:12.067677 (+ 98us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:12.067690 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:12.067698 (+ 8us) tablet.cc:683] Acquiring locks for 2 operations
0901 01:52:12.067797 (+ 99us) tablet.cc:699] Row locks acquired
0901 01:52:12.067801 (+ 4us) write_op.cc:260] PREPARE: finished
0901 01:52:12.067834 (+ 33us) write_op.cc:270] Start()
0901 01:52:12.067867 (+ 33us) write_op.cc:276] Timestamp: P: 1756691532067797 usec, L: 0
0901 01:52:12.067872 (+ 5us) op_driver.cc:348] REPLICATION: starting
0901 01:52:12.068076 (+ 204us) log.cc:844] Serialized 433 byte log entry
0901 01:52:12.319855 (+251779us) op_driver.cc:464] REPLICATION: finished
0901 01:52:12.323220 (+ 3365us) write_op.cc:301] APPLY: starting
0901 01:52:12.323233 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:12.323273 (+ 40us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:12.323295 (+ 22us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:12.323460 (+ 165us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:12.323488 (+ 28us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0901 01:52:12.323496 (+ 8us) write_op.cc:312] APPLY: finished
0901 01:52:12.323552 (+ 56us) log.cc:844] Serialized 29 byte log entry
0901 01:52:12.323903 (+ 351us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:12.323920 (+ 17us) write_op.cc:454] Released schema lock
0901 01:52:12.323926 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":3320,"apply.run_cpu_time_us":409,"apply.run_wall_time_us":755,"num_ops":2,"prepare.queue_time_us":4618,"prepare.run_cpu_time_us":726,"prepare.run_wall_time_us":725,"replication_time_us":251958,"spinlock_wait_cycles":77696}]]}
I20250901 01:52:12.325716 446 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:57810 (request call id 147) took 1749 ms. Trace:
I20250901 01:52:12.325801 446 rpcz_store.cc:276] 0901 01:52:10.576556 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:12.045253 (+1468697us) service_pool.cc:225] Handling call
0901 01:52:12.060426 (+ 15173us) catalog_manager.cc:2076] Acquired catalog manager lock
0901 01:52:12.065940 (+ 5514us) catalog_manager.cc:2122] Created new table and tablet info
0901 01:52:12.325500 (+259560us) catalog_manager.cc:2182] Wrote table and tablets to system table
0901 01:52:12.325579 (+ 79us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0901 01:52:12.325647 (+ 68us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0901 01:52:12.068881 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:12.068995 (+ 114us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:12.069003 (+ 8us) write_op.cc:435] Acquired schema lock
0901 01:52:12.069006 (+ 3us) tablet.cc:660] Decoding operations
0901 01:52:12.069107 (+ 101us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:12.069118 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:12.069125 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
0901 01:52:12.069203 (+ 78us) tablet.cc:699] Row locks acquired
0901 01:52:12.069208 (+ 5us) write_op.cc:260] PREPARE: finished
0901 01:52:12.069235 (+ 27us) write_op.cc:270] Start()
0901 01:52:12.069265 (+ 30us) write_op.cc:276] Timestamp: P: 1756691532069199 usec, L: 0
0901 01:52:12.069270 (+ 5us) op_driver.cc:348] REPLICATION: starting
0901 01:52:12.069431 (+ 161us) log.cc:844] Serialized 433 byte log entry
0901 01:52:12.324164 (+254733us) op_driver.cc:464] REPLICATION: finished
0901 01:52:12.324851 (+ 687us) write_op.cc:301] APPLY: starting
0901 01:52:12.324862 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:12.324889 (+ 27us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:12.324931 (+ 42us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:12.325024 (+ 93us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:12.325057 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0901 01:52:12.325064 (+ 7us) write_op.cc:312] APPLY: finished
0901 01:52:12.325114 (+ 50us) log.cc:844] Serialized 29 byte log entry
0901 01:52:12.325329 (+ 215us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:12.325346 (+ 17us) write_op.cc:454] Released schema lock
0901 01:52:12.325352 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0901 01:52:12.325418 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:12.325424 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":649,"apply.run_cpu_time_us":405,"apply.run_wall_time_us":556,"num_ops":2,"prepare.queue_time_us":2416,"prepare.run_cpu_time_us":620,"prepare.run_wall_time_us":619,"replication_time_us":254878}]]}
I20250901 01:52:12.326362 446 master_service.cc:948] ReplaceTablet: received request to replace tablet 4cdd369214ef4f5ebabddbf28b2ae3f7 from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:12.326553 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 3eaf8baac9f1485f944cd9e74b8b5569
I20250901 01:52:12.326927 449 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:57810 (request call id 144) took 2009 ms. Trace:
I20250901 01:52:12.327204 451 catalog_manager.cc:6652] ReplaceTablet: tablet 3eaf8baac9f1485f944cd9e74b8b5569 deleted and replaced by tablet 5adf6b492640417ca071c94702b48cb9
I20250901 01:52:12.327344 451 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:57810 (request call id 148) took 1532 ms. Trace:
I20250901 01:52:12.327459 451 rpcz_store.cc:276] 0901 01:52:10.795196 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:12.152372 (+1357176us) service_pool.cc:225] Handling call
0901 01:52:12.327298 (+174926us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0901 01:52:12.155868 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:12.156004 (+ 136us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:12.156012 (+ 8us) write_op.cc:435] Acquired schema lock
0901 01:52:12.156014 (+ 2us) tablet.cc:660] Decoding operations
0901 01:52:12.156132 (+ 118us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:12.156145 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:12.156149 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0901 01:52:12.156231 (+ 82us) tablet.cc:699] Row locks acquired
0901 01:52:12.156234 (+ 3us) write_op.cc:260] PREPARE: finished
0901 01:52:12.156277 (+ 43us) write_op.cc:270] Start()
0901 01:52:12.156318 (+ 41us) write_op.cc:276] Timestamp: P: 1756691532156228 usec, L: 0
0901 01:52:12.156322 (+ 4us) op_driver.cc:348] REPLICATION: starting
0901 01:52:12.156541 (+ 219us) log.cc:844] Serialized 656 byte log entry
0901 01:52:12.324221 (+167680us) op_driver.cc:464] REPLICATION: finished
0901 01:52:12.325483 (+ 1262us) write_op.cc:301] APPLY: starting
0901 01:52:12.325493 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:12.325521 (+ 28us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:12.325550 (+ 29us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:12.325633 (+ 83us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:12.325677 (+ 44us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0901 01:52:12.325683 (+ 6us) write_op.cc:312] APPLY: finished
0901 01:52:12.325733 (+ 50us) log.cc:844] Serialized 29 byte log entry
0901 01:52:12.326174 (+ 441us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:12.326187 (+ 13us) write_op.cc:454] Released schema lock
0901 01:52:12.326194 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1229,"num_ops":2,"prepare.queue_time_us":794,"prepare.run_cpu_time_us":755,"prepare.run_wall_time_us":756,"replication_time_us":167881}]]}
I20250901 01:52:12.328208 451 master_service.cc:948] ReplaceTablet: received request to replace tablet 4d333b670df84f308f0cbf6da636b768 from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:12.328928 1293 raft_consensus.cc:491] T 2a0d125423a043dcb11e67345599e468 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:12.329092 1293 raft_consensus.cc:513] T 2a0d125423a043dcb11e67345599e468 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:12.327026 449 rpcz_store.cc:276] 0901 01:52:10.317120 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:12.029965 (+1712845us) service_pool.cc:225] Handling call
0901 01:52:12.060335 (+ 30370us) catalog_manager.cc:2076] Acquired catalog manager lock
0901 01:52:12.064842 (+ 4507us) catalog_manager.cc:2122] Created new table and tablet info
0901 01:52:12.326718 (+261876us) catalog_manager.cc:2182] Wrote table and tablets to system table
0901 01:52:12.326799 (+ 81us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0901 01:52:12.326865 (+ 66us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0901 01:52:12.068195 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:12.068303 (+ 108us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:12.068310 (+ 7us) write_op.cc:435] Acquired schema lock
0901 01:52:12.068314 (+ 4us) tablet.cc:660] Decoding operations
0901 01:52:12.068440 (+ 126us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:12.068451 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:12.068458 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
0901 01:52:12.068536 (+ 78us) tablet.cc:699] Row locks acquired
0901 01:52:12.068540 (+ 4us) write_op.cc:260] PREPARE: finished
0901 01:52:12.068570 (+ 30us) write_op.cc:270] Start()
0901 01:52:12.068597 (+ 27us) write_op.cc:276] Timestamp: P: 1756691532068533 usec, L: 0
0901 01:52:12.068600 (+ 3us) op_driver.cc:348] REPLICATION: starting
0901 01:52:12.068782 (+ 182us) log.cc:844] Serialized 433 byte log entry
0901 01:52:12.324093 (+255311us) op_driver.cc:464] REPLICATION: finished
0901 01:52:12.324303 (+ 210us) write_op.cc:301] APPLY: starting
0901 01:52:12.324314 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:12.324355 (+ 41us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:12.324382 (+ 27us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:12.324571 (+ 189us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:12.324598 (+ 27us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0901 01:52:12.324605 (+ 7us) write_op.cc:312] APPLY: finished
0901 01:52:12.324661 (+ 56us) log.cc:844] Serialized 29 byte log entry
0901 01:52:12.325983 (+ 1322us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:12.326002 (+ 19us) write_op.cc:454] Released schema lock
0901 01:52:12.326010 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0901 01:52:12.326086 (+ 76us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:12.326092 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":168,"apply.run_cpu_time_us":523,"apply.run_wall_time_us":1769,"num_ops":2,"prepare.queue_time_us":2723,"prepare.run_cpu_time_us":654,"prepare.run_wall_time_us":653,"replication_time_us":255445}]]}
I20250901 01:52:12.329855 1293 leader_election.cc:290] T 2a0d125423a043dcb11e67345599e468 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:12.330313 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 3eaf8baac9f1485f944cd9e74b8b5569 with delete_type TABLET_DATA_DELETED (replaced by tablet 5adf6b492640417ca071c94702b48cb9) from {username='slave'} at 127.0.0.1:36178
I20250901 01:52:12.330834 898 tablet_service.cc:1515] Processing DeleteTablet for tablet 3eaf8baac9f1485f944cd9e74b8b5569 with delete_type TABLET_DATA_DELETED (replaced by tablet 5adf6b492640417ca071c94702b48cb9) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:12.332271 1306 tablet_replica.cc:331] T 3eaf8baac9f1485f944cd9e74b8b5569 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:12.332242 448 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_472dac9061664ce0afb468db1d8fa1de" } modify_external_catalogs: true
I20250901 01:52:12.332501 448 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_472dac9061664ce0afb468db1d8fa1de" } modify_external_catalogs: true
I20250901 01:52:12.332546 1306 raft_consensus.cc:2241] T 3eaf8baac9f1485f944cd9e74b8b5569 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:12.332836 1306 raft_consensus.cc:2270] T 3eaf8baac9f1485f944cd9e74b8b5569 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:12.333067 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2a0d125423a043dcb11e67345599e468" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:12.333336 1033 tablet_service.cc:1515] Processing DeleteTablet for tablet 3eaf8baac9f1485f944cd9e74b8b5569 with delete_type TABLET_DATA_DELETED (replaced by tablet 5adf6b492640417ca071c94702b48cb9) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:12.333739 1306 ts_tablet_manager.cc:1905] T 3eaf8baac9f1485f944cd9e74b8b5569 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:12.333761 447 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_12450bd55edc4f7d96797422cfae2c50" } modify_external_catalogs: true
I20250901 01:52:12.333559 992 leader_election.cc:304] T eeae5294bd474f368f5766c260ccf3fe P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:12.333971 447 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_12450bd55edc4f7d96797422cfae2c50" } modify_external_catalogs: true
I20250901 01:52:12.334167 1332 raft_consensus.cc:2887] T bf302e76746b4f7bbe4469914f83720f P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
W20250901 01:52:12.334561 859 leader_election.cc:343] T 2a0d125423a043dcb11e67345599e468 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:12.334766 446 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:57810 (request call id 153) took 1049 ms. Trace:
I20250901 01:52:12.341094 1306 ts_tablet_manager.cc:1918] T 3eaf8baac9f1485f944cd9e74b8b5569 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 01:52:12.341238 1306 log.cc:1199] T 3eaf8baac9f1485f944cd9e74b8b5569 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/3eaf8baac9f1485f944cd9e74b8b5569
I20250901 01:52:12.341617 1306 ts_tablet_manager.cc:1939] T 3eaf8baac9f1485f944cd9e74b8b5569 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:12.354334 1150 consensus_queue.cc:260] T 4fd0e4f9d86c4362af96ce9964b72328 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:12.355366 1150 ts_tablet_manager.cc:1428] T 4fd0e4f9d86c4362af96ce9964b72328 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.039s user 0.003s sys 0.000s
I20250901 01:52:12.356240 1150 tablet_bootstrap.cc:492] T b29e846dfae44f0e9ce3e5fea3763a4f P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:12.358194 1150 tablet_bootstrap.cc:654] T b29e846dfae44f0e9ce3e5fea3763a4f P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:12.334276 1374 raft_consensus.cc:2802] T eeae5294bd474f368f5766c260ccf3fe P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:12.359133 1374 raft_consensus.cc:491] T eeae5294bd474f368f5766c260ccf3fe P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:12.359310 1374 raft_consensus.cc:3058] T eeae5294bd474f368f5766c260ccf3fe P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:12.363023 1374 raft_consensus.cc:513] T eeae5294bd474f368f5766c260ccf3fe P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } }
I20250901 01:52:12.364307 922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eeae5294bd474f368f5766c260ccf3fe" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:12.364566 922 raft_consensus.cc:3058] T eeae5294bd474f368f5766c260ccf3fe P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:12.368319 922 raft_consensus.cc:2466] T eeae5294bd474f368f5766c260ccf3fe P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:12.369060 992 leader_election.cc:304] T eeae5294bd474f368f5766c260ccf3fe P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:12.370417 1374 leader_election.cc:290] T eeae5294bd474f368f5766c260ccf3fe P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561), 0cf599f74322478d91475df234e58965 (127.0.104.1:40731)
I20250901 01:52:12.370744 1374 raft_consensus.cc:2802] T eeae5294bd474f368f5766c260ccf3fe P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:12.370991 1374 raft_consensus.cc:695] T eeae5294bd474f368f5766c260ccf3fe P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:12.372311 1374 consensus_queue.cc:237] T eeae5294bd474f368f5766c260ccf3fe P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } }
I20250901 01:52:12.375506 1150 tablet_bootstrap.cc:492] T b29e846dfae44f0e9ce3e5fea3763a4f P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:12.375733 1150 ts_tablet_manager.cc:1397] T b29e846dfae44f0e9ce3e5fea3763a4f P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.001s
I20250901 01:52:12.375779 1332 raft_consensus.cc:2270] T bf302e76746b4f7bbe4469914f83720f P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:12.376986 1332 ts_tablet_manager.cc:1905] T bf302e76746b4f7bbe4469914f83720f P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:12.377121 1255 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
I20250901 01:52:12.376983 1150 raft_consensus.cc:357] T b29e846dfae44f0e9ce3e5fea3763a4f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
W20250901 01:52:12.377269 1255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 01:52:12.377388 1255 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
I20250901 01:52:12.377389 1150 raft_consensus.cc:383] T b29e846dfae44f0e9ce3e5fea3763a4f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250901 01:52:12.377501 1255 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250901 01:52:12.377563 1150 raft_consensus.cc:738] T b29e846dfae44f0e9ce3e5fea3763a4f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:12.377959 1150 consensus_queue.cc:260] T b29e846dfae44f0e9ce3e5fea3763a4f P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:12.379223 446 rpcz_store.cc:276] 0901 01:52:11.284969 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:12.326256 (+1041287us) service_pool.cc:225] Handling call
0901 01:52:12.334507 (+ 8251us) spinlock_profiling.cc:243] Waited 7.87 ms on lock 0x616000040f54. stack: 00007fc53edd57d6 00007fc53edd5590 00007fc53ddc83ad 0000000000972d94 0000000000972d5d 00007fc54dba014d 00007fc54dba0243 00007fc54a85f941 00007fc5523eaa5d 00007fc552580bdb 00007fc5493b80e7 00007fc5493b7e15 00007fc545b832ce 00007fc545b821ae 00007fc545b883e5 00007fc545b88d0f
0901 01:52:12.334716 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":18111232}
I20250901 01:52:12.379644 1150 ts_tablet_manager.cc:1428] T b29e846dfae44f0e9ce3e5fea3763a4f P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250901 01:52:12.380231 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 3eaf8baac9f1485f944cd9e74b8b5569 (table default.Table_947ec0e1a97a40b59be1e234bcaf4e0d [id=5d14706774d345c48ca3575feabb17b0]) successfully deleted
I20250901 01:52:12.380601 1150 tablet_bootstrap.cc:492] T 467c2bf4a8cb4937b6f627bea28d8ddb P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:12.383447 455 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_6a5316d194864096bdfdf11e382f69d2" } modify_external_catalogs: true
I20250901 01:52:12.383631 455 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_6a5316d194864096bdfdf11e382f69d2" } modify_external_catalogs: true
I20250901 01:52:12.385932 449 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_c766b74c950b433b963d93d396521bcd" } modify_external_catalogs: true
I20250901 01:52:12.386153 449 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_c766b74c950b433b963d93d396521bcd" } modify_external_catalogs: true
I20250901 01:52:12.386758 446 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_2f7b3be40c0240f9af56af1fec427da1" } new_table_name: "default.Table_300cde3d9237488ba65cab60dd8f8721" modify_external_catalogs: true
I20250901 01:52:12.388492 454 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet b11e8f77243e4764b96548d1132b8e5a (table default.Table_3c7c628f35fb4d02bb79a34995f12938 [id=a39e8b171a874df1890d072732e55df9]). Expected version 2 got 0
I20250901 01:52:12.388761 454 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 24ddc13d1bb04db4a7850acbff091577 (table default.Table_8c1924d21a91425cbbbd8429d56cf0ab [id=18aecbf3fe4643c4b589ee47706161bd]). Expected version 2 got 0
I20250901 01:52:12.389053 454 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 061b760d8e6647c6ab0b5d538840a230 (table default.Table_fbba493de02249499d26467bf15c3d5f [id=748daa63137c48a5b151bd34dc5b13fe]). Expected version 1 got 0
I20250901 01:52:12.389259 454 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 97e40c318f64455d91b0e4f8866990ce (table default.Table_0c6a1616a7b64455a3ae44c9983dc6c0 [id=9af016a062a64218ad3b62e131d8b65d]). Expected version 2 got 0
I20250901 01:52:12.389559 454 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 8d5b10ede9c84be098795c0a787e5a78 (table default.Table_472dac9061664ce0afb468db1d8fa1de [id=017c724f98ec404bab6e882ead9b8541]). Expected version 1 got 0
I20250901 01:52:12.389715 454 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 116f65a42ab34bf6b8f8113cb7afdc21 (table default.Table_57db59e62bcb45a8b2827445e56dc7de [id=70e493e2ad6b4c988f1d78784f892a35]). Expected version 1 got 0
I20250901 01:52:12.395463 760 tablet_service.cc:1468] Processing CreateTablet for tablet 3df908c3096a4df7bddd8ac88b49cfe0 (DEFAULT_TABLE table=default.Table_7bb211d400ca44d48416f325df223fa6 [id=cc26c89db2484a6fad919231116285f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:12.395990 760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3df908c3096a4df7bddd8ac88b49cfe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:12.396705 752 tablet_service.cc:1468] Processing CreateTablet for tablet fcb7b5b7d63d4f0b906fea97cb548de2 (DEFAULT_TABLE table=default.Table_25b49da03a1843a4ad60c23d2385c80a [id=4d9ea39016ca4bf4859f4f80ea4e9ed6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:12.397167 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcb7b5b7d63d4f0b906fea97cb548de2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:12.401146 898 tablet_service.cc:1468] Processing CreateTablet for tablet fcb7b5b7d63d4f0b906fea97cb548de2 (DEFAULT_TABLE table=default.Table_25b49da03a1843a4ad60c23d2385c80a [id=4d9ea39016ca4bf4859f4f80ea4e9ed6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:12.401742 451 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:57810 (request call id 154) took 1097 ms. Trace:
I20250901 01:52:12.401877 451 rpcz_store.cc:276] 0901 01:52:11.304394 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:12.328144 (+1023750us) service_pool.cc:225] Handling call
0901 01:52:12.392911 (+ 64767us) spinlock_profiling.cc:243] Waited 52.9 ms on lock 0x616000040f54. stack: 00007fc53edd57d6 00007fc53edd5590 00007fc53ddc83ad 0000000000972d94 0000000000972d5d 00007fc54dba014d 00007fc54dba0243 00007fc54a85f941 00007fc5523eaa5d 00007fc552580bdb 00007fc5493b80e7 00007fc5493b7e15 00007fc545b832ce 00007fc545b821ae 00007fc545b883e5 00007fc545b88d0f
0901 01:52:12.401673 (+ 8762us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":121584896}
I20250901 01:52:12.402609 751 tablet_service.cc:1468] Processing CreateTablet for tablet d60d571b2c2840a2b20629bb832d1309 (DEFAULT_TABLE table=default.Table_79c2dbec23ad41159b38d5177072d78c [id=d4c09fc79416433b9555ed44abf26bfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:12.402983 451 master_service.cc:948] ReplaceTablet: received request to replace tablet 88a6ab29d00542aba54f0e4185fbf615 from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:12.403061 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d60d571b2c2840a2b20629bb832d1309. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:12.404713 758 tablet_service.cc:1468] Processing CreateTablet for tablet 186ec810d02e4748aad15228b5866ac1 (DEFAULT_TABLE table=default.Table_9d0d7a93ff15424092012d4d681b8ae0 [id=aad143eea5c2439299aad87723399c58]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:12.405143 758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 186ec810d02e4748aad15228b5866ac1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:12.405926 1150 tablet_bootstrap.cc:654] T 467c2bf4a8cb4937b6f627bea28d8ddb P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:12.408950 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcb7b5b7d63d4f0b906fea97cb548de2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:12.409449 899 tablet_service.cc:1468] Processing CreateTablet for tablet 3df908c3096a4df7bddd8ac88b49cfe0 (DEFAULT_TABLE table=default.Table_7bb211d400ca44d48416f325df223fa6 [id=cc26c89db2484a6fad919231116285f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:12.409909 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3df908c3096a4df7bddd8ac88b49cfe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:12.412475 1033 tablet_service.cc:1468] Processing CreateTablet for tablet fcb7b5b7d63d4f0b906fea97cb548de2 (DEFAULT_TABLE table=default.Table_25b49da03a1843a4ad60c23d2385c80a [id=4d9ea39016ca4bf4859f4f80ea4e9ed6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:12.412739 897 tablet_service.cc:1468] Processing CreateTablet for tablet d60d571b2c2840a2b20629bb832d1309 (DEFAULT_TABLE table=default.Table_79c2dbec23ad41159b38d5177072d78c [id=d4c09fc79416433b9555ed44abf26bfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:12.413020 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcb7b5b7d63d4f0b906fea97cb548de2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:12.413120 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d60d571b2c2840a2b20629bb832d1309. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:12.416697 904 tablet_service.cc:1468] Processing CreateTablet for tablet 186ec810d02e4748aad15228b5866ac1 (DEFAULT_TABLE table=default.Table_9d0d7a93ff15424092012d4d681b8ae0 [id=aad143eea5c2439299aad87723399c58]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:12.417207 904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 186ec810d02e4748aad15228b5866ac1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:12.420768 1036 tablet_service.cc:1468] Processing CreateTablet for tablet 3df908c3096a4df7bddd8ac88b49cfe0 (DEFAULT_TABLE table=default.Table_7bb211d400ca44d48416f325df223fa6 [id=cc26c89db2484a6fad919231116285f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:12.421247 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3df908c3096a4df7bddd8ac88b49cfe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:12.421283 1332 ts_tablet_manager.cc:1918] T bf302e76746b4f7bbe4469914f83720f P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:12.421440 1332 log.cc:1199] T bf302e76746b4f7bbe4469914f83720f P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/bf302e76746b4f7bbe4469914f83720f
I20250901 01:52:12.421926 1332 ts_tablet_manager.cc:1939] T bf302e76746b4f7bbe4469914f83720f P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:12.422639 1332 tablet_replica.cc:331] T 77405097e68344628f4eda667afaed21 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:12.422734 441 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet bf302e76746b4f7bbe4469914f83720f (table default.Table_3063e0e5f956483db9161270ce39f4b9 [id=c393985e34824ce49877085a5f739e8f]) successfully deleted
I20250901 01:52:12.422843 1332 raft_consensus.cc:2241] T 77405097e68344628f4eda667afaed21 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:12.423030 1332 raft_consensus.cc:2270] T 77405097e68344628f4eda667afaed21 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:12.423596 1332 ts_tablet_manager.cc:1905] T 77405097e68344628f4eda667afaed21 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:12.424782 1035 tablet_service.cc:1468] Processing CreateTablet for tablet d60d571b2c2840a2b20629bb832d1309 (DEFAULT_TABLE table=default.Table_79c2dbec23ad41159b38d5177072d78c [id=d4c09fc79416433b9555ed44abf26bfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:12.425285 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d60d571b2c2840a2b20629bb832d1309. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:12.425958 1031 tablet_service.cc:1468] Processing CreateTablet for tablet 8381305e4c4148808803e255b8a43a90 (DEFAULT_TABLE table=default.Table_de8ae42a0de043129ceab6caec04d192 [id=488699b14a44496987bd0d3b7d64fa5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:12.427573 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 186ec810d02e4748aad15228b5866ac1 (DEFAULT_TABLE table=default.Table_9d0d7a93ff15424092012d4d681b8ae0 [id=aad143eea5c2439299aad87723399c58]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:12.432955 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 186ec810d02e4748aad15228b5866ac1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:12.436652 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8381305e4c4148808803e255b8a43a90. 1 dirs total, 0 dirs full, 0 dirs failed
W20250901 01:52:12.438556 702 kernel_stack_watchdog.cc:198] Thread 757 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 1426ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fe78f073980 (unknown) at ??:0
@ 0x92ce64 __sanitizer::BlockingMutex::Lock() at ??:?
@ 0x932e1b __sanitizer::ThreadRegistry::CreateThread() at ??:?
@ 0x91983b __asan::AsanThread::Create() at ??:?
@ 0x8921ac __interceptor_pthread_create at ??:?
@ 0x7fe78bc459c1 kudu::Thread::StartThread() at ??:0
@ 0xa63f97 kudu::Thread::Create() at ??:?
@ 0x7fe78bc6580b kudu::ThreadPool::CreateThread() at ??:0
@ 0x7fe78bc6122d kudu::ThreadPool::DoSubmit() at ??:0
@ 0x7fe78bc6655f kudu::ThreadPool::Submit() at ??:0
@ 0x7fe79b49d226 kudu::tserver::TSTabletManager::DeleteTabletAsync() at ??:0
@ 0x7fe79b3e7ed0 kudu::tserver::TabletServiceAdminImpl::DeleteTablet() at ??:0
@ 0x7fe7955e6507 kudu::tserver::TabletServerAdminServiceIf::TabletServerAdminServiceIf()::$_3::operator()() at ??:0
@ 0x7fe7955e6236 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fe7929cf2cf std::function<>::operator()() at ??:0
@ 0x7fe7929ce1af kudu::rpc::GeneratedServiceIf::Handle() at ??:0
I20250901 01:52:12.439879 1332 ts_tablet_manager.cc:1918] T 77405097e68344628f4eda667afaed21 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:12.440014 1332 log.cc:1199] T 77405097e68344628f4eda667afaed21 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/77405097e68344628f4eda667afaed21
I20250901 01:52:12.440171 1150 tablet_bootstrap.cc:492] T 467c2bf4a8cb4937b6f627bea28d8ddb P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:12.440441 1332 ts_tablet_manager.cc:1939] T 77405097e68344628f4eda667afaed21 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:12.440441 1150 ts_tablet_manager.cc:1397] T 467c2bf4a8cb4937b6f627bea28d8ddb P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.060s user 0.004s sys 0.000s
I20250901 01:52:12.441542 1150 raft_consensus.cc:357] T 467c2bf4a8cb4937b6f627bea28d8ddb P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:12.441812 1150 raft_consensus.cc:383] T 467c2bf4a8cb4937b6f627bea28d8ddb P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:12.441897 1150 raft_consensus.cc:738] T 467c2bf4a8cb4937b6f627bea28d8ddb P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:12.442186 1150 consensus_queue.cc:260] T 467c2bf4a8cb4937b6f627bea28d8ddb P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:12.445569 752 tablet_service.cc:1468] Processing CreateTablet for tablet 8381305e4c4148808803e255b8a43a90 (DEFAULT_TABLE table=default.Table_de8ae42a0de043129ceab6caec04d192 [id=488699b14a44496987bd0d3b7d64fa5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:12.445876 902 tablet_service.cc:1468] Processing CreateTablet for tablet 8381305e4c4148808803e255b8a43a90 (DEFAULT_TABLE table=default.Table_de8ae42a0de043129ceab6caec04d192 [id=488699b14a44496987bd0d3b7d64fa5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:12.446060 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8381305e4c4148808803e255b8a43a90. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:12.446254 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8381305e4c4148808803e255b8a43a90. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:12.447012 1150 ts_tablet_manager.cc:1428] T 467c2bf4a8cb4937b6f627bea28d8ddb P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250901 01:52:12.447786 1150 tablet_bootstrap.cc:492] T d945b310fe6344af9cc9dd68f84c4bdf P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:12.450788 447 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 2f28451a3ecc481a958588466f7fec77
I20250901 01:52:12.451288 447 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:57810 (request call id 152) took 1544 ms. Trace:
I20250901 01:52:12.451413 447 rpcz_store.cc:276] 0901 01:52:10.907262 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:12.325307 (+1418045us) service_pool.cc:225] Handling call
0901 01:52:12.333687 (+ 8380us) spinlock_profiling.cc:243] Waited 8.11 ms on lock 0x616000040f54. stack: 00007fc53edd57d6 00007fc53edd5590 00007fc53ddc83ad 0000000000972d94 0000000000972d5d 00007fc54dba014d 00007fc54dba0243 00007fc54a85f941 00007fc5523eaa5d 00007fc552574e86 00007fc5493ad466 00007fc5493ad195 00007fc545b832ce 00007fc545b821ae 00007fc545b883e5 00007fc545b88d0f
0901 01:52:12.334090 (+ 403us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0901 01:52:12.334217 (+ 127us) catalog_manager.cc:2790] Modifying in-memory table state
0901 01:52:12.334277 (+ 60us) catalog_manager.cc:2799] Locking tablets
0901 01:52:12.390162 (+ 55885us) catalog_manager.cc:2813] Removing table and tablets from system table
0901 01:52:12.450594 (+ 60432us) catalog_manager.cc:2831] Removing table from by-name map
0901 01:52:12.450689 (+ 95us) catalog_manager.cc:2847] Committing in-memory state
0901 01:52:12.450723 (+ 34us) catalog_manager.cc:2851] Aborting table tasks
0901 01:52:12.451256 (+ 533us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0901 01:52:12.392034 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:12.392164 (+ 130us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:12.392172 (+ 8us) write_op.cc:435] Acquired schema lock
0901 01:52:12.392176 (+ 4us) tablet.cc:660] Decoding operations
0901 01:52:12.392300 (+ 124us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:12.392312 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:12.392318 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
0901 01:52:12.392458 (+ 140us) tablet.cc:699] Row locks acquired
0901 01:52:12.392461 (+ 3us) write_op.cc:260] PREPARE: finished
0901 01:52:12.392494 (+ 33us) write_op.cc:270] Start()
0901 01:52:12.392524 (+ 30us) write_op.cc:276] Timestamp: P: 1756691532392425 usec, L: 0
0901 01:52:12.392528 (+ 4us) op_driver.cc:348] REPLICATION: starting
0901 01:52:12.392699 (+ 171us) log.cc:844] Serialized 725 byte log entry
0901 01:52:12.449323 (+ 56624us) op_driver.cc:464] REPLICATION: finished
0901 01:52:12.449593 (+ 270us) write_op.cc:301] APPLY: starting
0901 01:52:12.449611 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:12.449660 (+ 49us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:12.449691 (+ 31us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:12.449766 (+ 75us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:12.449802 (+ 36us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0901 01:52:12.449810 (+ 8us) write_op.cc:312] APPLY: finished
0901 01:52:12.449897 (+ 87us) log.cc:844] Serialized 29 byte log entry
0901 01:52:12.449964 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:12.449985 (+ 21us) write_op.cc:454] Released schema lock
0901 01:52:12.449991 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0901 01:52:12.450067 (+ 76us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:12.450070 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":18659200,"child_traces":[["op",{"apply.queue_time_us":208,"apply.run_cpu_time_us":466,"apply.run_wall_time_us":466,"mutex_wait_us":72,"num_ops":2,"prepare.queue_time_us":1163,"prepare.run_cpu_time_us":786,"prepare.run_wall_time_us":850,"raft.queue_time_us":922,"raft.run_cpu_time_us":318,"raft.run_wall_time_us":319,"replication_time_us":56764}]]}
I20250901 01:52:12.452528 452 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 6c15cd14831f422eb615038603c79ed0
I20250901 01:52:12.453321 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 2f28451a3ecc481a958588466f7fec77 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:12 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:12.453559 452 catalog_manager.cc:6652] ReplaceTablet: tablet 6c15cd14831f422eb615038603c79ed0 deleted and replaced by tablet 824229183e024feaa975c7f0387c9c1a
I20250901 01:52:12.453649 753 tablet_service.cc:1515] Processing DeleteTablet for tablet 2f28451a3ecc481a958588466f7fec77 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:12 UTC) from {username='slave'} at 127.0.0.1:36178
I20250901 01:52:12.453688 452 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:57810 (request call id 149) took 1658 ms. Trace:
I20250901 01:52:12.453934 754 tablet_service.cc:1515] Processing DeleteTablet for tablet 6c15cd14831f422eb615038603c79ed0 with delete_type TABLET_DATA_DELETED (replaced by tablet 824229183e024feaa975c7f0387c9c1a) from {username='slave'} at 127.0.0.1:36178
I20250901 01:52:12.453758 452 rpcz_store.cc:276] 0901 01:52:10.795336 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:12.308143 (+1512807us) service_pool.cc:225] Handling call
0901 01:52:12.331960 (+ 23817us) spinlock_profiling.cc:243] Waited 23.4 ms on lock 0x616000040f54. stack: 00007fc53edd57d6 00007fc53edd5590 00007fc53ddc83ad 0000000000972d94 0000000000972d5d 00007fc54dba014d 00007fc54dba0243 00007fc54a85f941 00007fc5523eaa5d 00007fc552580bdb 00007fc5493b80e7 00007fc5493b7e15 00007fc545b832ce 00007fc545b821ae 00007fc545b883e5 00007fc545b88d0f
0901 01:52:12.453642 (+121682us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0901 01:52:12.392912 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:12.393011 (+ 99us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:12.393017 (+ 6us) write_op.cc:435] Acquired schema lock
0901 01:52:12.393019 (+ 2us) tablet.cc:660] Decoding operations
0901 01:52:12.393137 (+ 118us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:12.393145 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:12.393149 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0901 01:52:12.393241 (+ 92us) tablet.cc:699] Row locks acquired
0901 01:52:12.393244 (+ 3us) write_op.cc:260] PREPARE: finished
0901 01:52:12.393268 (+ 24us) write_op.cc:270] Start()
0901 01:52:12.393291 (+ 23us) write_op.cc:276] Timestamp: P: 1756691532393203 usec, L: 0
0901 01:52:12.393294 (+ 3us) op_driver.cc:348] REPLICATION: starting
0901 01:52:12.393436 (+ 142us) log.cc:844] Serialized 622 byte log entry
0901 01:52:12.449411 (+ 55975us) op_driver.cc:464] REPLICATION: finished
0901 01:52:12.450134 (+ 723us) write_op.cc:301] APPLY: starting
0901 01:52:12.450143 (+ 9us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:12.450185 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:12.450212 (+ 27us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:12.450289 (+ 77us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:12.450329 (+ 40us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0901 01:52:12.450336 (+ 7us) write_op.cc:312] APPLY: finished
0901 01:52:12.450395 (+ 59us) log.cc:844] Serialized 29 byte log entry
0901 01:52:12.450431 (+ 36us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:12.450445 (+ 14us) write_op.cc:454] Released schema lock
0901 01:52:12.450452 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":53903104,"child_traces":[["op",{"apply.queue_time_us":692,"apply.run_cpu_time_us":394,"apply.run_wall_time_us":394,"num_ops":2,"prepare.queue_time_us":1274,"raft.queue_time_us":1080,"raft.run_cpu_time_us":6,"raft.run_wall_time_us":6,"replication_time_us":56096,"threads_started":1}]]}
W20250901 01:52:12.454386 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 2f28451a3ecc481a958588466f7fec77 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2f28451a3ecc481a958588466f7fec77 already in progress: creating tablet
I20250901 01:52:12.454798 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 6c15cd14831f422eb615038603c79ed0 with delete_type TABLET_DATA_DELETED (replaced by tablet 824229183e024feaa975c7f0387c9c1a) from {username='slave'} at 127.0.0.1:52846
W20250901 01:52:12.455358 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 6c15cd14831f422eb615038603c79ed0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6c15cd14831f422eb615038603c79ed0 already in progress: creating tablet
I20250901 01:52:12.455663 1038 tablet_service.cc:1515] Processing DeleteTablet for tablet 6c15cd14831f422eb615038603c79ed0 with delete_type TABLET_DATA_DELETED (replaced by tablet 824229183e024feaa975c7f0387c9c1a) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:12.457594 1033 tablet_service.cc:1515] Processing DeleteTablet for tablet 2f28451a3ecc481a958588466f7fec77 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:12 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:12.469702 1150 tablet_bootstrap.cc:654] T d945b310fe6344af9cc9dd68f84c4bdf P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:12.473431 1293 raft_consensus.cc:491] T 4fd0e4f9d86c4362af96ce9964b72328 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:12.473618 1293 raft_consensus.cc:513] T 4fd0e4f9d86c4362af96ce9964b72328 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:12.474195 1293 leader_election.cc:290] T 4fd0e4f9d86c4362af96ce9964b72328 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:12.474879 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4fd0e4f9d86c4362af96ce9964b72328" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:12.475392 859 leader_election.cc:343] T 4fd0e4f9d86c4362af96ce9964b72328 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:12.483851 1377 consensus_queue.cc:1035] T 061b760d8e6647c6ab0b5d538840a230 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:12.485982 1150 tablet_bootstrap.cc:492] T d945b310fe6344af9cc9dd68f84c4bdf P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:12.486178 1150 ts_tablet_manager.cc:1397] T d945b310fe6344af9cc9dd68f84c4bdf P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.039s user 0.004s sys 0.000s
I20250901 01:52:12.486502 1332 tablet_replica.cc:331] T 3eaf8baac9f1485f944cd9e74b8b5569 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:12.486629 441 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 77405097e68344628f4eda667afaed21 (table default.Table_f8793d5bb6ca480491a9729761f3f61b [id=bc05464fd80c46a5b4ea19d08d7baf49]) successfully deleted
I20250901 01:52:12.487222 1150 raft_consensus.cc:357] T d945b310fe6344af9cc9dd68f84c4bdf P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:12.487488 1150 raft_consensus.cc:383] T d945b310fe6344af9cc9dd68f84c4bdf P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:12.487587 1150 raft_consensus.cc:738] T d945b310fe6344af9cc9dd68f84c4bdf P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:12.487903 1150 consensus_queue.cc:260] T d945b310fe6344af9cc9dd68f84c4bdf P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:12.490307 1332 raft_consensus.cc:2241] T 3eaf8baac9f1485f944cd9e74b8b5569 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:12.490733 1332 raft_consensus.cc:2270] T 3eaf8baac9f1485f944cd9e74b8b5569 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:12.490839 1150 ts_tablet_manager.cc:1428] T d945b310fe6344af9cc9dd68f84c4bdf P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250901 01:52:12.491532 1150 tablet_bootstrap.cc:492] T 360755d30b5d47ba98c9c7e23ff5ab7e P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:12.491650 1332 ts_tablet_manager.cc:1905] T 3eaf8baac9f1485f944cd9e74b8b5569 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
#6 0x7fe79b477721 in std::default_delete<kudu::tablet::AlterSchemaOpState>::operator()(kudu::tablet::AlterSchemaOpState*) const ../../../include/c++/7.5.0/bits/unique_ptr.h:78:2
#7 0x7fe79b4333cb in std::unique_ptr<kudu::tablet::AlterSchemaOpState, std::default_delete<kudu::tablet::AlterSchemaOpState> >::~unique_ptr() ../../../include/c++/7.5.0/bits/unique_ptr.h:263:4
#8 0x7fe798a1d874 in kudu::tablet::AlterSchemaOp::~AlterSchemaOp() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tablet/ops/alter_schema_op.h:126:7
#9 0x7fe798a1d8b2 in kudu::tablet::AlterSchemaOp::~AlterSchemaOp() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tablet/ops/alter_schema_op.h:126:7
I20250901 01:52:12.523631 1150 tablet_bootstrap.cc:654] T 360755d30b5d47ba98c9c7e23ff5ab7e P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:12.529743 1150 tablet_bootstrap.cc:492] T 360755d30b5d47ba98c9c7e23ff5ab7e P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:12.529934 1150 ts_tablet_manager.cc:1397] T 360755d30b5d47ba98c9c7e23ff5ab7e P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.039s user 0.004s sys 0.000s
I20250901 01:52:12.530907 1150 raft_consensus.cc:357] T 360755d30b5d47ba98c9c7e23ff5ab7e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:12.531164 1150 raft_consensus.cc:383] T 360755d30b5d47ba98c9c7e23ff5ab7e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:12.531244 1150 raft_consensus.cc:738] T 360755d30b5d47ba98c9c7e23ff5ab7e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:12.532051 1150 consensus_queue.cc:260] T 360755d30b5d47ba98c9c7e23ff5ab7e P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:12.533568 1332 ts_tablet_manager.cc:1918] T 3eaf8baac9f1485f944cd9e74b8b5569 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 01:52:12.533759 1332 log.cc:1199] T 3eaf8baac9f1485f944cd9e74b8b5569 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/3eaf8baac9f1485f944cd9e74b8b5569
I20250901 01:52:12.533872 1150 ts_tablet_manager.cc:1428] T 360755d30b5d47ba98c9c7e23ff5ab7e P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20250901 01:52:12.534242 1150 tablet_bootstrap.cc:492] T 2685fb7258264463b5f1ec99cd7d67ad P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:12.534314 1332 ts_tablet_manager.cc:1939] T 3eaf8baac9f1485f944cd9e74b8b5569 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:12.535027 1377 raft_consensus.cc:491] T a4f5685260304d589dd467953d9bebfc P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:12.535223 1377 raft_consensus.cc:513] T a4f5685260304d589dd467953d9bebfc P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } }
I20250901 01:52:12.536039 1377 leader_election.cc:290] T a4f5685260304d589dd467953d9bebfc P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561), 0cf599f74322478d91475df234e58965 (127.0.104.1:40731)
W20250901 01:52:12.537374 992 leader_election.cc:343] T a4f5685260304d589dd467953d9bebfc P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: a4f5685260304d589dd467953d9bebfc
I20250901 01:52:12.538254 1150 tablet_bootstrap.cc:654] T 2685fb7258264463b5f1ec99cd7d67ad P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:12.335369 1151 tablet_bootstrap.cc:654] T f0affc7a78634607852a48458412761d P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:12.547950 1150 tablet_bootstrap.cc:492] T 2685fb7258264463b5f1ec99cd7d67ad P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:12.548146 1150 ts_tablet_manager.cc:1397] T 2685fb7258264463b5f1ec99cd7d67ad P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.002s
I20250901 01:52:12.549331 1150 raft_consensus.cc:357] T 2685fb7258264463b5f1ec99cd7d67ad P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:12.549585 1150 raft_consensus.cc:383] T 2685fb7258264463b5f1ec99cd7d67ad P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:12.549685 1150 raft_consensus.cc:738] T 2685fb7258264463b5f1ec99cd7d67ad P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:12.550001 1150 consensus_queue.cc:260] T 2685fb7258264463b5f1ec99cd7d67ad P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:12.550837 1150 ts_tablet_manager.cc:1428] T 2685fb7258264463b5f1ec99cd7d67ad P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250901 01:52:12.551657 1150 tablet_bootstrap.cc:492] T 6c15cd14831f422eb615038603c79ed0 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:12.553707 1150 tablet_bootstrap.cc:654] T 6c15cd14831f422eb615038603c79ed0 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:12.578312 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_52733c9a363b4ed9b6ff11ad8571e421"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:12.579246 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_52733c9a363b4ed9b6ff11ad8571e421 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:12.580758 447 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_a8b4fe01e7c54c3e95663bced8b3bbe4" } new_table_name: "default.Table_31f776f7dc534acf80b5896fffb00fe1" modify_external_catalogs: true
I20250901 01:52:12.581957 1293 raft_consensus.cc:491] T 467c2bf4a8cb4937b6f627bea28d8ddb P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:12.582139 1293 raft_consensus.cc:513] T 467c2bf4a8cb4937b6f627bea28d8ddb P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:12.582839 1293 leader_election.cc:290] T 467c2bf4a8cb4937b6f627bea28d8ddb P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:12.583642 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "467c2bf4a8cb4937b6f627bea28d8ddb" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:12.584208 859 leader_election.cc:343] T 467c2bf4a8cb4937b6f627bea28d8ddb P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:12.588496 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 88a6ab29d00542aba54f0e4185fbf615
I20250901 01:52:12.588992 451 catalog_manager.cc:6652] ReplaceTablet: tablet 88a6ab29d00542aba54f0e4185fbf615 deleted and replaced by tablet a774cc6ae3924d299127fd9ffb01ca57
I20250901 01:52:12.589104 451 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:57810 (request call id 157) took 1114 ms. Trace:
I20250901 01:52:12.589195 451 rpcz_store.cc:276] 0901 01:52:11.474850 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:12.402906 (+928056us) service_pool.cc:225] Handling call
0901 01:52:12.589074 (+186168us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0901 01:52:12.583010 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:12.583143 (+ 133us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:12.583150 (+ 7us) write_op.cc:435] Acquired schema lock
0901 01:52:12.583155 (+ 5us) tablet.cc:660] Decoding operations
0901 01:52:12.583302 (+ 147us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:12.583320 (+ 18us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:12.583328 (+ 8us) tablet.cc:683] Acquiring locks for 2 operations
0901 01:52:12.583435 (+ 107us) tablet.cc:699] Row locks acquired
0901 01:52:12.583439 (+ 4us) write_op.cc:260] PREPARE: finished
0901 01:52:12.583486 (+ 47us) write_op.cc:270] Start()
0901 01:52:12.583533 (+ 47us) write_op.cc:276] Timestamp: P: 1756691532583460 usec, L: 0
0901 01:52:12.583539 (+ 6us) op_driver.cc:348] REPLICATION: starting
0901 01:52:12.583788 (+ 249us) log.cc:844] Serialized 622 byte log entry
0901 01:52:12.587526 (+ 3738us) op_driver.cc:464] REPLICATION: finished
0901 01:52:12.587728 (+ 202us) write_op.cc:301] APPLY: starting
0901 01:52:12.587746 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:12.587795 (+ 49us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:12.587817 (+ 22us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:12.587934 (+ 117us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:12.587991 (+ 57us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0901 01:52:12.587998 (+ 7us) write_op.cc:312] APPLY: finished
0901 01:52:12.588065 (+ 67us) log.cc:844] Serialized 29 byte log entry
0901 01:52:12.588123 (+ 58us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:12.588138 (+ 15us) write_op.cc:454] Released schema lock
0901 01:52:12.588144 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0901 01:52:12.588212 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:12.588215 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":143,"apply.run_cpu_time_us":479,"apply.run_wall_time_us":480,"num_ops":2,"prepare.queue_time_us":179046,"prepare.run_cpu_time_us":964,"prepare.run_wall_time_us":2047,"raft.queue_time_us":101,"raft.run_cpu_time_us":335,"raft.run_wall_time_us":338,"replication_time_us":3959}]]}
I20250901 01:52:12.590067 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 88a6ab29d00542aba54f0e4185fbf615 with delete_type TABLET_DATA_DELETED (replaced by tablet a774cc6ae3924d299127fd9ffb01ca57) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:12.590938 898 tablet_service.cc:1515] Processing DeleteTablet for tablet 88a6ab29d00542aba54f0e4185fbf615 with delete_type TABLET_DATA_DELETED (replaced by tablet a774cc6ae3924d299127fd9ffb01ca57) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:12.591297 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_3d223b93128649e49a295d940feef90a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:12.591734 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 88a6ab29d00542aba54f0e4185fbf615 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 88a6ab29d00542aba54f0e4185fbf615 already in progress: creating tablet
W20250901 01:52:12.591902 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3d223b93128649e49a295d940feef90a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:12.595974 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 84fe314502444bd88b038b87eead9cb5
I20250901 01:52:12.596585 455 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:57810 (request call id 150) took 1736 ms. Trace:
I20250901 01:52:12.596714 455 rpcz_store.cc:276] 0901 01:52:10.860109 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:12.321707 (+1461598us) service_pool.cc:225] Handling call
0901 01:52:12.381061 (+ 59354us) spinlock_profiling.cc:243] Waited 13.1 ms on lock 0x616000040f54. stack: 00007fc53edd57d6 00007fc53edd5590 00007fc53ddc83ad 0000000000972d94 0000000000972d5d 00007fc54dba014d 00007fc54dba0243 00007fc54a85f941 00007fc5523eaa5d 00007fc552574e86 00007fc5493ad466 00007fc5493ad195 00007fc545b832ce 00007fc545b821ae 00007fc545b883e5 00007fc545b88d0f
0901 01:52:12.383716 (+ 2655us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0901 01:52:12.383936 (+ 220us) catalog_manager.cc:2790] Modifying in-memory table state
0901 01:52:12.384004 (+ 68us) catalog_manager.cc:2799] Locking tablets
0901 01:52:12.408366 (+ 24362us) catalog_manager.cc:2813] Removing table and tablets from system table
0901 01:52:12.595797 (+187431us) catalog_manager.cc:2831] Removing table from by-name map
0901 01:52:12.595846 (+ 49us) catalog_manager.cc:2847] Committing in-memory state
0901 01:52:12.595893 (+ 47us) catalog_manager.cc:2851] Aborting table tasks
0901 01:52:12.596549 (+ 656us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0901 01:52:12.586378 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:12.586473 (+ 95us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:12.586481 (+ 8us) write_op.cc:435] Acquired schema lock
0901 01:52:12.586485 (+ 4us) tablet.cc:660] Decoding operations
0901 01:52:12.586618 (+ 133us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:12.586630 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:12.586638 (+ 8us) tablet.cc:683] Acquiring locks for 2 operations
0901 01:52:12.586758 (+ 120us) tablet.cc:699] Row locks acquired
0901 01:52:12.586762 (+ 4us) write_op.cc:260] PREPARE: finished
0901 01:52:12.586795 (+ 33us) write_op.cc:270] Start()
0901 01:52:12.586828 (+ 33us) write_op.cc:276] Timestamp: P: 1756691532586773 usec, L: 0
0901 01:52:12.586834 (+ 6us) op_driver.cc:348] REPLICATION: starting
0901 01:52:12.587015 (+ 181us) log.cc:844] Serialized 756 byte log entry
0901 01:52:12.594609 (+ 7594us) op_driver.cc:464] REPLICATION: finished
0901 01:52:12.595238 (+ 629us) write_op.cc:301] APPLY: starting
0901 01:52:12.595248 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:12.595280 (+ 32us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:12.595319 (+ 39us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:12.595388 (+ 69us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:12.595414 (+ 26us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0901 01:52:12.595421 (+ 7us) write_op.cc:312] APPLY: finished
0901 01:52:12.595473 (+ 52us) log.cc:844] Serialized 29 byte log entry
0901 01:52:12.595508 (+ 35us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:12.595521 (+ 13us) write_op.cc:454] Released schema lock
0901 01:52:12.595526 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0901 01:52:12.595582 (+ 56us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:12.595585 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":30156672,"child_traces":[["op",{"apply.queue_time_us":597,"apply.run_cpu_time_us":338,"apply.run_wall_time_us":338,"mutex_wait_us":3398,"num_ops":2,"prepare.queue_time_us":177339,"prepare.run_cpu_time_us":825,"prepare.run_wall_time_us":6573,"raft.queue_time_us":3891,"raft.run_cpu_time_us":10,"raft.run_wall_time_us":9,"replication_time_us":7759,"spinlock_wait_cycles":24832}]]}
I20250901 01:52:12.599880 1151 tablet_bootstrap.cc:492] T f0affc7a78634607852a48458412761d P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:12.600100 1151 ts_tablet_manager.cc:1397] T f0affc7a78634607852a48458412761d P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.322s user 0.003s sys 0.002s
I20250901 01:52:12.601104 898 tablet_service.cc:1515] Processing DeleteTablet for tablet 84fe314502444bd88b038b87eead9cb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:12 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:12.601138 1151 raft_consensus.cc:357] T f0affc7a78634607852a48458412761d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } }
I20250901 01:52:12.601747 1151 raft_consensus.cc:383] T f0affc7a78634607852a48458412761d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:12.601824 1151 raft_consensus.cc:738] T f0affc7a78634607852a48458412761d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
W20250901 01:52:12.602010 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 84fe314502444bd88b038b87eead9cb5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 84fe314502444bd88b038b87eead9cb5 already in progress: creating tablet
I20250901 01:52:12.602149 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_c96393e11f3a4cca8202ccc2df6635e0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:12.602385 1151 consensus_queue.cc:260] T f0affc7a78634607852a48458412761d P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } }
W20250901 01:52:12.602727 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c96393e11f3a4cca8202ccc2df6635e0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:12.613139 453 catalog_manager.cc:5582] T 061b760d8e6647c6ab0b5d538840a230 P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } }
I20250901 01:52:12.613633 453 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 061b760d8e6647c6ab0b5d538840a230 (table default.Table_fbba493de02249499d26467bf15c3d5f [id=748daa63137c48a5b151bd34dc5b13fe]). Expected version 1 got 0
I20250901 01:52:12.613971 453 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet b11e8f77243e4764b96548d1132b8e5a (table default.Table_3c7c628f35fb4d02bb79a34995f12938 [id=a39e8b171a874df1890d072732e55df9]). Expected version 2 got 1
I20250901 01:52:12.614225 453 catalog_manager.cc:5582] T 116f65a42ab34bf6b8f8113cb7afdc21 P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: UNKNOWN } } }
I20250901 01:52:12.614563 453 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 116f65a42ab34bf6b8f8113cb7afdc21 (table default.Table_57db59e62bcb45a8b2827445e56dc7de [id=70e493e2ad6b4c988f1d78784f892a35]). Expected version 1 got 0
I20250901 01:52:12.614821 453 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 97e40c318f64455d91b0e4f8866990ce (table default.Table_0c6a1616a7b64455a3ae44c9983dc6c0 [id=9af016a062a64218ad3b62e131d8b65d]). Expected version 2 got 0
I20250901 01:52:12.615008 453 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 24ddc13d1bb04db4a7850acbff091577 (table default.Table_8c1924d21a91425cbbbd8429d56cf0ab [id=18aecbf3fe4643c4b589ee47706161bd]). Expected version 2 got 0
I20250901 01:52:12.617319 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 84fe314502444bd88b038b87eead9cb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:12 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:12.623339 441 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 3eaf8baac9f1485f944cd9e74b8b5569 (table default.Table_947ec0e1a97a40b59be1e234bcaf4e0d [id=5d14706774d345c48ca3575feabb17b0]) successfully deleted
W20250901 01:52:12.624248 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 6c15cd14831f422eb615038603c79ed0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6c15cd14831f422eb615038603c79ed0 already in progress: creating tablet
W20250901 01:52:12.627213 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 2f28451a3ecc481a958588466f7fec77 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2f28451a3ecc481a958588466f7fec77 already in progress: creating tablet
W20250901 01:52:12.629079 447 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 85322a864d154f299f2d670fdd2a7bfb consensus state has no leader
I20250901 01:52:12.636544 1293 raft_consensus.cc:491] T 447ffe73741640ceb73f35676232a094 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:12.636746 1293 raft_consensus.cc:513] T 447ffe73741640ceb73f35676232a094 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } }
I20250901 01:52:12.637423 1293 leader_election.cc:290] T 447ffe73741640ceb73f35676232a094 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447), 0cf599f74322478d91475df234e58965 (127.0.104.1:40731)
I20250901 01:52:12.637992 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "447ffe73741640ceb73f35676232a094" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:12.638885 859 leader_election.cc:343] T 447ffe73741640ceb73f35676232a094 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
W20250901 01:52:12.645599 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 88a6ab29d00542aba54f0e4185fbf615 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 88a6ab29d00542aba54f0e4185fbf615 already in progress: creating tablet
W20250901 01:52:12.646095 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 84fe314502444bd88b038b87eead9cb5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 84fe314502444bd88b038b87eead9cb5 already in progress: creating tablet
I20250901 01:52:12.647233 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_52927bd3bdf4467581d6d963a046e0a7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:12.647871 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_52927bd3bdf4467581d6d963a046e0a7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:12.649121 451 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_ac3b76f15c7743d0900e9a7a00348ec2" } new_table_name: "default.Table_a0e83c6959d64160a5fdec67e0e49ba3" modify_external_catalogs: true
I20250901 01:52:12.650668 455 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_b16d719f7f9041fd9391c5629066b1ad" } modify_external_catalogs: true
I20250901 01:52:12.650853 455 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_b16d719f7f9041fd9391c5629066b1ad" } modify_external_catalogs: true
I20250901 01:52:12.651057 447 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_1c3a507d45d64e389a6c07341b00b9fc" } new_table_name: "default.Table_727db72cb2654b679929453b3a8fb00a" modify_external_catalogs: true
#10 0x7fe7989f8f31 in std::default_delete<kudu::tablet::Op>::operator()(kudu::tablet::Op*) const ../../../include/c++/7.5.0/bits/unique_ptr.h:78:2
I20250901 01:52:12.658107 1150 tablet_bootstrap.cc:492] T 6c15cd14831f422eb615038603c79ed0 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:12.658288 1150 ts_tablet_manager.cc:1397] T 6c15cd14831f422eb615038603c79ed0 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.107s user 0.006s sys 0.028s
I20250901 01:52:12.659081 1150 raft_consensus.cc:357] T 6c15cd14831f422eb615038603c79ed0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:12.659276 1150 raft_consensus.cc:383] T 6c15cd14831f422eb615038603c79ed0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:12.659346 1150 raft_consensus.cc:738] T 6c15cd14831f422eb615038603c79ed0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:12.659621 1150 consensus_queue.cc:260] T 6c15cd14831f422eb615038603c79ed0 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:12.665294 1150 ts_tablet_manager.cc:1428] T 6c15cd14831f422eb615038603c79ed0 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250901 01:52:12.666006 1150 tablet_bootstrap.cc:492] T 0e698923812841b7a5d37d57da103c5c P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:12.666414 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_0e0c69a04fea4f5fab174e11a35bf628"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:12.667119 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0e0c69a04fea4f5fab174e11a35bf628 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250901 01:52:12.668962 451 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 467c2bf4a8cb4937b6f627bea28d8ddb consensus state has no leader
W20250901 01:52:12.670997 447 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7cc1efcb6bff4aba8015d2d3b7f4c923 consensus state has no leader
I20250901 01:52:12.671068 1150 tablet_bootstrap.cc:654] T 0e698923812841b7a5d37d57da103c5c P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
#11 0x7fe7989dc25b in std::unique_ptr<kudu::tablet::Op, std::default_delete<kudu::tablet::Op> >::~unique_ptr() ../../../include/c++/7.5.0/bits/unique_ptr.h:263:4
#12 0x7fe7989febb5 in kudu::tablet::OpDriver::~OpDriver() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tablet/ops/op_driver.h:226:23
#13 0x7fe7989feb65 in void __gnu_cxx::new_allocator<std::shared_ptr<kudu::tablet::OpDriver> enable_make_shared<kudu::tablet::OpDriver>::make_shared<kudu::tablet::OpTracker*, kudu::consensus::RaftConsensus*, kudu::log::Log*, kudu::ThreadPoolToken*, kudu::ThreadPool* const&, kudu::tablet::OpOrderVerifier*, kudu::MonoTime&>(kudu::tablet::OpTracker*&&, kudu::consensus::RaftConsensus*&&, kudu::log::Log*&&, kudu::ThreadPoolToken*&&, kudu::ThreadPool* const&, kudu::tablet::OpOrderVerifier*&&, kudu::MonoTime&)::make_shared_enabler>::destroy<std::shared_ptr<kudu::tablet::OpDriver> enable_make_shared<kudu::tablet::OpDriver>::make_shared<kudu::tablet::OpTracker*, kudu::consensus::RaftConsensus*, kudu::log::Log*, kudu::ThreadPoolToken*, kudu::ThreadPool* const&, kudu::tablet::OpOrderVerifier*, kudu::MonoTime&>(kudu::tablet::OpTracker*&&, kudu::consensus::RaftConsensus*&&, kudu::log::Log*&&, kudu::ThreadPoolToken*&&, kudu::ThreadPool* const&, kudu::tablet::OpOrderVerifier*&&, kudu::MonoTime&)::make_shared_enabler>(std::shared_ptr<kudu::tablet::OpDriver> enable_make_shared<kudu::tablet::OpDriver>::make_shared<kudu::tablet::OpTracker*, kudu::consensus::RaftConsensus*, kudu::log::Log*, kudu::ThreadPoolToken*, kudu::ThreadPool* const&, kudu::tablet::OpOrderVerifier*, kudu::MonoTime&>(kudu::tablet::OpTracker*&&, kudu::consensus::RaftConsensus*&&, kudu::log::Log*&&, kudu::ThreadPoolToken*&&, kudu::ThreadPool* const&, kudu::tablet::OpOrderVerifier*&&, kudu::MonoTime&)::make_shared_enabler*) ../../../include/c++/7.5.0/ext/new_allocator.h:140:28
I20250901 01:52:12.679692 451 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_25b49da03a1843a4ad60c23d2385c80a" } new_table_name: "default.Table_0ad58ab77e93494e8936a824b95886d3" modify_external_catalogs: true
I20250901 01:52:12.682828 1150 tablet_bootstrap.cc:492] T 0e698923812841b7a5d37d57da103c5c P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
W20250901 01:52:12.741494 702 kernel_stack_watchdog.cc:198] Thread 716 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 2738ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fe78f073980 (unknown) at ??:0
@ 0x92ce64 __sanitizer::BlockingMutex::Lock() at ??:?
@ 0x932e1b __sanitizer::ThreadRegistry::CreateThread() at ??:?
@ 0x91983b __asan::AsanThread::Create() at ??:?
@ 0x8921ac __interceptor_pthread_create at ??:?
@ 0x7fe78bc459c1 kudu::Thread::StartThread() at ??:0
@ 0xa63f97 kudu::Thread::Create() at ??:?
@ 0x7fe78bc6580b kudu::ThreadPool::CreateThread() at ??:0
@ 0x7fe78bc6122d kudu::ThreadPool::DoSubmit() at ??:0
@ 0x7fe78bc6655f kudu::ThreadPool::Submit() at ??:0
@ 0x7fe792947bb2 kudu::rpc::ReactorThread::StartConnectionNegotiation() at ??:0
@ 0x7fe7929492d1 kudu::rpc::ReactorThread::FindOrStartConnection() at ??:0
@ 0x7fe792948ad2 kudu::rpc::ReactorThread::AssignOutboundCall() at ??:0
@ 0x7fe79295842e kudu::rpc::AssignOutboundCallTask::Run() at ??:0
@ 0x7fe792946bcd kudu::rpc::ReactorThread::AsyncHandler() at ??:0
@ 0x7fe792958f45 ev::base<>::method_thunk<>() at ??:0
I20250901 01:52:12.689755 1151 ts_tablet_manager.cc:1428] T f0affc7a78634607852a48458412761d P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.090s user 0.003s sys 0.000s
I20250901 01:52:12.774574 1151 tablet_bootstrap.cc:492] T 447ffe73741640ceb73f35676232a094 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:12.702952 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 6c05eff658924ca4a9a1e7a37f917d67 (DEFAULT_TABLE table=default.Table_c1959bda12f14bc9995d832476abf670 [id=17355a61629849dd9d8228a469ab744f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:12.775954 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c05eff658924ca4a9a1e7a37f917d67. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:12.742390 1255 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:46333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:43871,127.0.104.61:42375,127.0.104.60:44231
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:44231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.104.60
--webserver_port=46373
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type FASTDEBUG
built by None at 01 Sep 2025 01:43:20 UTC on 24a791456cd2
build id 7840
ASAN enabled
I20250901 01:52:12.778695 1255 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
#14 0x7fe7989feb20 in void std::allocator_traits<std::allocator<std::shared_ptr<kudu::tablet::OpDriver> enable_make_shared<kudu::tablet::OpDriver>::make_shared<kudu::tablet::OpTracker*, kudu::consensus::RaftConsensus*, kudu::log::Log*, kudu::ThreadPoolToken*, kudu::ThreadPool* const&, kudu::tablet::OpOrderVerifier*, kudu::MonoTime&>(kudu::tablet::OpTracker*&&, kudu::consensus::RaftConsensus*&&, kudu::log::Log*&&, kudu::ThreadPoolToken*&&, kudu::ThreadPool* const&, kudu::tablet::OpOrderVerifier*&&, kudu::MonoTime&)::make_shared_enabler> >::destroy<std::shared_ptr<kudu::tablet::OpDriver> enable_make_shared<kudu::tablet::OpDriver>::make_shared<kudu::tablet::OpTracker*, kudu::consensus::RaftConsensus*, kudu::log::Log*, kudu::ThreadPoolToken*, kudu::ThreadPool* const&, kudu::tablet::OpOrderVerifier*, kudu::MonoTime&>(kudu::tablet::OpTracker*&&, kudu::consensus::RaftConsensus*&&, kudu::log::Log*&&, kudu::ThreadPoolToken*&&, kudu::ThreadPool* const&, kudu::tablet::OpOrderVerifier*&&, kudu::MonoTime&)::make_shared_enabler>(std::allocator<std::shared_ptr<kudu::tablet::OpDriver> enable_make_shared<kudu::tablet::OpDriver>::make_shared<kudu::tablet::OpTracker*, kudu::consensus::RaftConsensus*, kudu::log::Log*, kudu::ThreadPoolToken*, kudu::ThreadPool* const&, kudu::tablet::OpOrderVerifier*, kudu::MonoTime&>(kudu::tablet::OpTracker*&&, kudu::consensus::RaftConsensus*&&, kudu::log::Log*&&, kudu::ThreadPoolToken*&&, kudu::ThreadPool* const&, kudu::tablet::OpOrderVerifier*&&, kudu::MonoTime&)::make_shared_enabler>&, std::shared_ptr<kudu::tablet::OpDriver> enable_make_shared<kudu::tablet::OpDriver>::make_shared<kudu::tablet::OpTracker*, kudu::consensus::RaftConsensus*, kudu::log::Log*, kudu::ThreadPoolToken*, kudu::ThreadPool* const&, kudu::tablet::OpOrderVerifier*, kudu::MonoTime&>(kudu::tablet::OpTracker*&&, kudu::consensus::RaftConsensus*&&, kudu::log::Log*&&, kudu::ThreadPoolToken*&&, kudu::ThreadPool* const&, kudu::tablet::OpOrderVerifier*&&, kudu::MonoTime&)::make_shared_enabler*) ../../../include/c++/7.5.0/bits/alloc_traits.h:487:8
I20250901 01:52:12.681802 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_117c6c43b552476dbb8b7b5ccdce47b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:12.703377 1031 tablet_service.cc:1468] Processing CreateTablet for tablet 5adf6b492640417ca071c94702b48cb9 (DEFAULT_TABLE table=default.Table_947ec0e1a97a40b59be1e234bcaf4e0d [id=5d14706774d345c48ca3575feabb17b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:12.780977 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5adf6b492640417ca071c94702b48cb9. 1 dirs total, 0 dirs full, 0 dirs failed
W20250901 01:52:12.781083 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_117c6c43b552476dbb8b7b5ccdce47b7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:12.683476 1150 ts_tablet_manager.cc:1397] T 0e698923812841b7a5d37d57da103c5c P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.003s
W20250901 01:52:12.691643 859 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250901 01:52:12.699864 1293 raft_consensus.cc:491] T b29e846dfae44f0e9ce3e5fea3763a4f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:12.782564 1293 raft_consensus.cc:513] T b29e846dfae44f0e9ce3e5fea3763a4f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:12.783144 1150 raft_consensus.cc:357] T 0e698923812841b7a5d37d57da103c5c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:12.783334 1150 raft_consensus.cc:383] T 0e698923812841b7a5d37d57da103c5c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:12.783438 1150 raft_consensus.cc:738] T 0e698923812841b7a5d37d57da103c5c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:12.783471 1293 leader_election.cc:290] T b29e846dfae44f0e9ce3e5fea3763a4f P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:12.701615 898 tablet_service.cc:1468] Processing CreateTablet for tablet 6c05eff658924ca4a9a1e7a37f917d67 (DEFAULT_TABLE table=default.Table_c1959bda12f14bc9995d832476abf670 [id=17355a61629849dd9d8228a469ab744f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:12.783664 1150 consensus_queue.cc:260] T 0e698923812841b7a5d37d57da103c5c P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:12.783993 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c05eff658924ca4a9a1e7a37f917d67. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:12.784188 1150 ts_tablet_manager.cc:1428] T 0e698923812841b7a5d37d57da103c5c P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250901 01:52:12.784541 1150 tablet_bootstrap.cc:492] T 85322a864d154f299f2d670fdd2a7bfb P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:12.703759 1035 tablet_service.cc:1468] Processing CreateTablet for tablet a59ac57aebe24860aa02f53ce0485910 (DEFAULT_TABLE table=default.Table_c2829420d635461ea141ba803c71bdf9 [id=b8f69313439543be9c11eea88dfa69c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:12.785456 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a59ac57aebe24860aa02f53ce0485910. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:12.786302 1150 tablet_bootstrap.cc:654] T 85322a864d154f299f2d670fdd2a7bfb P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:12.694329 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_8a4599f08716440e947a2d356d6363a5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:12.787750 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8a4599f08716440e947a2d356d6363a5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:12.702071 904 tablet_service.cc:1468] Processing CreateTablet for tablet a59ac57aebe24860aa02f53ce0485910 (DEFAULT_TABLE table=default.Table_c2829420d635461ea141ba803c71bdf9 [id=b8f69313439543be9c11eea88dfa69c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:12.788801 904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a59ac57aebe24860aa02f53ce0485910. 1 dirs total, 0 dirs full, 0 dirs failed
W20250901 01:52:12.695883 451 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fcb7b5b7d63d4f0b906fea97cb548de2 consensus state has no leader
I20250901 01:52:12.789414 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_3a7da8de6b044deda677302fd0c530ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:12.789889 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3a7da8de6b044deda677302fd0c530ea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:12.698727 450 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 116f65a42ab34bf6b8f8113cb7afdc21 (table default.Table_57db59e62bcb45a8b2827445e56dc7de [id=70e493e2ad6b4c988f1d78784f892a35]). Expected version 1 got 0
I20250901 01:52:12.791149 450 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 97e40c318f64455d91b0e4f8866990ce (table default.Table_0c6a1616a7b64455a3ae44c9983dc6c0 [id=9af016a062a64218ad3b62e131d8b65d]). Expected version 2 got 0
I20250901 01:52:12.791392 450 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet f0affc7a78634607852a48458412761d (table default.Table_b7bffcbb78614d3cad2b1d375d7a97cc [id=535cf5f83c56443d93935ce80ff87efb]). Expected version 1 got 0
I20250901 01:52:12.791548 450 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 061b760d8e6647c6ab0b5d538840a230 (table default.Table_fbba493de02249499d26467bf15c3d5f [id=748daa63137c48a5b151bd34dc5b13fe]). Expected version 1 got 0
I20250901 01:52:12.791770 450 catalog_manager.cc:5582] T 24ddc13d1bb04db4a7850acbff091577 P 00c282f779c1477d920f24dc0dc16628 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00c282f779c1477d920f24dc0dc16628 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "00c282f779c1477d920f24dc0dc16628" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } }
I20250901 01:52:12.792197 450 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 24ddc13d1bb04db4a7850acbff091577 (table default.Table_8c1924d21a91425cbbbd8429d56cf0ab [id=18aecbf3fe4643c4b589ee47706161bd]). Expected version 2 got 0
I20250901 01:52:12.792474 450 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet b11e8f77243e4764b96548d1132b8e5a (table default.Table_3c7c628f35fb4d02bb79a34995f12938 [id=a39e8b171a874df1890d072732e55df9]). Expected version 2 got 1
W20250901 01:52:12.793296 446 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet eeae5294bd474f368f5766c260ccf3fe consensus state has no leader
I20250901 01:52:12.793452 446 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:57810 (request call id 156) took 1318 ms. Trace:
I20250901 01:52:12.793543 446 rpcz_store.cc:276] 0901 01:52:11.474657 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:12.379535 (+904878us) service_pool.cc:225] Handling call
0901 01:52:12.386683 (+ 7148us) spinlock_profiling.cc:243] Waited 5.06 ms on lock 0x616000040f54. stack: 00007fc53edd57d6 00007fc53edd5590 00007fc53ddc83ad 0000000000972d94 0000000000972d5d 00007fc54dba014d 00007fc54dba0243 00007fc54a85f941 00007fc5523eaa5d 00007fc552576126 00007fc5493af126 00007fc5493aee55 00007fc545b832ce 00007fc545b821ae 00007fc545b883e5 00007fc545b88d0f
0901 01:52:12.443810 (+ 57127us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0901 01:52:12.444022 (+ 212us) catalog_manager.cc:3683] Apply alter schema
0901 01:52:12.444224 (+ 202us) catalog_manager.cc:3715] Acquired catalog manager lock
0901 01:52:12.444306 (+ 82us) catalog_manager.cc:3873] Updating metadata on disk
0901 01:52:12.622202 (+177896us) catalog_manager.cc:3904] Committing alterations to in-memory state
0901 01:52:12.793421 (+171219us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0901 01:52:12.595069 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:12.595201 (+ 132us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:12.595209 (+ 8us) write_op.cc:435] Acquired schema lock
0901 01:52:12.595213 (+ 4us) tablet.cc:660] Decoding operations
0901 01:52:12.595289 (+ 76us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:12.595296 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:12.595304 (+ 8us) tablet.cc:683] Acquiring locks for 1 operations
0901 01:52:12.595357 (+ 53us) tablet.cc:699] Row locks acquired
0901 01:52:12.595360 (+ 3us) write_op.cc:260] PREPARE: finished
0901 01:52:12.595387 (+ 27us) write_op.cc:270] Start()
0901 01:52:12.595412 (+ 25us) write_op.cc:276] Timestamp: P: 1756691532595367 usec, L: 0
0901 01:52:12.595415 (+ 3us) op_driver.cc:348] REPLICATION: starting
0901 01:52:12.601866 (+ 6451us) log.cc:844] Serialized 405 byte log entry
0901 01:52:12.604096 (+ 2230us) spinlock_profiling.cc:243] Waited 6.27 ms on lock 0x616000040f54. stack: 00007fc53edd57d6 00007fc53edd5590 00007fc53ddc83ad 0000000000972d94 0000000000972d5d 000000000097272a 00007fc54a831d62 00007fc54bbda2d5 00007fc54bbd8df9 00007fc54bbe1220 00007fc54bbe102d 0000000000a774dc 00007fc53ee1bb5f 00007fc53ee1ee2b 00007fc53ee1eca8 0000000000a774dc
0901 01:52:12.621115 (+ 17019us) op_driver.cc:464] REPLICATION: finished
0901 01:52:12.621621 (+ 506us) write_op.cc:301] APPLY: starting
0901 01:52:12.621645 (+ 24us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:12.621705 (+ 60us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:12.621710 (+ 5us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:12.621779 (+ 69us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:12.621827 (+ 48us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0901 01:52:12.621839 (+ 12us) write_op.cc:312] APPLY: finished
0901 01:52:12.621941 (+ 102us) log.cc:844] Serialized 23 byte log entry
0901 01:52:12.622020 (+ 79us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:12.622043 (+ 23us) write_op.cc:454] Released schema lock
0901 01:52:12.622052 (+ 9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":11627392,"child_traces":[["op",{"apply.queue_time_us":423,"apply.run_cpu_time_us":507,"apply.run_wall_time_us":506,"num_ops":1,"prepare.queue_time_us":146207,"prepare.run_cpu_time_us":989,"prepare.run_wall_time_us":9092,"raft.queue_time_us":190,"raft.run_cpu_time_us":25,"raft.run_wall_time_us":27,"replication_time_us":25633,"spinlock_wait_cycles":14420352}]]}
I20250901 01:52:12.794304 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_30943b75394441a5a55b195055d1e0cc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:12.794899 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_30943b75394441a5a55b195055d1e0cc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:12.706430 1036 tablet_service.cc:1468] Processing CreateTablet for tablet 154574e17f214cc0ad56e6e6be3af899 (DEFAULT_TABLE table=default.Table_cb6a9252a90d4184b917db4c7edc06b1 [id=b550b0e6553e41a2a20ea3bf23e1c7a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:12.795888 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 154574e17f214cc0ad56e6e6be3af899. 1 dirs total, 0 dirs full, 0 dirs failed
#15 0x7fe7989fe4b2 in std::_Sp_counted_ptr_inplace<std::shared_ptr<kudu::tablet::OpDriver> enable_make_shared<kudu::tablet::OpDriver>::make_shared<kudu::tablet::OpTracker*, kudu::consensus::RaftConsensus*, kudu::log::Log*, kudu::ThreadPoolToken*, kudu::ThreadPool* const&, kudu::tablet::OpOrderVerifier*, kudu::MonoTime&>(kudu::tablet::OpTracker*&&, kudu::consensus::RaftConsensus*&&, kudu::log::Log*&&, kudu::ThreadPoolToken*&&, kudu::ThreadPool* const&, kudu::tablet::OpOrderVerifier*&&, kudu::MonoTime&)::make_shared_enabler, std::allocator<std::shared_ptr<kudu::tablet::OpDriver> enable_make_shared<kudu::tablet::OpDriver>::make_shared<kudu::tablet::OpTracker*, kudu::consensus::RaftConsensus*, kudu::log::Log*, kudu::ThreadPoolToken*, kudu::ThreadPool* const&, kudu::tablet::OpOrderVerifier*, kudu::MonoTime&>(kudu::tablet::OpTracker*&&, kudu::consensus::RaftConsensus*&&, kudu::log::Log*&&, kudu::ThreadPoolToken*&&, kudu::ThreadPool* const&, kudu::tablet::OpOrderVerifier*&&, kudu::MonoTime&)::make_shared_enabler>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() ../../../include/c++/7.5.0/bits/shared_ptr_base.h:535:2
W20250901 01:52:12.796747 858 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.104.1:40731 timed out after 3.219s (SENT)
I20250901 01:52:12.702474 897 tablet_service.cc:1468] Processing CreateTablet for tablet 5adf6b492640417ca071c94702b48cb9 (DEFAULT_TABLE table=default.Table_947ec0e1a97a40b59be1e234bcaf4e0d [id=5d14706774d345c48ca3575feabb17b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:12.800866 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5adf6b492640417ca071c94702b48cb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:12.804630 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 8d5b10ede9c84be098795c0a787e5a78
I20250901 01:52:12.805063 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_d6119d1604674319914fa8cb698f16c5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:12.805215 448 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:57810 (request call id 151) took 1939 ms. Trace:
W20250901 01:52:12.805687 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d6119d1604674319914fa8cb698f16c5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:12.805836 449 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet c445c300d71643609071f0f36528a9c4
I20250901 01:52:12.806310 1151 tablet_bootstrap.cc:654] T 447ffe73741640ceb73f35676232a094 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:12.806423 449 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:57810 (request call id 155) took 1352 ms. Trace:
I20250901 01:52:12.806557 449 rpcz_store.cc:276] 0901 01:52:11.453945 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:12.329794 (+875849us) service_pool.cc:225] Handling call
0901 01:52:12.385020 (+ 55226us) spinlock_profiling.cc:243] Waited 51.3 ms on lock 0x616000040f54. stack: 00007fc53edd57d6 00007fc53edd5590 00007fc53ddc83ad 0000000000972d94 0000000000972d5d 00007fc54dba014d 00007fc54dba0243 00007fc54a85f941 00007fc5523eaa5d 00007fc552574e86 00007fc5493ad466 00007fc5493ad195 00007fc545b832ce 00007fc545b821ae 00007fc545b883e5 00007fc545b88d0f
0901 01:52:12.386264 (+ 1244us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0901 01:52:12.441428 (+ 55164us) catalog_manager.cc:2790] Modifying in-memory table state
0901 01:52:12.441525 (+ 97us) catalog_manager.cc:2799] Locking tablets
0901 01:52:12.441731 (+ 206us) catalog_manager.cc:2813] Removing table and tablets from system table
0901 01:52:12.612081 (+170350us) catalog_manager.cc:2831] Removing table from by-name map
0901 01:52:12.612195 (+ 114us) catalog_manager.cc:2847] Committing in-memory state
0901 01:52:12.805764 (+193569us) catalog_manager.cc:2851] Aborting table tasks
0901 01:52:12.806382 (+ 618us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0901 01:52:12.592986 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:12.593076 (+ 90us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:12.593080 (+ 4us) write_op.cc:435] Acquired schema lock
0901 01:52:12.593082 (+ 2us) tablet.cc:660] Decoding operations
0901 01:52:12.593184 (+ 102us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:12.593193 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:12.593198 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0901 01:52:12.593283 (+ 85us) tablet.cc:699] Row locks acquired
0901 01:52:12.593285 (+ 2us) write_op.cc:260] PREPARE: finished
0901 01:52:12.593318 (+ 33us) write_op.cc:270] Start()
0901 01:52:12.593356 (+ 38us) write_op.cc:276] Timestamp: P: 1756691532593295 usec, L: 0
0901 01:52:12.593360 (+ 4us) op_driver.cc:348] REPLICATION: starting
0901 01:52:12.593507 (+ 147us) log.cc:844] Serialized 759 byte log entry
0901 01:52:12.611242 (+ 17735us) op_driver.cc:464] REPLICATION: finished
0901 01:52:12.611438 (+ 196us) write_op.cc:301] APPLY: starting
0901 01:52:12.611456 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:12.611511 (+ 55us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:12.611555 (+ 44us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:12.611634 (+ 79us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:12.611680 (+ 46us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0901 01:52:12.611692 (+ 12us) write_op.cc:312] APPLY: finished
0901 01:52:12.611777 (+ 85us) log.cc:844] Serialized 29 byte log entry
0901 01:52:12.611848 (+ 71us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:12.611869 (+ 21us) write_op.cc:454] Released schema lock
0901 01:52:12.611879 (+ 10us) write_op.cc:341] FINISH: Updating metrics
0901 01:52:12.611959 (+ 80us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:12.611965 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":117948288,"child_traces":[["op",{"apply.queue_time_us":138,"apply.run_cpu_time_us":508,"apply.run_wall_time_us":508,"num_ops":2,"prepare.queue_time_us":150264,"prepare.run_cpu_time_us":635,"prepare.run_wall_time_us":2011,"raft.queue_time_us":657,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":6,"replication_time_us":17845}]]}
I20250901 01:52:12.807469 449 master_service.cc:948] ReplaceTablet: received request to replace tablet 3df908c3096a4df7bddd8ac88b49cfe0 from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:12.805380 448 rpcz_store.cc:276] 0901 01:52:10.866228 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:12.324827 (+1458599us) service_pool.cc:225] Handling call
0901 01:52:12.332182 (+ 7355us) spinlock_profiling.cc:243] Waited 7.14 ms on lock 0x616000040f54. stack: 00007fc53edd57d6 00007fc53edd5590 00007fc53ddc83ad 0000000000972d94 0000000000972d5d 00007fc54dba014d 00007fc54dba0243 00007fc54a85f941 00007fc5523eaa5d 00007fc552574e86 00007fc5493ad466 00007fc5493ad195 00007fc545b832ce 00007fc545b821ae 00007fc545b883e5 00007fc545b88d0f
0901 01:52:12.332618 (+ 436us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0901 01:52:12.332766 (+ 148us) catalog_manager.cc:2790] Modifying in-memory table state
0901 01:52:12.332853 (+ 87us) catalog_manager.cc:2799] Locking tablets
0901 01:52:12.405381 (+ 72528us) catalog_manager.cc:2813] Removing table and tablets from system table
0901 01:52:12.595681 (+190300us) catalog_manager.cc:2831] Removing table from by-name map
0901 01:52:12.595760 (+ 79us) catalog_manager.cc:2847] Committing in-memory state
0901 01:52:12.804544 (+208784us) catalog_manager.cc:2851] Aborting table tasks
0901 01:52:12.805147 (+ 603us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0901 01:52:12.585107 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:12.585274 (+ 167us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:12.585282 (+ 8us) write_op.cc:435] Acquired schema lock
0901 01:52:12.585287 (+ 5us) tablet.cc:660] Decoding operations
0901 01:52:12.585421 (+ 134us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:12.585435 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:12.585443 (+ 8us) tablet.cc:683] Acquiring locks for 2 operations
0901 01:52:12.585647 (+ 204us) tablet.cc:699] Row locks acquired
0901 01:52:12.585652 (+ 5us) write_op.cc:260] PREPARE: finished
0901 01:52:12.585697 (+ 45us) write_op.cc:270] Start()
0901 01:52:12.585740 (+ 43us) write_op.cc:276] Timestamp: P: 1756691532585673 usec, L: 0
0901 01:52:12.585746 (+ 6us) op_driver.cc:348] REPLICATION: starting
0901 01:52:12.585944 (+ 198us) log.cc:844] Serialized 756 byte log entry
0901 01:52:12.594539 (+ 8595us) op_driver.cc:464] REPLICATION: finished
0901 01:52:12.594781 (+ 242us) write_op.cc:301] APPLY: starting
0901 01:52:12.594793 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:12.594832 (+ 39us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:12.594856 (+ 24us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:12.594915 (+ 59us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:12.594947 (+ 32us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0901 01:52:12.594955 (+ 8us) write_op.cc:312] APPLY: finished
0901 01:52:12.595022 (+ 67us) log.cc:844] Serialized 29 byte log entry
0901 01:52:12.595066 (+ 44us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:12.595083 (+ 17us) write_op.cc:454] Released schema lock
0901 01:52:12.595093 (+ 10us) write_op.cc:341] FINISH: Updating metrics
0901 01:52:12.595166 (+ 73us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:12.595171 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":16428416,"child_traces":[["op",{"apply.queue_time_us":194,"apply.run_cpu_time_us":374,"apply.run_wall_time_us":373,"num_ops":2,"prepare.queue_time_us":179010,"prepare.run_cpu_time_us":886,"prepare.run_wall_time_us":1261,"raft.queue_time_us":33,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":5,"replication_time_us":8761}]]}
I20250901 01:52:12.808444 448 master_service.cc:948] ReplaceTablet: received request to replace tablet 8d5b10ede9c84be098795c0a787e5a78 from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:12.808548 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet ce2a1ba75d674c95bdba9d6f12273bee
I20250901 01:52:12.809126 447 master_service.cc:948] ReplaceTablet: received request to replace tablet 0e698923812841b7a5d37d57da103c5c from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:12.809410 455 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_57db59e62bcb45a8b2827445e56dc7de" } modify_external_catalogs: true
I20250901 01:52:12.809654 447 master_service.cc:948] ReplaceTablet: received request to replace tablet 97e40c318f64455d91b0e4f8866990ce from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:12.809628 455 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_57db59e62bcb45a8b2827445e56dc7de" } modify_external_catalogs: true
I20250901 01:52:12.709187 1033 tablet_service.cc:1468] Processing CreateTablet for tablet a7ff996482164776b468cf305989866b (DEFAULT_TABLE table=default.Table_3c7fe20853f245469347813ac5161b25 [id=37883dea90384a2a8aa832201d66b7b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:12.810745 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7ff996482164776b468cf305989866b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:12.705070 899 tablet_service.cc:1468] Processing CreateTablet for tablet 154574e17f214cc0ad56e6e6be3af899 (DEFAULT_TABLE table=default.Table_cb6a9252a90d4184b917db4c7edc06b1 [id=b550b0e6553e41a2a20ea3bf23e1c7a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:12.815562 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 154574e17f214cc0ad56e6e6be3af899. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:12.816026 1255 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250901 01:52:12.712805 1038 tablet_service.cc:1468] Processing CreateTablet for tablet dd694cfa09564d3da5d279c0861a4c32 (DEFAULT_TABLE table=default.Table_7183dbd577624461af0f861fa4f63d6d [id=a785d19b66e546b5901b6d7e93106161]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:12.819867 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd694cfa09564d3da5d279c0861a4c32. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:12.707974 902 tablet_service.cc:1468] Processing CreateTablet for tablet a7ff996482164776b468cf305989866b (DEFAULT_TABLE table=default.Table_3c7fe20853f245469347813ac5161b25 [id=37883dea90384a2a8aa832201d66b7b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:12.823792 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7ff996482164776b468cf305989866b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:12.826256 451 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_7183dbd577624461af0f861fa4f63d6d" } modify_external_catalogs: true
I20250901 01:52:12.826534 451 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_7183dbd577624461af0f861fa4f63d6d" } modify_external_catalogs: true
I20250901 01:52:12.828637 446 rpcz_store.cc:275] Call kudu.master.MasterService.GetMasterRegistration from 127.0.0.1:53712 (request call id 2) took 1578 ms. Trace:
I20250901 01:52:12.715579 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 60285f2ba037433999f5455fb9cc7acd (DEFAULT_TABLE table=default.Table_b582d9a2134b4184a195cfe0a8335a33 [id=cdf183c0a78b473b8d3428663415ab8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:12.828776 446 rpcz_store.cc:276] 0901 01:52:11.250260 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:12.819909 (+1569649us) service_pool.cc:225] Handling call
0901 01:52:12.828369 (+ 8460us) spinlock_profiling.cc:243] Waited 8.11 ms on lock 0x616000040f54. stack: 00007fc53edd57d6 00007fc53edd5590 00007fc53ddc83ad 0000000000972d94 0000000000972d5d 00007fc54dba014d 00007fc54dba0243 00007fc54a85f941 00007fc5523eaa5d 00007fc55257c595 00007fc5493bd727 00007fc5493bd455 00007fc545b832ce 00007fc545b821ae 00007fc545b883e5 00007fc545b88d0f
0901 01:52:12.828582 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":18663680}
I20250901 01:52:12.829316 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60285f2ba037433999f5455fb9cc7acd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:12.829458 448 rpcz_store.cc:275] Call kudu.master.MasterService.GetMasterRegistration from 127.0.0.1:53726 (request call id 11) took 1726 ms. Trace:
I20250901 01:52:12.829555 448 rpcz_store.cc:276] 0901 01:52:11.102790 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:12.818944 (+1716154us) service_pool.cc:225] Handling call
0901 01:52:12.829285 (+ 10341us) spinlock_profiling.cc:243] Waited 10.1 ms on lock 0x616000040f54. stack: 00007fc53edd57d6 00007fc53edd5590 00007fc53ddc83ad 0000000000972d94 0000000000972d5d 00007fc54dba014d 00007fc54dba0243 00007fc54a85f941 00007fc5523eaa5d 00007fc55257c595 00007fc5493bd727 00007fc5493bd455 00007fc545b832ce 00007fc545b821ae 00007fc545b883e5 00007fc545b88d0f
0901 01:52:12.829400 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":23210240}
I20250901 01:52:12.831269 448 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_3c7fe20853f245469347813ac5161b25" } new_table_name: "default.Table_daa5affe30374472879ede5e985a529d" modify_external_catalogs: true
I20250901 01:52:12.710728 894 tablet_service.cc:1468] Processing CreateTablet for tablet dd694cfa09564d3da5d279c0861a4c32 (DEFAULT_TABLE table=default.Table_7183dbd577624461af0f861fa4f63d6d [id=a785d19b66e546b5901b6d7e93106161]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:12.833010 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd694cfa09564d3da5d279c0861a4c32. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:12.833801 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_fb2584edbb714913a6766458dffef25f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:12.834357 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fb2584edbb714913a6766458dffef25f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250901 01:52:12.764544 991 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250901 01:52:12.714404 903 tablet_service.cc:1468] Processing CreateTablet for tablet 60285f2ba037433999f5455fb9cc7acd (DEFAULT_TABLE table=default.Table_b582d9a2134b4184a195cfe0a8335a33 [id=cdf183c0a78b473b8d3428663415ab8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:12.836987 903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60285f2ba037433999f5455fb9cc7acd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:12.752954 922 raft_consensus.cc:1273] T 116f65a42ab34bf6b8f8113cb7afdc21 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Refusing update from remote peer fd27ab6665304aaea53adbd102dd4f58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250901 01:52:12.842833 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_2f5cfcd78bb0468385bf77c14de3583a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:12.843492 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2f5cfcd78bb0468385bf77c14de3583a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:12.844762 1293 raft_consensus.cc:491] T 6c15cd14831f422eb615038603c79ed0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:12.844955 1293 raft_consensus.cc:513] T 6c15cd14831f422eb615038603c79ed0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:12.845729 1293 leader_election.cc:290] T 6c15cd14831f422eb615038603c79ed0 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:12.845741 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet c445c300d71643609071f0f36528a9c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:12 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:12.845913 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b29e846dfae44f0e9ce3e5fea3763a4f" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:12.846028 1332 tablet_replica.cc:331] T c445c300d71643609071f0f36528a9c4 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:12.846253 1332 raft_consensus.cc:2241] T c445c300d71643609071f0f36528a9c4 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:12.846556 1332 raft_consensus.cc:2270] T c445c300d71643609071f0f36528a9c4 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:12.847361 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 8d5b10ede9c84be098795c0a787e5a78 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:12 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:12.847545 901 tablet_service.cc:1515] Processing DeleteTablet for tablet ce2a1ba75d674c95bdba9d6f12273bee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:12 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:12.847399 896 tablet_service.cc:1515] Processing DeleteTablet for tablet c445c300d71643609071f0f36528a9c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:12 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:12.848220 1306 tablet_replica.cc:331] T 8d5b10ede9c84be098795c0a787e5a78 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:12.848196 1332 ts_tablet_manager.cc:1905] T c445c300d71643609071f0f36528a9c4 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:12.848455 1306 raft_consensus.cc:2241] T 8d5b10ede9c84be098795c0a787e5a78 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:12.848652 1306 raft_consensus.cc:2270] T 8d5b10ede9c84be098795c0a787e5a78 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:12.849136 1306 ts_tablet_manager.cc:1905] T 8d5b10ede9c84be098795c0a787e5a78 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:12.849390 1374 consensus_queue.cc:1035] T 116f65a42ab34bf6b8f8113cb7afdc21 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:12.850001 1374 consensus_queue.cc:1035] T 97e40c318f64455d91b0e4f8866990ce P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:12.850641 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c15cd14831f422eb615038603c79ed0" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:12.852685 859 leader_election.cc:343] T b29e846dfae44f0e9ce3e5fea3763a4f P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
W20250901 01:52:12.855729 859 leader_election.cc:343] T 6c15cd14831f422eb615038603c79ed0 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:12.856184 1332 ts_tablet_manager.cc:1918] T c445c300d71643609071f0f36528a9c4 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:12.856300 1332 log.cc:1199] T c445c300d71643609071f0f36528a9c4 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/c445c300d71643609071f0f36528a9c4
I20250901 01:52:12.856580 1035 tablet_service.cc:1515] Processing DeleteTablet for tablet ce2a1ba75d674c95bdba9d6f12273bee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:12 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:12.856894 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 8d5b10ede9c84be098795c0a787e5a78 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:12 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:12.859490 1332 ts_tablet_manager.cc:1939] T c445c300d71643609071f0f36528a9c4 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:12.860633 1332 tablet_replica.cc:331] T ce2a1ba75d674c95bdba9d6f12273bee P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:12.862254 1332 raft_consensus.cc:2241] T ce2a1ba75d674c95bdba9d6f12273bee P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:12.864560 441 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet c445c300d71643609071f0f36528a9c4 (table default.Table_c766b74c950b433b963d93d396521bcd [id=739509ac727245c9a2d63f409e8f0f52]) successfully deleted
I20250901 01:52:12.866039 1293 raft_consensus.cc:491] T 2685fb7258264463b5f1ec99cd7d67ad P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:12.866210 1293 raft_consensus.cc:513] T 2685fb7258264463b5f1ec99cd7d67ad P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:12.866973 1293 leader_election.cc:290] T 2685fb7258264463b5f1ec99cd7d67ad P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:12.869380 1332 raft_consensus.cc:2270] T ce2a1ba75d674c95bdba9d6f12273bee P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:12.869608 1306 ts_tablet_manager.cc:1918] T 8d5b10ede9c84be098795c0a787e5a78 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:12.867610 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2685fb7258264463b5f1ec99cd7d67ad" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:12.870157 1306 log.cc:1199] T 8d5b10ede9c84be098795c0a787e5a78 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/8d5b10ede9c84be098795c0a787e5a78
I20250901 01:52:12.870452 1332 ts_tablet_manager.cc:1905] T ce2a1ba75d674c95bdba9d6f12273bee P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:12.870574 859 leader_election.cc:343] T 2685fb7258264463b5f1ec99cd7d67ad P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:12.875792 424 master_runner.cc:186] Error getting master registration for 127.0.104.60:44231: Network error: Client connection negotiation failed: client connection to 127.0.104.60:44231: connect: Connection refused (error 111),
I20250901 01:52:12.875972 424 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 01:52:12.877086 447 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 97e40c318f64455d91b0e4f8866990ce
I20250901 01:52:12.877655 447 catalog_manager.cc:6652] ReplaceTablet: tablet 97e40c318f64455d91b0e4f8866990ce deleted and replaced by tablet f50f33de1b654663b9ef8a91e3436f11
I20250901 01:52:12.878006 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_58fc08621bb54011abf9c7484939eb75"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:12.878609 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_58fc08621bb54011abf9c7484939eb75 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:12.882750 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 97e40c318f64455d91b0e4f8866990ce with delete_type TABLET_DATA_DELETED (replaced by tablet f50f33de1b654663b9ef8a91e3436f11) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:12.883399 896 tablet_service.cc:1515] Processing DeleteTablet for tablet 97e40c318f64455d91b0e4f8866990ce with delete_type TABLET_DATA_DELETED (replaced by tablet f50f33de1b654663b9ef8a91e3436f11) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:12.884014 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet dd694cfa09564d3da5d279c0861a4c32
I20250901 01:52:12.884781 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_1426e76865454f76a0529397d709ac7f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:12.885354 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1426e76865454f76a0529397d709ac7f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250901 01:52:12.886857 448 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a7ff996482164776b468cf305989866b consensus state has no leader
I20250901 01:52:12.887234 448 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_a3a2b89da62b445e83cd3c88fd4fc775" } new_table_name: "default.Table_e9b3e507b30344d3a70b1f97bd72b6e0" modify_external_catalogs: true
I20250901 01:52:12.891928 446 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_45ba5a130a824accac159d040d1b6a9d" } new_table_name: "default.Table_5e65c5f068b84995a2d7a76fa65dda0f" modify_external_catalogs: true
I20250901 01:52:12.899757 1332 ts_tablet_manager.cc:1918] T ce2a1ba75d674c95bdba9d6f12273bee P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:12.902886 1332 log.cc:1199] T ce2a1ba75d674c95bdba9d6f12273bee P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/ce2a1ba75d674c95bdba9d6f12273bee
I20250901 01:52:12.903827 1332 ts_tablet_manager.cc:1939] T ce2a1ba75d674c95bdba9d6f12273bee P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
#16 0x96bf76 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() ../../../include/c++/7.5.0/bits/shared_ptr_base.h:154:6
#17 0x96be51 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() ../../../include/c++/7.5.0/bits/shared_ptr_base.h:684:11
#18 0x7fe7989dc2d1 in std::__shared_ptr<kudu::tablet::OpDriver, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() ../../../include/c++/7.5.0/bits/shared_ptr_base.h:1123:31
#19 0x7fe7989c520f in kudu::tablet::TabletReplica::SubmitAlterSchema(std::unique_ptr<kudu::tablet::AlterSchemaOpState, std::default_delete<kudu::tablet::AlterSchemaOpState> >, kudu::MonoTime) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tablet/tablet_replica.cc:628:1
#20 0x7fe79b3dd6e3 in kudu::tserver::TabletServiceAdminImpl::AlterSchema(kudu::tserver::AlterSchemaRequestPB const*, kudu::tserver::AlterSchemaResponsePB*, kudu::rpc::RpcContext*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tserver/tablet_service.cc:1214:23
I20250901 01:52:12.904844 896 tablet_service.cc:1515] Processing DeleteTablet for tablet dd694cfa09564d3da5d279c0861a4c32 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:12 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:12.905124 1306 ts_tablet_manager.cc:1939] T 8d5b10ede9c84be098795c0a787e5a78 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:12.905833 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 8d5b10ede9c84be098795c0a787e5a78 (table default.Table_472dac9061664ce0afb468db1d8fa1de [id=017c724f98ec404bab6e882ead9b8541]) successfully deleted
I20250901 01:52:12.906240 1306 tablet_replica.cc:331] T ce2a1ba75d674c95bdba9d6f12273bee P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:12.906554 1306 raft_consensus.cc:2241] T ce2a1ba75d674c95bdba9d6f12273bee P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:12.906760 1306 raft_consensus.cc:2270] T ce2a1ba75d674c95bdba9d6f12273bee P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:12.906924 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_12f764717f384ee4b906dec11b1212f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:12.907395 1306 ts_tablet_manager.cc:1905] T ce2a1ba75d674c95bdba9d6f12273bee P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:12.907464 449 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 3df908c3096a4df7bddd8ac88b49cfe0
W20250901 01:52:12.907605 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_12f764717f384ee4b906dec11b1212f8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:12.907977 449 catalog_manager.cc:6652] ReplaceTablet: tablet 3df908c3096a4df7bddd8ac88b49cfe0 deleted and replaced by tablet b009cc95c819431ea9827a3e453b8da3
I20250901 01:52:12.908335 1332 tablet_replica.cc:331] T 8d5b10ede9c84be098795c0a787e5a78 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:12.909408 1332 raft_consensus.cc:2241] T 8d5b10ede9c84be098795c0a787e5a78 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:12.909611 1332 raft_consensus.cc:2270] T 8d5b10ede9c84be098795c0a787e5a78 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:12.910209 1332 ts_tablet_manager.cc:1905] T 8d5b10ede9c84be098795c0a787e5a78 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:12.910902 1036 tablet_service.cc:1515] Processing DeleteTablet for tablet dd694cfa09564d3da5d279c0861a4c32 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:12 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:12.913504 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 116f65a42ab34bf6b8f8113cb7afdc21
W20250901 01:52:12.917430 994 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250901 01:52:12.907475 441 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet ce2a1ba75d674c95bdba9d6f12273bee (table default.Table_b16d719f7f9041fd9391c5629066b1ad [id=7e36d9fe490e41c9af1a870ec27c6727]) successfully deleted
W20250901 01:52:12.918673 454 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:41695 (request call id 16) took 1618 ms (client timeout 1000 ms). Trace:
W20250901 01:52:12.918787 454 rpcz_store.cc:269] 0901 01:52:11.299808 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:11.803295 (+503487us) service_pool.cc:225] Handling call
0901 01:52:12.918633 (+1115338us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2779,"reported_tablets":19}
I20250901 01:52:12.919124 1036 tablet_service.cc:1515] Processing DeleteTablet for tablet 3df908c3096a4df7bddd8ac88b49cfe0 with delete_type TABLET_DATA_DELETED (replaced by tablet b009cc95c819431ea9827a3e453b8da3) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:12.919370 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 116f65a42ab34bf6b8f8113cb7afdc21 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:12 UTC) from {username='slave'} at 127.0.0.1:44246
W20250901 01:52:12.922237 446 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 824229183e024feaa975c7f0387c9c1a has no consensus state
W20250901 01:52:12.926739 448 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a774cc6ae3924d299127fd9ffb01ca57 has no consensus state
I20250901 01:52:12.930888 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet c297e8bf11ff467a854eab41117ac376 with delete_type TABLET_DATA_DELETED (replaced by tablet 186ec810d02e4748aad15228b5866ac1) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:12.931030 1033 tablet_service.cc:1515] Processing DeleteTablet for tablet 7a8a3aca289c4735abeb9c95c414ab76 with delete_type TABLET_DATA_DELETED (replaced by tablet 85322a864d154f299f2d670fdd2a7bfb) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:12.930696 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 288e8029382f4154bc1e2103b3035a60 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:09 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:12.931358 1033 tablet_service.cc:1515] Processing DeleteTablet for tablet 77405097e68344628f4eda667afaed21 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:09 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:12.931466 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 4f8e3fcaa19740d397f4d118ac4cbdb0 with delete_type TABLET_DATA_DELETED (replaced by tablet 88a6ab29d00542aba54f0e4185fbf615) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:12.931602 1038 tablet_service.cc:1515] Processing DeleteTablet for tablet 8e6e96b4de214cadba1ccd19390cffb0 with delete_type TABLET_DATA_DELETED (replaced by tablet 3b18a7084727490380d26b3485e4e799) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:12.931664 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet bf302e76746b4f7bbe4469914f83720f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:08 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:12.931720 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 3eaf8baac9f1485f944cd9e74b8b5569 with delete_type TABLET_DATA_DELETED (replaced by tablet 5adf6b492640417ca071c94702b48cb9) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:12.931352 1036 tablet_service.cc:1515] Processing DeleteTablet for tablet c445c300d71643609071f0f36528a9c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:12 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:12.936424 896 tablet_service.cc:1515] Processing DeleteTablet for tablet 3df908c3096a4df7bddd8ac88b49cfe0 with delete_type TABLET_DATA_DELETED (replaced by tablet b009cc95c819431ea9827a3e453b8da3) from {username='slave'} at 127.0.0.1:52846
W20250901 01:52:12.936949 992 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250901 01:52:12.937659 900 tablet_service.cc:1515] Processing DeleteTablet for tablet c297e8bf11ff467a854eab41117ac376 with delete_type TABLET_DATA_DELETED (replaced by tablet 186ec810d02e4748aad15228b5866ac1) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:12.938613 1332 ts_tablet_manager.cc:1918] T 8d5b10ede9c84be098795c0a787e5a78 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:12.938699 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 7a8a3aca289c4735abeb9c95c414ab76 with delete_type TABLET_DATA_DELETED (replaced by tablet 85322a864d154f299f2d670fdd2a7bfb) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:12.939065 1332 log.cc:1199] T 8d5b10ede9c84be098795c0a787e5a78 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/8d5b10ede9c84be098795c0a787e5a78
I20250901 01:52:12.939718 1332 ts_tablet_manager.cc:1939] T 8d5b10ede9c84be098795c0a787e5a78 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:12.940066 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 4f8e3fcaa19740d397f4d118ac4cbdb0 with delete_type TABLET_DATA_DELETED (replaced by tablet 88a6ab29d00542aba54f0e4185fbf615) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:12.940289 901 tablet_service.cc:1515] Processing DeleteTablet for tablet 77405097e68344628f4eda667afaed21 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:09 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:12.940958 441 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 8d5b10ede9c84be098795c0a787e5a78 (table default.Table_472dac9061664ce0afb468db1d8fa1de [id=017c724f98ec404bab6e882ead9b8541]) successfully deleted
I20250901 01:52:12.941421 1332 tablet_replica.cc:331] T 97e40c318f64455d91b0e4f8866990ce P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:12.941754 1332 raft_consensus.cc:2241] T 97e40c318f64455d91b0e4f8866990ce P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:12.942097 1332 pending_rounds.cc:70] T 97e40c318f64455d91b0e4f8866990ce P fd27ab6665304aaea53adbd102dd4f58: Trying to abort 1 pending ops.
I20250901 01:52:12.942355 1332 pending_rounds.cc:77] T 97e40c318f64455d91b0e4f8866990ce P fd27ab6665304aaea53adbd102dd4f58: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7195408515016101888 op_type: NO_OP noop_request { }
I20250901 01:52:12.942446 896 tablet_service.cc:1515] Processing DeleteTablet for tablet 116f65a42ab34bf6b8f8113cb7afdc21 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:12 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:12.942549 1150 tablet_bootstrap.cc:492] T 85322a864d154f299f2d670fdd2a7bfb P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:12.942720 1332 raft_consensus.cc:2887] T 97e40c318f64455d91b0e4f8866990ce P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250901 01:52:12.942979 1332 raft_consensus.cc:2270] T 97e40c318f64455d91b0e4f8866990ce P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:12.946444 1434 tablet.cc:1722] T 061b760d8e6647c6ab0b5d538840a230 P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:12.949951 1151 tablet_bootstrap.cc:492] T 447ffe73741640ceb73f35676232a094 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:12.950153 1151 ts_tablet_manager.cc:1397] T 447ffe73741640ceb73f35676232a094 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.176s user 0.001s sys 0.013s
I20250901 01:52:12.955907 1151 raft_consensus.cc:357] T 447ffe73741640ceb73f35676232a094 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } }
I20250901 01:52:12.956096 1151 raft_consensus.cc:383] T 447ffe73741640ceb73f35676232a094 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:12.956176 1151 raft_consensus.cc:738] T 447ffe73741640ceb73f35676232a094 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:12.956657 1151 consensus_queue.cc:260] T 447ffe73741640ceb73f35676232a094 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } }
#21 0x7fe7955e7366 in kudu::tserver::TabletServerAdminServiceIf::TabletServerAdminServiceIf(scoped_refptr<kudu::MetricEntity> const&, scoped_refptr<kudu::rpc::ResultTracker> const&)::$_5::operator()(google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*) const /home/jenkins-slave/workspace/build_and_test_flaky@2/build/asan/src/kudu/tserver/tserver_admin.service.cc:241:13
I20250901 01:52:12.957202 1151 ts_tablet_manager.cc:1428] T 447ffe73741640ceb73f35676232a094 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.007s user 0.001s sys 0.002s
#22 0x7fe7955e7095 in std::_Function_handler<void (google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*), kudu::tserver::TabletServerAdminServiceIf::TabletServerAdminServiceIf(scoped_refptr<kudu::MetricEntity> const&, scoped_refptr<kudu::rpc::ResultTracker> const&)::$_5>::_M_invoke(std::_Any_data const&, google::protobuf::Message const*&&, google::protobuf::Message*&&, kudu::rpc::RpcContext*&&) ../../../include/c++/7.5.0/bits/std_function.h:316:2
I20250901 01:52:12.960007 1151 tablet_bootstrap.cc:492] T 342b7fde3ede494b9eec6e361db255dc P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:12.951512 492 master_runner.cc:186] Error getting master registration for 127.0.104.60:44231: Network error: Client connection negotiation failed: client connection to 127.0.104.60:44231: connect: Connection refused (error 111),
I20250901 01:52:12.963768 492 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250901 01:52:12.965997 901 tablet_service.cc:1515] Processing DeleteTablet for tablet 3eaf8baac9f1485f944cd9e74b8b5569 with delete_type TABLET_DATA_DELETED (replaced by tablet 5adf6b492640417ca071c94702b48cb9) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:12.966238 896 tablet_service.cc:1515] Processing DeleteTablet for tablet 4d333b670df84f308f0cbf6da636b768 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:08 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:12.966956 1332 ts_tablet_manager.cc:1905] T 97e40c318f64455d91b0e4f8866990ce P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:12.975297 1438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250901 01:52:12.975891 1440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 01:52:12.978809 1150 ts_tablet_manager.cc:1397] T 85322a864d154f299f2d670fdd2a7bfb P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.158s user 0.005s sys 0.004s
I20250901 01:52:12.979157 896 tablet_service.cc:1515] Processing DeleteTablet for tablet 4cdd369214ef4f5ebabddbf28b2ae3f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:08 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:12.979346 897 tablet_service.cc:1515] Processing DeleteTablet for tablet 8d5b10ede9c84be098795c0a787e5a78 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:12 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:12.981319 904 tablet_service.cc:1515] Processing DeleteTablet for tablet 2a0d125423a043dcb11e67345599e468 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:09 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:12.983783 448 catalog_manager.cc:5582] T eeae5294bd474f368f5766c260ccf3fe P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } }
I20250901 01:52:12.984462 896 tablet_service.cc:1515] Processing DeleteTablet for tablet 342b7fde3ede494b9eec6e361db255dc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:10 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:12.985659 896 tablet_service.cc:1515] Processing DeleteTablet for tablet 8e6e96b4de214cadba1ccd19390cffb0 with delete_type TABLET_DATA_DELETED (replaced by tablet 3b18a7084727490380d26b3485e4e799) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:12.985826 448 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet eeae5294bd474f368f5766c260ccf3fe (table default.Table_300cde3d9237488ba65cab60dd8f8721 [id=a48c2b3f8c4847cf9f9cf29548db5c50]). Expected version 1 got 0
I20250901 01:52:12.986182 448 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet f0affc7a78634607852a48458412761d (table default.Table_b7bffcbb78614d3cad2b1d375d7a97cc [id=535cf5f83c56443d93935ce80ff87efb]). Expected version 1 got 0
I20250901 01:52:12.986486 894 tablet_service.cc:1515] Processing DeleteTablet for tablet bf302e76746b4f7bbe4469914f83720f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:08 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:12.989540 896 tablet_service.cc:1468] Processing CreateTablet for tablet 4d27fb61943c45d4aca67689cc3a9ff3 (DEFAULT_TABLE table=default.Table_0e0c69a04fea4f5fab174e11a35bf628 [id=78de546347e74cc89be668054e292f04]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:12.989612 1255 server_base.cc:1047] running on GCE node
I20250901 01:52:12.991338 897 tablet_service.cc:1515] Processing DeleteTablet for tablet fab3c1a8928a4066a15868ba708267f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:08 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:12.992211 902 tablet_service.cc:1515] Processing DeleteTablet for tablet 447ffe73741640ceb73f35676232a094 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:09 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:12.993741 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d27fb61943c45d4aca67689cc3a9ff3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:12.997946 894 tablet_service.cc:1468] Processing CreateTablet for tablet b541a6e14f114ab29a864cca11806316 (DEFAULT_TABLE table=default.Table_52927bd3bdf4467581d6d963a046e0a7 [id=81323e6fbe424655b8e185f52657fadc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:12.998585 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b541a6e14f114ab29a864cca11806316. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:13.000916 451 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_31f776f7dc534acf80b5896fffb00fe1" } new_table_name: "default.Table_8df8ec915ec84331933c6de7ed2983b3" modify_external_catalogs: true
I20250901 01:52:13.002633 452 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_0c6a1616a7b64455a3ae44c9983dc6c0" } new_table_name: "default.Table_4f1c4695ad9e4f24a31d3522281a41a5" modify_external_catalogs: true
I20250901 01:52:13.004469 904 tablet_service.cc:1515] Processing DeleteTablet for tablet 288e8029382f4154bc1e2103b3035a60 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:09 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:13.006083 902 tablet_service.cc:1468] Processing CreateTablet for tablet 49d30da95b14400b845270122487a2a0 (DEFAULT_TABLE table=default.Table_c96393e11f3a4cca8202ccc2df6635e0 [id=091e1ea029a4404e87561de62e038043]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:13.006666 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49d30da95b14400b845270122487a2a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:13.009938 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_655d2fe6f5f044d2ac3c542497704343"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:13.010529 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_655d2fe6f5f044d2ac3c542497704343 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:13.011688 899 tablet_service.cc:1468] Processing CreateTablet for tablet 12c04e0c1f804179ae23509cdb24d395 (DEFAULT_TABLE table=default.Table_3d223b93128649e49a295d940feef90a [id=348190236c694eab9a318040bb19c1f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:13.011823 446 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_3a7da8de6b044deda677302fd0c530ea" } modify_external_catalogs: true
I20250901 01:52:13.012007 446 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_3a7da8de6b044deda677302fd0c530ea" } modify_external_catalogs: true
I20250901 01:52:13.012812 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12c04e0c1f804179ae23509cdb24d395. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:13.014070 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_ae7ef97906764641b443ff461e4aedf2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:13.014647 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ae7ef97906764641b443ff461e4aedf2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:13.015125 450 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_cb6a9252a90d4184b917db4c7edc06b1" } modify_external_catalogs: true
I20250901 01:52:13.015376 454 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_95c28a4758e148368604152622fe51be" } modify_external_catalogs: true
I20250901 01:52:13.016139 904 tablet_service.cc:1468] Processing CreateTablet for tablet b633b461dbf34e13bc6e1cc08e6d20c1 (DEFAULT_TABLE table=default.Table_52733c9a363b4ed9b6ff11ad8571e421 [id=453153553bea48319e132857f628d48a]), partition=RANGE (key) PARTITION UNBOUNDED
W20250901 01:52:13.017127 513 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.GetMasterRegistration -> {remote=127.0.104.60:44231, user_credentials={real_user=slave}} blocked reactor thread for 65607us
I20250901 01:52:13.018177 904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b633b461dbf34e13bc6e1cc08e6d20c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:13.020431 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_706ad9744bcd4923b9a627c93f3cfde7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:13.021057 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_706ad9744bcd4923b9a627c93f3cfde7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:13.028039 449 master_service.cc:948] ReplaceTablet: received request to replace tablet 3df908c3096a4df7bddd8ac88b49cfe0 from {username='slave'} at 127.0.0.1:57810
W20250901 01:52:13.029253 451 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 85322a864d154f299f2d670fdd2a7bfb consensus state has no leader
I20250901 01:52:13.030584 450 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_cb6a9252a90d4184b917db4c7edc06b1" } modify_external_catalogs: true
I20250901 01:52:13.031591 923 raft_consensus.cc:1273] T eeae5294bd474f368f5766c260ccf3fe P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Refusing update from remote peer fd27ab6665304aaea53adbd102dd4f58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250901 01:52:13.031613 454 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_95c28a4758e148368604152622fe51be" } modify_external_catalogs: true
I20250901 01:52:13.035552 898 tablet_service.cc:1468] Processing CreateTablet for tablet a774cc6ae3924d299127fd9ffb01ca57 (DEFAULT_TABLE table=default.Table_e9b3e507b30344d3a70b1f97bd72b6e0 [id=1d0bc46740444f1fbf2129270e806259]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:13.036448 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a774cc6ae3924d299127fd9ffb01ca57. 1 dirs total, 0 dirs full, 0 dirs failed
W20250901 01:52:13.038398 452 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f50f33de1b654663b9ef8a91e3436f11 has no consensus state
I20250901 01:52:13.040598 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_e50c89f096ef4139a5b214c59cba886b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:13.041261 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e50c89f096ef4139a5b214c59cba886b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:13.044188 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_cecc206bb66146c7ad35d7731bc02859"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:13.044793 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cecc206bb66146c7ad35d7731bc02859 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:13.046217 1150 raft_consensus.cc:357] T 85322a864d154f299f2d670fdd2a7bfb P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:13.046522 1150 raft_consensus.cc:383] T 85322a864d154f299f2d670fdd2a7bfb P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:13.046603 1150 raft_consensus.cc:738] T 85322a864d154f299f2d670fdd2a7bfb P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:13.046904 1150 consensus_queue.cc:260] T 85322a864d154f299f2d670fdd2a7bfb P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:13.047595 1150 ts_tablet_manager.cc:1428] T 85322a864d154f299f2d670fdd2a7bfb P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.069s user 0.001s sys 0.002s
I20250901 01:52:13.047917 1150 tablet_bootstrap.cc:492] T 756cfe84a517459ca015aecf5cd81571 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:13.050200 448 catalog_manager.cc:6127] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: default.Table_3c7c628f35fb4d02bb79a34995f12938 [id=a39e8b171a874df1890d072732e55df9] alter complete (version 2)
I20250901 01:52:13.053627 446 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 121b82d63e714ab2a57101faa81b2cdb
I20250901 01:52:13.054316 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 154574e17f214cc0ad56e6e6be3af899
I20250901 01:52:13.054461 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 9496ec6021784800aef62996b0dee593
I20250901 01:52:13.057953 897 tablet_service.cc:1515] Processing DeleteTablet for tablet 154574e17f214cc0ad56e6e6be3af899 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:13 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:13.058514 897 tablet_service.cc:1515] Processing DeleteTablet for tablet 9496ec6021784800aef62996b0dee593 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:13 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:13.060228 1435 tablet.cc:1722] T 061b760d8e6647c6ab0b5d538840a230 P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:12.979898 1036 tablet_service.cc:1468] Processing CreateTablet for tablet 4d27fb61943c45d4aca67689cc3a9ff3 (DEFAULT_TABLE table=default.Table_0e0c69a04fea4f5fab174e11a35bf628 [id=78de546347e74cc89be668054e292f04]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:13.064299 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d27fb61943c45d4aca67689cc3a9ff3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:13.059598 903 tablet_service.cc:1468] Processing CreateTablet for tablet 824229183e024feaa975c7f0387c9c1a (DEFAULT_TABLE table=default.Table_5e65c5f068b84995a2d7a76fa65dda0f [id=b9d81ddd38fe47838877b010be2b40d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:13.065065 903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 824229183e024feaa975c7f0387c9c1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:12.980273 1027 tablet_service.cc:1468] Processing CreateTablet for tablet b541a6e14f114ab29a864cca11806316 (DEFAULT_TABLE table=default.Table_52927bd3bdf4467581d6d963a046e0a7 [id=81323e6fbe424655b8e185f52657fadc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:12.983019 1151 tablet_bootstrap.cc:654] T 342b7fde3ede494b9eec6e361db255dc P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:13.068729 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b541a6e14f114ab29a864cca11806316. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:12.985548 1038 tablet_service.cc:1468] Processing CreateTablet for tablet 12c04e0c1f804179ae23509cdb24d395 (DEFAULT_TABLE table=default.Table_3d223b93128649e49a295d940feef90a [id=348190236c694eab9a318040bb19c1f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:13.069664 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12c04e0c1f804179ae23509cdb24d395. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:12.985548 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 49d30da95b14400b845270122487a2a0 (DEFAULT_TABLE table=default.Table_c96393e11f3a4cca8202ccc2df6635e0 [id=091e1ea029a4404e87561de62e038043]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:12.988678 1033 tablet_service.cc:1468] Processing CreateTablet for tablet b633b461dbf34e13bc6e1cc08e6d20c1 (DEFAULT_TABLE table=default.Table_52733c9a363b4ed9b6ff11ad8571e421 [id=453153553bea48319e132857f628d48a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:13.078871 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b633b461dbf34e13bc6e1cc08e6d20c1. 1 dirs total, 0 dirs full, 0 dirs failed
W20250901 01:52:13.082530 1437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 01:52:13.083928 1255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 01:52:12.990658 1035 tablet_service.cc:1468] Processing CreateTablet for tablet 824229183e024feaa975c7f0387c9c1a (DEFAULT_TABLE table=default.Table_5e65c5f068b84995a2d7a76fa65dda0f [id=b9d81ddd38fe47838877b010be2b40d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:13.084681 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 824229183e024feaa975c7f0387c9c1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:13.087523 450 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_0e0c69a04fea4f5fab174e11a35bf628" } modify_external_catalogs: true
I20250901 01:52:13.087723 450 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_0e0c69a04fea4f5fab174e11a35bf628" } modify_external_catalogs: true
I20250901 01:52:13.011242 1029 tablet_service.cc:1468] Processing CreateTablet for tablet a774cc6ae3924d299127fd9ffb01ca57 (DEFAULT_TABLE table=default.Table_e9b3e507b30344d3a70b1f97bd72b6e0 [id=1d0bc46740444f1fbf2129270e806259]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:13.088728 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a774cc6ae3924d299127fd9ffb01ca57. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:13.092701 454 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_2f5cfcd78bb0468385bf77c14de3583a" } new_table_name: "default.Table_dfd01d6b61ac43538c37f74330579cbd" modify_external_catalogs: true
I20250901 01:52:13.096902 448 catalog_manager.cc:6127] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: default.Table_fbba493de02249499d26467bf15c3d5f [id=748daa63137c48a5b151bd34dc5b13fe] alter complete (version 1)
I20250901 01:52:13.098212 1036 tablet_service.cc:1515] Processing DeleteTablet for tablet a4f5685260304d589dd467953d9bebfc with delete_type TABLET_DATA_DELETED (replaced by tablet 1f18420261ae41459e1f25b927bd84a9) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:13.101250 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_647007ba008f42bcb27440e4d85a1d90"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:13.101888 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_647007ba008f42bcb27440e4d85a1d90 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:13.103405 1036 tablet_service.cc:1515] Processing DeleteTablet for tablet 447ffe73741640ceb73f35676232a094 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:09 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:13.108127 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49d30da95b14400b845270122487a2a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:13.034739 1377 consensus_queue.cc:1035] T eeae5294bd474f368f5766c260ccf3fe P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:13.055510 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet 9496ec6021784800aef62996b0dee593 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:13 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:13.055704 1034 tablet_service.cc:1515] Processing DeleteTablet for tablet 154574e17f214cc0ad56e6e6be3af899 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:13 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:13.120869 452 master_service.cc:948] ReplaceTablet: received request to replace tablet 0e698923812841b7a5d37d57da103c5c from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:13.123075 452 master_service.cc:948] ReplaceTablet: received request to replace tablet 58c0e7a2e90e48528bc2bc7ec889f608 from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:13.132282 446 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_daa5affe30374472879ede5e985a529d" } new_table_name: "default.Table_1f8f7c00026749f593d89bc9e92f71cc" modify_external_catalogs: true
I20250901 01:52:13.132424 453 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_3d223b93128649e49a295d940feef90a" } modify_external_catalogs: true
I20250901 01:52:13.132562 453 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_3d223b93128649e49a295d940feef90a" } modify_external_catalogs: true
I20250901 01:52:13.133152 455 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_87c07d8f93414f3aaf9b47d8f7bbd24b" } new_table_name: "default.Table_c7b9e3df2cf843a885789d479e0e2d25" modify_external_catalogs: true
I20250901 01:52:13.135317 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_749fcd4bf97d439c9fd49b138a52a3fd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:13.136039 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_874154cd2b0c4afcb0fa684f312984db"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:13.136538 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_874154cd2b0c4afcb0fa684f312984db in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:13.135247 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_4ce8198ebbed4cc4beba453f1faf486a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:13.137578 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4ce8198ebbed4cc4beba453f1faf486a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250901 01:52:13.138139 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_749fcd4bf97d439c9fd49b138a52a3fd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:13.145035 1151 tablet_bootstrap.cc:492] T 342b7fde3ede494b9eec6e361db255dc P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:13.146128 1448 tablet.cc:1722] T 116f65a42ab34bf6b8f8113cb7afdc21 P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:13.146852 1151 ts_tablet_manager.cc:1397] T 342b7fde3ede494b9eec6e361db255dc P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.189s user 0.008s sys 0.007s
I20250901 01:52:13.148991 1151 raft_consensus.cc:357] T 342b7fde3ede494b9eec6e361db255dc P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } }
I20250901 01:52:13.151741 1306 ts_tablet_manager.cc:1918] T ce2a1ba75d674c95bdba9d6f12273bee P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:13.152233 1151 raft_consensus.cc:383] T 342b7fde3ede494b9eec6e361db255dc P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:13.152941 1151 raft_consensus.cc:738] T 342b7fde3ede494b9eec6e361db255dc P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:13.153579 1151 consensus_queue.cc:260] T 342b7fde3ede494b9eec6e361db255dc P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } }
I20250901 01:52:13.156800 1151 ts_tablet_manager.cc:1428] T 342b7fde3ede494b9eec6e361db255dc P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20250901 01:52:13.157318 1151 tablet_bootstrap.cc:492] T 4d333b670df84f308f0cbf6da636b768 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
W20250901 01:52:13.157913 454 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 82d163681fda47f38d79863bd28f576d has no consensus state
I20250901 01:52:13.160522 1151 tablet_bootstrap.cc:654] T 4d333b670df84f308f0cbf6da636b768 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:13.169977 1446 tablet.cc:1722] T 116f65a42ab34bf6b8f8113cb7afdc21 P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250901 01:52:13.182485 446 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a7ff996482164776b468cf305989866b consensus state has no leader
I20250901 01:52:13.182799 446 master_service.cc:948] ReplaceTablet: received request to replace tablet 22691943a134400c8df4617240e5d3ef from {username='slave'} at 127.0.0.1:57810
W20250901 01:52:13.183480 455 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f65ec84958eb488d8b0f812b59b5b3d5 consensus state has no leader
I20250901 01:52:13.183863 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_54d5a8d2e22840d6a43c4eeb44478d1d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:13.184453 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_54d5a8d2e22840d6a43c4eeb44478d1d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:13.196580 454 master_service.cc:948] ReplaceTablet: received request to replace tablet 360755d30b5d47ba98c9c7e23ff5ab7e from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:13.212677 1255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 01:52:13.214313 1255 hybrid_clock.cc:648] HybridClock initialized: now 1756691533214251 us; error 43 us; skew 500 ppm
I20250901 01:52:13.215031 1255 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250901 01:52:13.217483 1306 log.cc:1199] T ce2a1ba75d674c95bdba9d6f12273bee P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/ce2a1ba75d674c95bdba9d6f12273bee
I20250901 01:52:13.181408 1150 tablet_bootstrap.cc:700] T 756cfe84a517459ca015aecf5cd81571 P 00c282f779c1477d920f24dc0dc16628: Time spent opening tablet: real 0.132s user 0.001s sys 0.000s
I20250901 01:52:13.218389 1150 tablet_bootstrap.cc:654] T 756cfe84a517459ca015aecf5cd81571 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:13.222460 1306 ts_tablet_manager.cc:1939] T ce2a1ba75d674c95bdba9d6f12273bee P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:13.223769 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet ce2a1ba75d674c95bdba9d6f12273bee (table default.Table_b16d719f7f9041fd9391c5629066b1ad [id=7e36d9fe490e41c9af1a870ec27c6727]) successfully deleted
I20250901 01:52:13.223845 1306 tablet_replica.cc:331] T c445c300d71643609071f0f36528a9c4 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:13.224291 1306 raft_consensus.cc:2241] T c445c300d71643609071f0f36528a9c4 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:13.224623 1306 raft_consensus.cc:2270] T c445c300d71643609071f0f36528a9c4 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:13.225363 1306 ts_tablet_manager.cc:1905] T c445c300d71643609071f0f36528a9c4 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:13.240541 1343 mvcc.cc:204] Tried to move back new op lower bound from 7195408519221014528 to 7195408516600815616. Current Snapshot: MvccSnapshot[applied={T|T < 7195408519221014528}]
I20250901 01:52:13.241533 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 342b7fde3ede494b9eec6e361db255dc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:10 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:13.243793 1150 tablet_bootstrap.cc:492] T 756cfe84a517459ca015aecf5cd81571 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:13.243973 1150 ts_tablet_manager.cc:1397] T 756cfe84a517459ca015aecf5cd81571 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.196s user 0.002s sys 0.004s
I20250901 01:52:13.244306 1058 raft_consensus.cc:1273] T 24ddc13d1bb04db4a7850acbff091577 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Refusing update from remote peer 00c282f779c1477d920f24dc0dc16628: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250901 01:52:13.245661 1293 consensus_queue.cc:1035] T 24ddc13d1bb04db4a7850acbff091577 P 00c282f779c1477d920f24dc0dc16628 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250901 01:52:13.245846 451 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_c7b9e3df2cf843a885789d479e0e2d25" } new_table_name: "default.Table_a3ff88e352884abdba91e6d03acbae22" modify_external_catalogs: true
I20250901 01:52:13.246459 1150 raft_consensus.cc:357] T 756cfe84a517459ca015aecf5cd81571 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:13.246749 1150 raft_consensus.cc:383] T 756cfe84a517459ca015aecf5cd81571 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:13.246883 1150 raft_consensus.cc:738] T 756cfe84a517459ca015aecf5cd81571 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:13.247296 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_3f2af1af32b546d6b1eba5cc3195acd8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:13.247789 1150 consensus_queue.cc:260] T 756cfe84a517459ca015aecf5cd81571 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:13.247938 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3f2af1af32b546d6b1eba5cc3195acd8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:13.249516 449 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_05707bc6c5ca46c69e791188350c8615" } new_table_name: "default.Table_da2c70a8f2fe49a0a8184e88c0a9e0ea" modify_external_catalogs: true
I20250901 01:52:13.252631 1150 ts_tablet_manager.cc:1428] T 756cfe84a517459ca015aecf5cd81571 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20250901 01:52:13.257750 455 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_b582d9a2134b4184a195cfe0a8335a33" } modify_external_catalogs: true
I20250901 01:52:13.257939 455 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_b582d9a2134b4184a195cfe0a8335a33" } modify_external_catalogs: true
I20250901 01:52:13.258354 1448 tablet.cc:1722] T eeae5294bd474f368f5766c260ccf3fe P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:13.258499 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_685b0eabd08542a98940dc6cbbf00b92"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:13.259020 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_685b0eabd08542a98940dc6cbbf00b92 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:13.261534 447 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_a2bf411424c74109a7c2baa28a4db8b9" } modify_external_catalogs: true
W20250901 01:52:13.261638 449 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3b18a7084727490380d26b3485e4e799 consensus state has no leader
I20250901 01:52:13.261727 447 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_a2bf411424c74109a7c2baa28a4db8b9" } modify_external_catalogs: true
I20250901 01:52:13.262032 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_7f507bbabca54260a4a5d9092c274e41"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:13.262662 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7f507bbabca54260a4a5d9092c274e41 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250901 01:52:13.264590 451 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f65ec84958eb488d8b0f812b59b5b3d5 consensus state has no leader
I20250901 01:52:13.265046 451 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_e50c89f096ef4139a5b214c59cba886b" } modify_external_catalogs: true
I20250901 01:52:13.265232 451 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_e50c89f096ef4139a5b214c59cba886b" } modify_external_catalogs: true
I20250901 01:52:13.267381 1150 tablet_bootstrap.cc:492] T 84fe314502444bd88b038b87eead9cb5 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:13.273660 1150 tablet_bootstrap.cc:654] T 84fe314502444bd88b038b87eead9cb5 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:13.275923 1434 tablet.cc:1722] T eeae5294bd474f368f5766c260ccf3fe P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:13.278127 1255 webserver.cc:480] Webserver started at http://127.0.104.60:46373/ using document root <none> and password file <none>
I20250901 01:52:13.278182 454 master_service.cc:948] ReplaceTablet: received request to replace tablet d945b310fe6344af9cc9dd68f84c4bdf from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:13.279378 1255 fs_manager.cc:362] Metadata directory not provided
I20250901 01:52:13.279537 1255 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20250901 01:52:13.280128 439 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet a774cc6ae3924d299127fd9ffb01ca57 (table default.Table_e9b3e507b30344d3a70b1f97bd72b6e0 [id=1d0bc46740444f1fbf2129270e806259], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet a774cc6ae3924d299127fd9ffb01ca57 consensus state has no leader
I20250901 01:52:13.285549 1332 ts_tablet_manager.cc:1918] T 97e40c318f64455d91b0e4f8866990ce P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:13.285868 1332 log.cc:1199] T 97e40c318f64455d91b0e4f8866990ce P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/97e40c318f64455d91b0e4f8866990ce
I20250901 01:52:13.286376 453 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 12c04e0c1f804179ae23509cdb24d395
I20250901 01:52:13.286723 1332 ts_tablet_manager.cc:1939] T 97e40c318f64455d91b0e4f8866990ce P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:13.288777 441 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 97e40c318f64455d91b0e4f8866990ce (table default.Table_4f1c4695ad9e4f24a31d3522281a41a5 [id=9af016a062a64218ad3b62e131d8b65d]) successfully deleted
I20250901 01:52:13.288897 898 tablet_service.cc:1515] Processing DeleteTablet for tablet 12c04e0c1f804179ae23509cdb24d395 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:13 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:13.289115 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 12c04e0c1f804179ae23509cdb24d395 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:13 UTC) from {username='slave'} at 127.0.0.1:44246
W20250901 01:52:13.292258 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet dd694cfa09564d3da5d279c0861a4c32 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dd694cfa09564d3da5d279c0861a4c32 already in progress: creating tablet
I20250901 01:52:13.292599 1332 tablet_replica.cc:331] T 116f65a42ab34bf6b8f8113cb7afdc21 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
W20250901 01:52:13.292800 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 3df908c3096a4df7bddd8ac88b49cfe0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3df908c3096a4df7bddd8ac88b49cfe0 already in progress: creating tablet
I20250901 01:52:13.293413 1374 raft_consensus.cc:491] T 447ffe73741640ceb73f35676232a094 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:13.293700 1374 raft_consensus.cc:513] T 447ffe73741640ceb73f35676232a094 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } }
I20250901 01:52:13.294064 451 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 5d60d040ee794dab870add6675e6e60d
I20250901 01:52:13.294855 1374 leader_election.cc:290] T 447ffe73741640ceb73f35676232a094 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561), 0cf599f74322478d91475df234e58965 (127.0.104.1:40731)
I20250901 01:52:13.295598 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_ce16a1b94f714c2699ee6cd916bb0215"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:13.295666 923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "447ffe73741640ceb73f35676232a094" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:13.295861 923 raft_consensus.cc:2466] T 447ffe73741640ceb73f35676232a094 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:13.296281 992 leader_election.cc:304] T 447ffe73741640ceb73f35676232a094 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
W20250901 01:52:13.296350 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ce16a1b94f714c2699ee6cd916bb0215 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:13.296610 1374 raft_consensus.cc:2802] T 447ffe73741640ceb73f35676232a094 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:13.296766 1374 raft_consensus.cc:491] T 447ffe73741640ceb73f35676232a094 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:13.296916 1374 raft_consensus.cc:3058] T 447ffe73741640ceb73f35676232a094 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:13.294952 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_3408ffb7316a4f1da2f045225e881c76"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:13.298187 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3408ffb7316a4f1da2f045225e881c76 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:13.298637 1306 ts_tablet_manager.cc:1918] T c445c300d71643609071f0f36528a9c4 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:13.298980 1306 log.cc:1199] T c445c300d71643609071f0f36528a9c4 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/c445c300d71643609071f0f36528a9c4
I20250901 01:52:13.299688 1306 ts_tablet_manager.cc:1939] T c445c300d71643609071f0f36528a9c4 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:13.300104 1374 raft_consensus.cc:513] T 447ffe73741640ceb73f35676232a094 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } }
I20250901 01:52:13.301424 1374 leader_election.cc:290] T 447ffe73741640ceb73f35676232a094 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561), 0cf599f74322478d91475df234e58965 (127.0.104.1:40731)
I20250901 01:52:13.301260 923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "447ffe73741640ceb73f35676232a094" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:13.301831 923 raft_consensus.cc:3058] T 447ffe73741640ceb73f35676232a094 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:13.306790 923 raft_consensus.cc:2466] T 447ffe73741640ceb73f35676232a094 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:13.307559 992 leader_election.cc:304] T 447ffe73741640ceb73f35676232a094 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:13.307960 1306 tablet_replica.cc:331] T 97e40c318f64455d91b0e4f8866990ce P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:13.308264 1306 raft_consensus.cc:2241] T 97e40c318f64455d91b0e4f8866990ce P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:13.308511 1306 pending_rounds.cc:70] T 97e40c318f64455d91b0e4f8866990ce P 00c282f779c1477d920f24dc0dc16628: Trying to abort 1 pending ops.
I20250901 01:52:13.308615 1306 pending_rounds.cc:77] T 97e40c318f64455d91b0e4f8866990ce P 00c282f779c1477d920f24dc0dc16628: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7195408515016101888 op_type: NO_OP noop_request { }
I20250901 01:52:13.308760 1306 raft_consensus.cc:2887] T 97e40c318f64455d91b0e4f8866990ce P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250901 01:52:13.308938 1306 raft_consensus.cc:2270] T 97e40c318f64455d91b0e4f8866990ce P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:13.309314 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet c445c300d71643609071f0f36528a9c4 (table default.Table_c766b74c950b433b963d93d396521bcd [id=739509ac727245c9a2d63f409e8f0f52]) successfully deleted
I20250901 01:52:13.312083 1255 fs_manager.cc:714] Time spent opening directory manager: real 0.024s user 0.001s sys 0.004s
I20250901 01:52:13.316809 1374 raft_consensus.cc:2802] T 447ffe73741640ceb73f35676232a094 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:13.317212 1150 tablet_bootstrap.cc:492] T 84fe314502444bd88b038b87eead9cb5 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:13.317533 1374 raft_consensus.cc:695] T 447ffe73741640ceb73f35676232a094 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:13.318004 1374 consensus_queue.cc:237] T 447ffe73741640ceb73f35676232a094 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } }
I20250901 01:52:13.317364 1150 ts_tablet_manager.cc:1397] T 84fe314502444bd88b038b87eead9cb5 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.051s user 0.004s sys 0.001s
I20250901 01:52:13.319185 1150 raft_consensus.cc:357] T 84fe314502444bd88b038b87eead9cb5 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:13.319403 1150 raft_consensus.cc:383] T 84fe314502444bd88b038b87eead9cb5 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:13.319515 1150 raft_consensus.cc:738] T 84fe314502444bd88b038b87eead9cb5 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:13.319773 1150 consensus_queue.cc:260] T 84fe314502444bd88b038b87eead9cb5 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:13.321746 1150 ts_tablet_manager.cc:1428] T 84fe314502444bd88b038b87eead9cb5 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250901 01:52:13.321802 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_3ffe49903e864e8e8cc92d82f4c14a3a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:13.322201 1150 tablet_bootstrap.cc:492] T 88a6ab29d00542aba54f0e4185fbf615 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
W20250901 01:52:13.322441 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3ffe49903e864e8e8cc92d82f4c14a3a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:13.330246 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 533d14f5903e49599a34ea353ffd8c00 (DEFAULT_TABLE table=default.Table_12f764717f384ee4b906dec11b1212f8 [id=5fdd352d5e5e47a3a2bd8fd608a263dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:13.330729 1306 ts_tablet_manager.cc:1905] T 97e40c318f64455d91b0e4f8866990ce P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:13.331389 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 533d14f5903e49599a34ea353ffd8c00. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:13.334275 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 4d27fb61943c45d4aca67689cc3a9ff3
I20250901 01:52:13.336808 446 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 22691943a134400c8df4617240e5d3ef
I20250901 01:52:13.336915 1150 tablet_bootstrap.cc:654] T 88a6ab29d00542aba54f0e4185fbf615 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:13.337296 446 catalog_manager.cc:6652] ReplaceTablet: tablet 22691943a134400c8df4617240e5d3ef deleted and replaced by tablet 438cd35ff3f74c49b908d072560077db
I20250901 01:52:13.338375 447 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 1f18420261ae41459e1f25b927bd84a9
W20250901 01:52:13.338457 992 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.104.1:40731 timed out after 3.122s (SENT)
I20250901 01:52:13.341163 452 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 58c0e7a2e90e48528bc2bc7ec889f608
I20250901 01:52:13.341656 452 catalog_manager.cc:6652] ReplaceTablet: tablet 58c0e7a2e90e48528bc2bc7ec889f608 deleted and replaced by tablet a6445b29591548be9f9577db1ba0bfa5
I20250901 01:52:13.343688 898 tablet_service.cc:1468] Processing CreateTablet for tablet 533d14f5903e49599a34ea353ffd8c00 (DEFAULT_TABLE table=default.Table_12f764717f384ee4b906dec11b1212f8 [id=5fdd352d5e5e47a3a2bd8fd608a263dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:13.344525 903 tablet_service.cc:1468] Processing CreateTablet for tablet 4a87e68850c4459eb2c8eafac89c6aa9 (DEFAULT_TABLE table=default.Table_58fc08621bb54011abf9c7484939eb75 [id=cecdd8d6ca6d445c9fd82decd2a95366]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:13.344967 902 tablet_service.cc:1515] Processing DeleteTablet for tablet 4d27fb61943c45d4aca67689cc3a9ff3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:13 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:13.345255 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 22691943a134400c8df4617240e5d3ef with delete_type TABLET_DATA_DELETED (replaced by tablet 438cd35ff3f74c49b908d072560077db) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:13.345467 904 tablet_service.cc:1515] Processing DeleteTablet for tablet 1f18420261ae41459e1f25b927bd84a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:13 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:13.345676 899 tablet_service.cc:1515] Processing DeleteTablet for tablet 58c0e7a2e90e48528bc2bc7ec889f608 with delete_type TABLET_DATA_DELETED (replaced by tablet a6445b29591548be9f9577db1ba0bfa5) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:13.346045 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 533d14f5903e49599a34ea353ffd8c00. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:13.348618 903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a87e68850c4459eb2c8eafac89c6aa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:13.349144 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet d945b310fe6344af9cc9dd68f84c4bdf
I20250901 01:52:13.349618 454 catalog_manager.cc:6652] ReplaceTablet: tablet d945b310fe6344af9cc9dd68f84c4bdf deleted and replaced by tablet f06a77ecc841462396a8d5026c7327f4
I20250901 01:52:13.351464 1431 raft_consensus.cc:491] T d945b310fe6344af9cc9dd68f84c4bdf P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:13.351624 1431 raft_consensus.cc:513] T d945b310fe6344af9cc9dd68f84c4bdf P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:13.352434 1431 leader_election.cc:290] T d945b310fe6344af9cc9dd68f84c4bdf P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:13.352770 448 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 467c2bf4a8cb4937b6f627bea28d8ddb (table default.Table_a0e83c6959d64160a5fdec67e0e49ba3 [id=f3735a7cf9634c11abc2c8121e2ed930]). Expected version 1 got 0
I20250901 01:52:13.353019 448 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 061b760d8e6647c6ab0b5d538840a230 (table default.Table_fbba493de02249499d26467bf15c3d5f [id=748daa63137c48a5b151bd34dc5b13fe]). Expected version 1 got 0
I20250901 01:52:13.353384 448 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 24ddc13d1bb04db4a7850acbff091577 (table default.Table_8c1924d21a91425cbbbd8429d56cf0ab [id=18aecbf3fe4643c4b589ee47706161bd]). Expected version 2 got 0
I20250901 01:52:13.353619 448 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 4fd0e4f9d86c4362af96ce9964b72328 (table default.Table_d4d399ae87234df7a710191d0ecf02d0 [id=f48fd53522234290b6c0373fd7b02570]). Expected version 1 got 0
I20250901 01:52:13.356665 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 60285f2ba037433999f5455fb9cc7acd
I20250901 01:52:13.359249 455 master_service.cc:948] ReplaceTablet: received request to replace tablet 60285f2ba037433999f5455fb9cc7acd from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:13.361075 1332 raft_consensus.cc:2241] T 116f65a42ab34bf6b8f8113cb7afdc21 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:13.361585 1332 raft_consensus.cc:2270] T 116f65a42ab34bf6b8f8113cb7afdc21 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:13.366689 448 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 85322a864d154f299f2d670fdd2a7bfb (table default.Table_8df8ec915ec84331933c6de7ed2983b3 [id=1d8f34cc14cb4e90a562ff4424b8577a]). Expected version 3 got 0
I20250901 01:52:13.369031 899 tablet_service.cc:1515] Processing DeleteTablet for tablet 6c15cd14831f422eb615038603c79ed0 with delete_type TABLET_DATA_DELETED (replaced by tablet 824229183e024feaa975c7f0387c9c1a) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:13.369336 904 tablet_service.cc:1515] Processing DeleteTablet for tablet 0e698923812841b7a5d37d57da103c5c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:10 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:13.369472 898 tablet_service.cc:1515] Processing DeleteTablet for tablet d945b310fe6344af9cc9dd68f84c4bdf with delete_type TABLET_DATA_DELETED (replaced by tablet f06a77ecc841462396a8d5026c7327f4) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:13.369565 894 tablet_service.cc:1515] Processing DeleteTablet for tablet b29e846dfae44f0e9ce3e5fea3763a4f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:09 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:13.369740 899 tablet_service.cc:1515] Processing DeleteTablet for tablet 60285f2ba037433999f5455fb9cc7acd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:13 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:13.369881 903 tablet_service.cc:1515] Processing DeleteTablet for tablet 360755d30b5d47ba98c9c7e23ff5ab7e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:10 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:13.370445 1150 tablet_bootstrap.cc:492] T 88a6ab29d00542aba54f0e4185fbf615 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:13.370620 1150 ts_tablet_manager.cc:1397] T 88a6ab29d00542aba54f0e4185fbf615 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.048s user 0.002s sys 0.002s
I20250901 01:52:13.370904 903 tablet_service.cc:1468] Processing CreateTablet for tablet 82d163681fda47f38d79863bd28f576d (DEFAULT_TABLE table=default.Table_dfd01d6b61ac43538c37f74330579cbd [id=b067d050196e4185b76c016c93f27c02]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:13.371307 903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82d163681fda47f38d79863bd28f576d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:13.371444 1150 raft_consensus.cc:357] T 88a6ab29d00542aba54f0e4185fbf615 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:13.371639 1150 raft_consensus.cc:383] T 88a6ab29d00542aba54f0e4185fbf615 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:13.371732 1150 raft_consensus.cc:738] T 88a6ab29d00542aba54f0e4185fbf615 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:13.365111 1151 tablet_bootstrap.cc:492] T 4d333b670df84f308f0cbf6da636b768 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:13.374058 899 tablet_service.cc:1468] Processing CreateTablet for tablet 606fac5c6efe48d1877104b01c23ba5b (DEFAULT_TABLE table=default.Table_fb2584edbb714913a6766458dffef25f [id=91472f220fe44297b79d3eec8b9b80e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:13.374248 1151 ts_tablet_manager.cc:1397] T 4d333b670df84f308f0cbf6da636b768 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.217s user 0.013s sys 0.017s
I20250901 01:52:13.374435 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 606fac5c6efe48d1877104b01c23ba5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:13.375224 1332 ts_tablet_manager.cc:1905] T 116f65a42ab34bf6b8f8113cb7afdc21 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:13.376945 448 catalog_manager.cc:6127] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: default.Table_300cde3d9237488ba65cab60dd8f8721 [id=a48c2b3f8c4847cf9f9cf29548db5c50] alter complete (version 1)
I20250901 01:52:13.377938 1150 consensus_queue.cc:260] T 88a6ab29d00542aba54f0e4185fbf615 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:13.378813 1150 ts_tablet_manager.cc:1428] T 88a6ab29d00542aba54f0e4185fbf615 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250901 01:52:13.379496 1150 tablet_bootstrap.cc:492] T 9496ec6021784800aef62996b0dee593 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:13.380497 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 84fe314502444bd88b038b87eead9cb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:12 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:13.383412 1150 tablet_bootstrap.cc:654] T 9496ec6021784800aef62996b0dee593 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:13.385037 455 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_fb2584edbb714913a6766458dffef25f" } new_table_name: "default.Table_7c12d66048a6470e8c227dc1d9f45fa4" modify_external_catalogs: true
I20250901 01:52:13.389608 1306 ts_tablet_manager.cc:1918] T 97e40c318f64455d91b0e4f8866990ce P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:13.390440 453 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_52927bd3bdf4467581d6d963a046e0a7" } new_table_name: "default.Table_0e63d21acfcb4c028c9ea6406b6a04e4" modify_external_catalogs: true
W20250901 01:52:13.391672 992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:40731, user_credentials={real_user=slave}} blocked reactor thread for 53347.4us
W20250901 01:52:13.393913 455 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 606fac5c6efe48d1877104b01c23ba5b consensus state has no leader
I20250901 01:52:13.394769 448 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_dd24f0428569419fad1bbe8a7f9f08c1" } new_table_name: "default.Table_d3ec2cb111794bb48cac5240950ea847" modify_external_catalogs: true
I20250901 01:52:13.394953 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 4a87e68850c4459eb2c8eafac89c6aa9 (DEFAULT_TABLE table=default.Table_58fc08621bb54011abf9c7484939eb75 [id=cecdd8d6ca6d445c9fd82decd2a95366]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:13.395442 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a87e68850c4459eb2c8eafac89c6aa9. 1 dirs total, 0 dirs full, 0 dirs failed
W20250901 01:52:13.396386 453 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b541a6e14f114ab29a864cca11806316 consensus state has no leader
I20250901 01:52:13.398751 1034 tablet_service.cc:1515] Processing DeleteTablet for tablet 4d27fb61943c45d4aca67689cc3a9ff3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:13 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:13.399067 1460 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 01:52:13.399187 1034 tablet_service.cc:1515] Processing DeleteTablet for tablet 22691943a134400c8df4617240e5d3ef with delete_type TABLET_DATA_DELETED (replaced by tablet 438cd35ff3f74c49b908d072560077db) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:13.399538 1034 tablet_service.cc:1515] Processing DeleteTablet for tablet 1f18420261ae41459e1f25b927bd84a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:13 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:13.399855 1034 tablet_service.cc:1515] Processing DeleteTablet for tablet 58c0e7a2e90e48528bc2bc7ec889f608 with delete_type TABLET_DATA_DELETED (replaced by tablet a6445b29591548be9f9577db1ba0bfa5) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:13.400225 1034 tablet_service.cc:1515] Processing DeleteTablet for tablet d945b310fe6344af9cc9dd68f84c4bdf with delete_type TABLET_DATA_DELETED (replaced by tablet f06a77ecc841462396a8d5026c7327f4) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:13.401202 1034 tablet_service.cc:1515] Processing DeleteTablet for tablet 60285f2ba037433999f5455fb9cc7acd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:13 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:13.401942 1034 tablet_service.cc:1468] Processing CreateTablet for tablet 82d163681fda47f38d79863bd28f576d (DEFAULT_TABLE table=default.Table_dfd01d6b61ac43538c37f74330579cbd [id=b067d050196e4185b76c016c93f27c02]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:13.402463 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82d163681fda47f38d79863bd28f576d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:13.407862 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_113664a018e84769a7ff61aa35a38618"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:13.408550 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_113664a018e84769a7ff61aa35a38618 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:13.411054 1255 fs_manager.cc:730] Time spent opening block manager: real 0.082s user 0.004s sys 0.000s
I20250901 01:52:13.411306 1255 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/master-2/data,/tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/master-2/wal
uuid: "7ce2288f784044eea010664e3def133f"
format_stamp: "Formatted at 2025-09-01 01:52:05 on dist-test-slave-l1vd"
I20250901 01:52:13.413056 1255 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20250901 01:52:13.413805 448 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 438cd35ff3f74c49b908d072560077db has no consensus state
I20250901 01:52:13.415673 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_71659da6afd8410fafb7a26532d52c9e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:13.416335 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_71659da6afd8410fafb7a26532d52c9e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:13.417161 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_da8cc5e7d1e24421b170ecca2fc2147b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:13.417843 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_da8cc5e7d1e24421b170ecca2fc2147b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:13.391477 894 tablet_service.cc:1468] Processing CreateTablet for tablet 98cca86164db4252b1b49de88747822a (DEFAULT_TABLE table=default.Table_8a4599f08716440e947a2d356d6363a5 [id=808a8849c99e485b8be7cd17397874ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:13.401777 899 tablet_service.cc:1468] Processing CreateTablet for tablet abe27e749fa0485195e0064a49d5431e (DEFAULT_TABLE table=default.Table_30943b75394441a5a55b195055d1e0cc [id=7e7a4771910546f7ba8c4369c768c943]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:13.408507 904 tablet_service.cc:1468] Processing CreateTablet for tablet f50f33de1b654663b9ef8a91e3436f11 (DEFAULT_TABLE table=default.Table_4f1c4695ad9e4f24a31d3522281a41a5 [id=9af016a062a64218ad3b62e131d8b65d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:13.410305 902 tablet_service.cc:1468] Processing CreateTablet for tablet 5a8c21aee9f84216b6f90538a5bf4c17 (DEFAULT_TABLE table=default.Table_d6119d1604674319914fa8cb698f16c5 [id=40d05d90b0354334840ad17764f5b6bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:13.418877 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_95884295301d476fb3e03054727f9cf1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:13.411875 896 tablet_service.cc:1468] Processing CreateTablet for tablet 2ff914352c7c43de800f3d54ecf63072 (DEFAULT_TABLE table=default.Table_1426e76865454f76a0529397d709ac7f [id=74d89178d7594613ba29d2bb01358e80]), partition=RANGE (key) PARTITION UNBOUNDED
W20250901 01:52:13.419333 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_95884295301d476fb3e03054727f9cf1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:13.413344 898 tablet_service.cc:1468] Processing CreateTablet for tablet c9a4c80bec1d41458974a0c449aecaf7 (DEFAULT_TABLE table=default.Table_117c6c43b552476dbb8b7b5ccdce47b7 [id=a4c947d85200437395c66bb4f7e607af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:13.420202 1306 log.cc:1199] T 97e40c318f64455d91b0e4f8866990ce P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/97e40c318f64455d91b0e4f8866990ce
I20250901 01:52:13.421914 1031 tablet_service.cc:1468] Processing CreateTablet for tablet abe27e749fa0485195e0064a49d5431e (DEFAULT_TABLE table=default.Table_30943b75394441a5a55b195055d1e0cc [id=7e7a4771910546f7ba8c4369c768c943]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:13.422386 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abe27e749fa0485195e0064a49d5431e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:13.430732 1431 raft_consensus.cc:491] T 360755d30b5d47ba98c9c7e23ff5ab7e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:13.430948 1431 raft_consensus.cc:513] T 360755d30b5d47ba98c9c7e23ff5ab7e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:13.431814 1431 leader_election.cc:290] T 360755d30b5d47ba98c9c7e23ff5ab7e P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:13.434028 903 tablet_service.cc:1515] Processing DeleteTablet for tablet 88a6ab29d00542aba54f0e4185fbf615 with delete_type TABLET_DATA_DELETED (replaced by tablet a774cc6ae3924d299127fd9ffb01ca57) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:13.435396 1033 tablet_service.cc:1468] Processing CreateTablet for tablet c9a4c80bec1d41458974a0c449aecaf7 (DEFAULT_TABLE table=default.Table_117c6c43b552476dbb8b7b5ccdce47b7 [id=a4c947d85200437395c66bb4f7e607af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:13.435518 1150 tablet_bootstrap.cc:492] T 9496ec6021784800aef62996b0dee593 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:13.435665 1150 ts_tablet_manager.cc:1397] T 9496ec6021784800aef62996b0dee593 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.056s user 0.002s sys 0.002s
I20250901 01:52:13.435874 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9a4c80bec1d41458974a0c449aecaf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:13.436769 1150 raft_consensus.cc:357] T 9496ec6021784800aef62996b0dee593 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:13.437058 1150 raft_consensus.cc:383] T 9496ec6021784800aef62996b0dee593 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:13.437160 1150 raft_consensus.cc:738] T 9496ec6021784800aef62996b0dee593 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:13.438628 1150 consensus_queue.cc:260] T 9496ec6021784800aef62996b0dee593 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:13.444108 1150 ts_tablet_manager.cc:1428] T 9496ec6021784800aef62996b0dee593 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s
I20250901 01:52:13.446902 1150 tablet_bootstrap.cc:492] T 31390f38258c465cb1798615a1fcc790 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:13.447971 1431 raft_consensus.cc:491] T 9496ec6021784800aef62996b0dee593 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:13.448179 1431 raft_consensus.cc:513] T 9496ec6021784800aef62996b0dee593 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:13.449375 1431 leader_election.cc:290] T 9496ec6021784800aef62996b0dee593 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:13.450802 1034 tablet_service.cc:1468] Processing CreateTablet for tablet 606fac5c6efe48d1877104b01c23ba5b (DEFAULT_TABLE table=default.Table_fb2584edbb714913a6766458dffef25f [id=91472f220fe44297b79d3eec8b9b80e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:13.451982 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 606fac5c6efe48d1877104b01c23ba5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:13.456353 1031 tablet_service.cc:1468] Processing CreateTablet for tablet 98cca86164db4252b1b49de88747822a (DEFAULT_TABLE table=default.Table_8a4599f08716440e947a2d356d6363a5 [id=808a8849c99e485b8be7cd17397874ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:13.456980 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98cca86164db4252b1b49de88747822a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:13.460471 1151 raft_consensus.cc:357] T 4d333b670df84f308f0cbf6da636b768 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } }
I20250901 01:52:13.460855 1151 raft_consensus.cc:383] T 4d333b670df84f308f0cbf6da636b768 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:13.460891 903 tablet_service.cc:1468] Processing CreateTablet for tablet b009cc95c819431ea9827a3e453b8da3 (DEFAULT_TABLE table=default.Table_7bb211d400ca44d48416f325df223fa6 [id=cc26c89db2484a6fad919231116285f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:13.461027 1151 raft_consensus.cc:738] T 4d333b670df84f308f0cbf6da636b768 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:13.461341 1151 consensus_queue.cc:260] T 4d333b670df84f308f0cbf6da636b768 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } }
I20250901 01:52:13.460150 1306 ts_tablet_manager.cc:1939] T 97e40c318f64455d91b0e4f8866990ce P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:13.462122 1151 ts_tablet_manager.cc:1428] T 4d333b670df84f308f0cbf6da636b768 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.088s user 0.002s sys 0.000s
I20250901 01:52:13.464793 448 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_685b0eabd08542a98940dc6cbbf00b92" } new_table_name: "default.Table_a89cc8e48563460baabc34c821ceae35" modify_external_catalogs: true
I20250901 01:52:13.467113 1151 tablet_bootstrap.cc:492] T 4cdd369214ef4f5ebabddbf28b2ae3f7 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:13.467109 451 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_ae7ef97906764641b443ff461e4aedf2" } new_table_name: "default.Table_0877ac8202a24e9da650a589bc6dbfcc" modify_external_catalogs: true
I20250901 01:52:13.468034 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_b480de33d02948349d86df945c0fb525"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:13.468649 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b480de33d02948349d86df945c0fb525 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:13.478597 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 97e40c318f64455d91b0e4f8866990ce (table default.Table_4f1c4695ad9e4f24a31d3522281a41a5 [id=9af016a062a64218ad3b62e131d8b65d]) successfully deleted
I20250901 01:52:13.479754 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98cca86164db4252b1b49de88747822a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:13.480428 1035 tablet_service.cc:1468] Processing CreateTablet for tablet 5a8c21aee9f84216b6f90538a5bf4c17 (DEFAULT_TABLE table=default.Table_d6119d1604674319914fa8cb698f16c5 [id=40d05d90b0354334840ad17764f5b6bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:13.480724 903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b009cc95c819431ea9827a3e453b8da3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:13.480921 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a8c21aee9f84216b6f90538a5bf4c17. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:13.484747 904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f50f33de1b654663b9ef8a91e3436f11. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:13.484823 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_eacb38fbc98046dfac27344cfbed0a7c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:13.485610 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eacb38fbc98046dfac27344cfbed0a7c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:13.488368 447 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_79c2dbec23ad41159b38d5177072d78c" } modify_external_catalogs: true
I20250901 01:52:13.488595 447 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_79c2dbec23ad41159b38d5177072d78c" } modify_external_catalogs: true
I20250901 01:52:13.488781 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ff914352c7c43de800f3d54ecf63072. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:13.491598 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_7736ed4b5a0d4b1a816500c525e68ad3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:13.492148 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7736ed4b5a0d4b1a816500c525e68ad3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:13.492673 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a8c21aee9f84216b6f90538a5bf4c17. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:13.492326 1033 tablet_service.cc:1468] Processing CreateTablet for tablet f50f33de1b654663b9ef8a91e3436f11 (DEFAULT_TABLE table=default.Table_4f1c4695ad9e4f24a31d3522281a41a5 [id=9af016a062a64218ad3b62e131d8b65d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:13.493148 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f50f33de1b654663b9ef8a91e3436f11. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:13.495855 450 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_7f507bbabca54260a4a5d9092c274e41" } modify_external_catalogs: true
I20250901 01:52:13.496047 450 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_7f507bbabca54260a4a5d9092c274e41" } modify_external_catalogs: true
I20250901 01:52:13.496850 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abe27e749fa0485195e0064a49d5431e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:13.499034 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9a4c80bec1d41458974a0c449aecaf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:13.503669 1029 tablet_service.cc:1468] Processing CreateTablet for tablet b009cc95c819431ea9827a3e453b8da3 (DEFAULT_TABLE table=default.Table_7bb211d400ca44d48416f325df223fa6 [id=cc26c89db2484a6fad919231116285f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:13.504153 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b009cc95c819431ea9827a3e453b8da3. 1 dirs total, 0 dirs full, 0 dirs failed
W20250901 01:52:13.504783 442 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.60:44231, user_credentials={real_user=slave}} blocked reactor thread for 46811.3us
I20250901 01:52:13.505172 449 master_service.cc:948] ReplaceTablet: received request to replace tablet 12c04e0c1f804179ae23509cdb24d395 from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:13.505954 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_b23a4778615d4597950ce1709ab00d37"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:13.506570 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b23a4778615d4597950ce1709ab00d37 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:13.510345 1150 tablet_bootstrap.cc:654] T 31390f38258c465cb1798615a1fcc790 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:13.510962 1035 tablet_service.cc:1468] Processing CreateTablet for tablet 2ff914352c7c43de800f3d54ecf63072 (DEFAULT_TABLE table=default.Table_1426e76865454f76a0529397d709ac7f [id=74d89178d7594613ba29d2bb01358e80]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:13.511508 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ff914352c7c43de800f3d54ecf63072. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:13.514432 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_ac8bd7697e9c4195b4a16a6a44e9594b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:13.515079 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ac8bd7697e9c4195b4a16a6a44e9594b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250901 01:52:13.516113 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet dd694cfa09564d3da5d279c0861a4c32 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dd694cfa09564d3da5d279c0861a4c32 already in progress: creating tablet
I20250901 01:52:13.517247 1448 tablet.cc:1722] T 24ddc13d1bb04db4a7850acbff091577 P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250901 01:52:13.518182 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 3df908c3096a4df7bddd8ac88b49cfe0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3df908c3096a4df7bddd8ac88b49cfe0 already in progress: creating tablet
#23 0x7fe7929cf2ce in std::function<void (google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*)>::operator()(google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*) const ../../../include/c++/7.5.0/bits/std_function.h:706:14
W20250901 01:52:13.518848 440 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet c297e8bf11ff467a854eab41117ac376 because the tablet was not found. No further retry: Not found: Tablet not found: c297e8bf11ff467a854eab41117ac376
I20250901 01:52:13.524021 1332 ts_tablet_manager.cc:1918] T 116f65a42ab34bf6b8f8113cb7afdc21 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 01:52:13.524197 1332 log.cc:1199] T 116f65a42ab34bf6b8f8113cb7afdc21 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/116f65a42ab34bf6b8f8113cb7afdc21
W20250901 01:52:13.526675 440 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 7a8a3aca289c4735abeb9c95c414ab76 because the tablet was not found. No further retry: Not found: Tablet not found: 7a8a3aca289c4735abeb9c95c414ab76
W20250901 01:52:13.526980 440 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 4f8e3fcaa19740d397f4d118ac4cbdb0 because the tablet was not found. No further retry: Not found: Tablet not found: 4f8e3fcaa19740d397f4d118ac4cbdb0
W20250901 01:52:13.527240 440 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 77405097e68344628f4eda667afaed21 because the tablet was not found. No further retry: Not found: Tablet not found: 77405097e68344628f4eda667afaed21
I20250901 01:52:13.527956 1374 raft_consensus.cc:491] T f0affc7a78634607852a48458412761d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:13.528208 1374 raft_consensus.cc:513] T f0affc7a78634607852a48458412761d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } }
I20250901 01:52:13.528569 1306 tablet_replica.cc:331] T 116f65a42ab34bf6b8f8113cb7afdc21 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:13.528877 1306 raft_consensus.cc:2241] T 116f65a42ab34bf6b8f8113cb7afdc21 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:13.529155 1306 raft_consensus.cc:2270] T 116f65a42ab34bf6b8f8113cb7afdc21 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:13.530470 1374 leader_election.cc:290] T f0affc7a78634607852a48458412761d P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561), 0cf599f74322478d91475df234e58965 (127.0.104.1:40731)
I20250901 01:52:13.531965 1306 ts_tablet_manager.cc:1905] T 116f65a42ab34bf6b8f8113cb7afdc21 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:13.538620 992 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250901 01:52:13.541306 448 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c95798f0daf4437db18a291c4464522e has no consensus state
W20250901 01:52:13.541960 451 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f62d96c37cca4143a4b098b7c69979e6 has no consensus state
I20250901 01:52:13.543777 448 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_4f1c4695ad9e4f24a31d3522281a41a5" } modify_external_catalogs: true
I20250901 01:52:13.544014 448 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_4f1c4695ad9e4f24a31d3522281a41a5" } modify_external_catalogs: true
I20250901 01:52:13.545135 452 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_1f8f7c00026749f593d89bc9e92f71cc" } modify_external_catalogs: true
I20250901 01:52:13.545337 452 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_1f8f7c00026749f593d89bc9e92f71cc" } modify_external_catalogs: true
I20250901 01:52:13.546176 453 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 24ddc13d1bb04db4a7850acbff091577 (table default.Table_8c1924d21a91425cbbbd8429d56cf0ab [id=18aecbf3fe4643c4b589ee47706161bd]). Expected version 2 got 0
I20250901 01:52:13.550578 1058 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d945b310fe6344af9cc9dd68f84c4bdf" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:13.552100 1058 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "360755d30b5d47ba98c9c7e23ff5ab7e" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:13.554395 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_ca246ea8f871412d949125b3714d8b67"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:13.555151 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ca246ea8f871412d949125b3714d8b67 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:13.558339 1058 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9496ec6021784800aef62996b0dee593" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:13.560555 446 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_b7bffcbb78614d3cad2b1d375d7a97cc" } modify_external_catalogs: true
I20250901 01:52:13.560766 446 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_b7bffcbb78614d3cad2b1d375d7a97cc" } modify_external_catalogs: true
I20250901 01:52:13.562520 1452 tablet.cc:1722] T 24ddc13d1bb04db4a7850acbff091577 P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250901 01:52:13.578102 1306 ts_tablet_manager.cc:1918] T 116f65a42ab34bf6b8f8113cb7afdc21 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 01:52:13.578264 1306 log.cc:1199] T 116f65a42ab34bf6b8f8113cb7afdc21 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/116f65a42ab34bf6b8f8113cb7afdc21
I20250901 01:52:13.578655 1306 ts_tablet_manager.cc:1939] T 116f65a42ab34bf6b8f8113cb7afdc21 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:13.578903 454 master_service.cc:948] ReplaceTablet: received request to replace tablet 533d14f5903e49599a34ea353ffd8c00 from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:13.579221 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 116f65a42ab34bf6b8f8113cb7afdc21 (table default.Table_57db59e62bcb45a8b2827445e56dc7de [id=70e493e2ad6b4c988f1d78784f892a35]) successfully deleted
I20250901 01:52:13.579658 923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0affc7a78634607852a48458412761d" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:13.579900 923 raft_consensus.cc:2466] T f0affc7a78634607852a48458412761d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
W20250901 01:52:13.581076 859 leader_election.cc:343] T d945b310fe6344af9cc9dd68f84c4bdf P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:13.581152 451 master_service.cc:948] ReplaceTablet: received request to replace tablet c9a4c80bec1d41458974a0c449aecaf7 from {username='slave'} at 127.0.0.1:57810
W20250901 01:52:13.581614 859 leader_election.cc:343] T 360755d30b5d47ba98c9c7e23ff5ab7e P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
W20250901 01:52:13.582039 859 leader_election.cc:343] T 9496ec6021784800aef62996b0dee593 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
W20250901 01:52:13.583055 992 consensus_peers.cc:489] T 4f8e3fcaa19740d397f4d118ac4cbdb0 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Couldn't send request to peer 00c282f779c1477d920f24dc0dc16628. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4f8e3fcaa19740d397f4d118ac4cbdb0. This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:13.583374 992 leader_election.cc:304] T f0affc7a78634607852a48458412761d P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:13.583884 1374 raft_consensus.cc:2802] T f0affc7a78634607852a48458412761d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:13.584084 1374 raft_consensus.cc:491] T f0affc7a78634607852a48458412761d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:13.584282 1374 raft_consensus.cc:3058] T f0affc7a78634607852a48458412761d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:13.584964 455 master_service.cc:948] ReplaceTablet: received request to replace tablet 31390f38258c465cb1798615a1fcc790 from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:13.585543 1150 tablet_bootstrap.cc:492] T 31390f38258c465cb1798615a1fcc790 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:13.585839 1150 ts_tablet_manager.cc:1397] T 31390f38258c465cb1798615a1fcc790 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.140s user 0.014s sys 0.018s
I20250901 01:52:13.586820 1150 raft_consensus.cc:357] T 31390f38258c465cb1798615a1fcc790 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:13.587049 1150 raft_consensus.cc:383] T 31390f38258c465cb1798615a1fcc790 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:13.587129 1150 raft_consensus.cc:738] T 31390f38258c465cb1798615a1fcc790 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:13.587275 1374 raft_consensus.cc:513] T f0affc7a78634607852a48458412761d P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } }
I20250901 01:52:13.587436 1150 consensus_queue.cc:260] T 31390f38258c465cb1798615a1fcc790 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:13.588060 1150 ts_tablet_manager.cc:1428] T 31390f38258c465cb1798615a1fcc790 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250901 01:52:13.588415 1150 tablet_bootstrap.cc:492] T 22691943a134400c8df4617240e5d3ef P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:13.590503 1150 tablet_bootstrap.cc:654] T 22691943a134400c8df4617240e5d3ef P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:13.592844 1306 tablet_replica.cc:331] T 4cdd369214ef4f5ebabddbf28b2ae3f7 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
W20250901 01:52:13.592808 440 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 3eaf8baac9f1485f944cd9e74b8b5569 because the tablet was not found. No further retry: Not found: Tablet not found: 3eaf8baac9f1485f944cd9e74b8b5569
I20250901 01:52:13.593060 1306 raft_consensus.cc:2241] T 4cdd369214ef4f5ebabddbf28b2ae3f7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250901 01:52:13.593246 440 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 4d333b670df84f308f0cbf6da636b768 because the tablet was not found. No further retry: Not found: Tablet not found: 4d333b670df84f308f0cbf6da636b768
I20250901 01:52:13.593266 1306 raft_consensus.cc:2270] T 4cdd369214ef4f5ebabddbf28b2ae3f7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:13.593938 1306 ts_tablet_manager.cc:1905] T 4cdd369214ef4f5ebabddbf28b2ae3f7 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:13.598656 449 catalog_manager.cc:6127] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: default.Table_8c1924d21a91425cbbbd8429d56cf0ab [id=18aecbf3fe4643c4b589ee47706161bd] alter complete (version 2)
I20250901 01:52:13.599177 449 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_6aa17d35edb04da19ea5d0ce969b34e2" } new_table_name: "default.Table_537be4a82c0348f6952b911b167d02ad" modify_external_catalogs: true
I20250901 01:52:13.600097 1150 tablet_bootstrap.cc:492] T 22691943a134400c8df4617240e5d3ef P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:13.600270 1150 ts_tablet_manager.cc:1397] T 22691943a134400c8df4617240e5d3ef P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.001s
I20250901 01:52:13.601379 1150 raft_consensus.cc:357] T 22691943a134400c8df4617240e5d3ef P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:13.601644 1150 raft_consensus.cc:383] T 22691943a134400c8df4617240e5d3ef P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:13.601763 1150 raft_consensus.cc:738] T 22691943a134400c8df4617240e5d3ef P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:13.602020 1150 consensus_queue.cc:260] T 22691943a134400c8df4617240e5d3ef P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:13.602517 1150 ts_tablet_manager.cc:1428] T 22691943a134400c8df4617240e5d3ef P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250901 01:52:13.602807 1150 tablet_bootstrap.cc:492] T 7cc1efcb6bff4aba8015d2d3b7f4c923 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:13.609755 1150 tablet_bootstrap.cc:654] T 7cc1efcb6bff4aba8015d2d3b7f4c923 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:13.614652 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 31390f38258c465cb1798615a1fcc790
I20250901 01:52:13.615207 455 catalog_manager.cc:6652] ReplaceTablet: tablet 31390f38258c465cb1798615a1fcc790 deleted and replaced by tablet 4f4be080f20d41c798835b7111a5318d
I20250901 01:52:13.617529 1151 tablet_bootstrap.cc:654] T 4cdd369214ef4f5ebabddbf28b2ae3f7 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:13.617986 904 tablet_service.cc:1515] Processing DeleteTablet for tablet 31390f38258c465cb1798615a1fcc790 with delete_type TABLET_DATA_DELETED (replaced by tablet 4f4be080f20d41c798835b7111a5318d) from {username='slave'} at 127.0.0.1:52846
W20250901 01:52:13.619853 449 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 920925ee9b014e14a79ceb7796ff7828 consensus state has no leader
I20250901 01:52:13.620434 455 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 24ddc13d1bb04db4a7850acbff091577 (table default.Table_8c1924d21a91425cbbbd8429d56cf0ab [id=18aecbf3fe4643c4b589ee47706161bd]). Expected version 2 got 0
I20250901 01:52:13.620362 449 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_4ce8198ebbed4cc4beba453f1faf486a" } new_table_name: "default.Table_ac4ff85b5f09468d870ec5fd9b9abd93" modify_external_catalogs: true
I20250901 01:52:13.624428 455 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_b480de33d02948349d86df945c0fb525" } new_table_name: "default.Table_fad16c8446b049ec9b4a90aba2ef14de" modify_external_catalogs: true
I20250901 01:52:13.625053 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 31390f38258c465cb1798615a1fcc790 with delete_type TABLET_DATA_DELETED (replaced by tablet 4f4be080f20d41c798835b7111a5318d) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:13.625746 1374 leader_election.cc:290] T f0affc7a78634607852a48458412761d P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561), 0cf599f74322478d91475df234e58965 (127.0.104.1:40731)
I20250901 01:52:13.626250 1150 tablet_bootstrap.cc:492] T 7cc1efcb6bff4aba8015d2d3b7f4c923 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:13.626441 1150 ts_tablet_manager.cc:1397] T 7cc1efcb6bff4aba8015d2d3b7f4c923 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.005s
I20250901 01:52:13.627354 1150 raft_consensus.cc:357] T 7cc1efcb6bff4aba8015d2d3b7f4c923 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:13.627553 1150 raft_consensus.cc:383] T 7cc1efcb6bff4aba8015d2d3b7f4c923 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:13.627635 1150 raft_consensus.cc:738] T 7cc1efcb6bff4aba8015d2d3b7f4c923 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:13.627885 1150 consensus_queue.cc:260] T 7cc1efcb6bff4aba8015d2d3b7f4c923 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:13.628496 1150 ts_tablet_manager.cc:1428] T 7cc1efcb6bff4aba8015d2d3b7f4c923 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250901 01:52:13.628893 1150 tablet_bootstrap.cc:492] T d6a4040747fa4d588756506ee122d2dc P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:13.631804 1150 tablet_bootstrap.cc:654] T d6a4040747fa4d588756506ee122d2dc P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
#24 0x7fe7929ce1ae in kudu::rpc::GeneratedServiceIf::Handle(kudu::rpc::InboundCall*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/service_if.cc:137:3
I20250901 01:52:13.634923 923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0affc7a78634607852a48458412761d" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:13.635200 923 raft_consensus.cc:3058] T f0affc7a78634607852a48458412761d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:13.638111 923 raft_consensus.cc:2466] T f0affc7a78634607852a48458412761d P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:13.638563 1306 ts_tablet_manager.cc:1918] T 4cdd369214ef4f5ebabddbf28b2ae3f7 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:13.638724 1306 log.cc:1199] T 4cdd369214ef4f5ebabddbf28b2ae3f7 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/4cdd369214ef4f5ebabddbf28b2ae3f7
I20250901 01:52:13.639171 1306 ts_tablet_manager.cc:1939] T 4cdd369214ef4f5ebabddbf28b2ae3f7 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:13.639501 992 leader_election.cc:304] T f0affc7a78634607852a48458412761d P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:13.639844 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 4cdd369214ef4f5ebabddbf28b2ae3f7 (table default.Table_cf1c9f5fc5104126b163225918707284 [id=00ae7e46c65d4a56b1ffa193b853421a]) successfully deleted
I20250901 01:52:13.639951 1306 tablet_replica.cc:331] T 2a0d125423a043dcb11e67345599e468 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
W20250901 01:52:13.640153 440 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 8d5b10ede9c84be098795c0a787e5a78 because the tablet was not found. No further retry: Not found: Tablet not found: 8d5b10ede9c84be098795c0a787e5a78
I20250901 01:52:13.640230 1306 raft_consensus.cc:2241] T 2a0d125423a043dcb11e67345599e468 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:13.640468 1306 raft_consensus.cc:2270] T 2a0d125423a043dcb11e67345599e468 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:13.641162 1306 ts_tablet_manager.cc:1905] T 2a0d125423a043dcb11e67345599e468 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:13.646072 449 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1286c0ab85704837a99b6e5a62184d77 has no consensus state
I20250901 01:52:13.646222 1033 tablet_service.cc:1515] Processing DeleteTablet for tablet 4d333b670df84f308f0cbf6da636b768 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:08 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:13.647318 1150 tablet_bootstrap.cc:492] T d6a4040747fa4d588756506ee122d2dc P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:13.647495 1150 ts_tablet_manager.cc:1397] T d6a4040747fa4d588756506ee122d2dc P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.001s
I20250901 01:52:13.647222 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_79a6a0f0c6944313b9987f23b6a09557"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:13.648129 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_79a6a0f0c6944313b9987f23b6a09557 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:13.648543 1150 raft_consensus.cc:357] T d6a4040747fa4d588756506ee122d2dc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:13.648805 1150 raft_consensus.cc:383] T d6a4040747fa4d588756506ee122d2dc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:13.648932 1150 raft_consensus.cc:738] T d6a4040747fa4d588756506ee122d2dc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:13.649238 1150 consensus_queue.cc:260] T d6a4040747fa4d588756506ee122d2dc P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:13.649888 1150 ts_tablet_manager.cc:1428] T d6a4040747fa4d588756506ee122d2dc P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250901 01:52:13.650231 1150 tablet_bootstrap.cc:492] T f65ec84958eb488d8b0f812b59b5b3d5 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
W20250901 01:52:13.650740 455 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b2e8b2cd10154de08ddd30c214592a4a has no consensus state
I20250901 01:52:13.651697 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_4aa01c0e60364804a5e99d552ca3956c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:13.652261 1150 tablet_bootstrap.cc:654] T f65ec84958eb488d8b0f812b59b5b3d5 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:13.652429 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4aa01c0e60364804a5e99d552ca3956c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:13.655517 1374 raft_consensus.cc:2802] T f0affc7a78634607852a48458412761d P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:13.657840 1374 raft_consensus.cc:695] T f0affc7a78634607852a48458412761d P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:13.658331 1374 consensus_queue.cc:237] T f0affc7a78634607852a48458412761d P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } }
I20250901 01:52:13.660535 1151 tablet_bootstrap.cc:492] T 4cdd369214ef4f5ebabddbf28b2ae3f7 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:13.660775 1151 ts_tablet_manager.cc:1397] T 4cdd369214ef4f5ebabddbf28b2ae3f7 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.194s user 0.005s sys 0.000s
I20250901 01:52:13.661789 1151 raft_consensus.cc:357] T 4cdd369214ef4f5ebabddbf28b2ae3f7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:13.662128 1151 raft_consensus.cc:383] T 4cdd369214ef4f5ebabddbf28b2ae3f7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:13.662293 1151 raft_consensus.cc:738] T 4cdd369214ef4f5ebabddbf28b2ae3f7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:13.662720 1151 consensus_queue.cc:260] T 4cdd369214ef4f5ebabddbf28b2ae3f7 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:13.665541 1332 ts_tablet_manager.cc:1939] T 116f65a42ab34bf6b8f8113cb7afdc21 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:13.666520 1332 tablet_replica.cc:331] T 7a8a3aca289c4735abeb9c95c414ab76 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:13.666786 1332 raft_consensus.cc:2241] T 7a8a3aca289c4735abeb9c95c414ab76 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:13.667058 1332 raft_consensus.cc:2270] T 7a8a3aca289c4735abeb9c95c414ab76 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:13.667770 1332 ts_tablet_manager.cc:1905] T 7a8a3aca289c4735abeb9c95c414ab76 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:13.669795 702 kernel_stack_watchdog.cc:198] Thread 1149 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 3394ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fe78f073980 (unknown) at ??:0
@ 0x92cb07 __sanitizer::internal_sched_yield() at ??:?
@ 0x8921bd __interceptor_pthread_create at ??:?
@ 0x7fe78bc459c1 kudu::Thread::StartThread() at ??:0
@ 0xa63f97 kudu::Thread::Create() at ??:?
@ 0x7fe78bc6580b kudu::ThreadPool::CreateThread() at ??:0
@ 0x7fe78bc6122d kudu::ThreadPool::DoSubmit() at ??:0
@ 0x7fe78bc6655f kudu::ThreadPool::Submit() at ??:0
@ 0x7fe79701088d kudu::log::SegmentAllocator::AsyncAllocateSegmentUnlocked() at ??:0
@ 0x7fe79700f87f kudu::log::SegmentAllocator::AllocateSegmentAndRollOver() at ??:0
@ 0x7fe79700f43e kudu::log::SegmentAllocator::Init() at ??:0
@ 0x7fe7970174b5 kudu::log::Log::Init() at ??:0
@ 0x7fe797016811 kudu::log::Log::Open() at ??:0
@ 0x7fe79895a5e1 kudu::tablet::TabletBootstrap::OpenNewLog() at ??:0
@ 0x7fe79895694c kudu::tablet::TabletBootstrap::RunBootstrap() at ??:0
@ 0x7fe7989550cf kudu::tablet::TabletBootstrap::Bootstrap() at ??:0
I20250901 01:52:13.673211 1293 raft_consensus.cc:491] T 0e698923812841b7a5d37d57da103c5c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:13.673463 1293 raft_consensus.cc:513] T 0e698923812841b7a5d37d57da103c5c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:13.674226 1293 leader_election.cc:290] T 0e698923812841b7a5d37d57da103c5c P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:13.675503 1306 ts_tablet_manager.cc:1918] T 2a0d125423a043dcb11e67345599e468 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:13.675673 1306 log.cc:1199] T 2a0d125423a043dcb11e67345599e468 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/2a0d125423a043dcb11e67345599e468
I20250901 01:52:13.675997 441 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 116f65a42ab34bf6b8f8113cb7afdc21 (table default.Table_57db59e62bcb45a8b2827445e56dc7de [id=70e493e2ad6b4c988f1d78784f892a35]) successfully deleted
I20250901 01:52:13.676098 1306 ts_tablet_manager.cc:1939] T 2a0d125423a043dcb11e67345599e468 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
W20250901 01:52:13.676270 441 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet c297e8bf11ff467a854eab41117ac376 because the tablet was not found. No further retry: Not found: Tablet not found: c297e8bf11ff467a854eab41117ac376
I20250901 01:52:13.676782 1058 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e698923812841b7a5d37d57da103c5c" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:13.677541 859 leader_election.cc:343] T 0e698923812841b7a5d37d57da103c5c P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:13.693416 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 2a0d125423a043dcb11e67345599e468 (table default.Table_b27678f366d74143b716da42e7ca32c3 [id=500406ff1eb342f6af09fbb6f818e160]) successfully deleted
I20250901 01:52:13.693581 1306 tablet_replica.cc:331] T 447ffe73741640ceb73f35676232a094 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
W20250901 01:52:13.693776 440 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 342b7fde3ede494b9eec6e361db255dc because the tablet was not found. No further retry: Not found: Tablet not found: 342b7fde3ede494b9eec6e361db255dc
I20250901 01:52:13.693857 1306 raft_consensus.cc:2241] T 447ffe73741640ceb73f35676232a094 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:13.694067 1306 raft_consensus.cc:2270] T 447ffe73741640ceb73f35676232a094 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:13.694087 449 master_service.cc:948] ReplaceTablet: received request to replace tablet 6c15cd14831f422eb615038603c79ed0 from {username='slave'} at 127.0.0.1:57810
W20250901 01:52:13.694625 440 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 8e6e96b4de214cadba1ccd19390cffb0 because the tablet was not found. No further retry: Not found: Tablet not found: 8e6e96b4de214cadba1ccd19390cffb0
I20250901 01:52:13.694769 1306 ts_tablet_manager.cc:1905] T 447ffe73741640ceb73f35676232a094 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:13.694837 440 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet bf302e76746b4f7bbe4469914f83720f because the tablet was not found. No further retry: Not found: Tablet not found: bf302e76746b4f7bbe4469914f83720f
W20250901 01:52:13.695127 440 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet fab3c1a8928a4066a15868ba708267f1 because the tablet was not found. No further retry: Not found: Tablet not found: fab3c1a8928a4066a15868ba708267f1
I20250901 01:52:13.696619 1151 ts_tablet_manager.cc:1428] T 4cdd369214ef4f5ebabddbf28b2ae3f7 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250901 01:52:13.698527 1151 tablet_bootstrap.cc:492] T b29e846dfae44f0e9ce3e5fea3763a4f P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:13.701074 1151 tablet_bootstrap.cc:654] T b29e846dfae44f0e9ce3e5fea3763a4f P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
#25 0x7fe7929d43e5 in kudu::rpc::ServicePool::RunThread() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/service_pool.cc:229:15
I20250901 01:52:13.718044 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_b300f31bdab14e24ad40f265ac4de311"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:13.718730 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b300f31bdab14e24ad40f265ac4de311 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:13.722111 1306 ts_tablet_manager.cc:1918] T 447ffe73741640ceb73f35676232a094 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:13.722246 1306 log.cc:1199] T 447ffe73741640ceb73f35676232a094 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/447ffe73741640ceb73f35676232a094
I20250901 01:52:13.722627 1306 ts_tablet_manager.cc:1939] T 447ffe73741640ceb73f35676232a094 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:13.723433 1306 tablet_replica.cc:331] T 9496ec6021784800aef62996b0dee593 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:13.723397 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 447ffe73741640ceb73f35676232a094 (table default.Table_ac8663b3415c4c668e298a84b2deb8c3 [id=33cd13ca007c4308957250255f985e70]) successfully deleted
I20250901 01:52:13.723644 1306 raft_consensus.cc:2241] T 9496ec6021784800aef62996b0dee593 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250901 01:52:13.723728 440 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 288e8029382f4154bc1e2103b3035a60 because the tablet was not found. No further retry: Not found: Tablet not found: 288e8029382f4154bc1e2103b3035a60
I20250901 01:52:13.723814 1306 raft_consensus.cc:2270] T 9496ec6021784800aef62996b0dee593 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:13.723987 1255 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
W20250901 01:52:13.724126 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 154574e17f214cc0ad56e6e6be3af899 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 154574e17f214cc0ad56e6e6be3af899 already in progress: creating tablet
I20250901 01:52:13.725330 1255 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 01:52:13.726431 1150 tablet_bootstrap.cc:492] T f65ec84958eb488d8b0f812b59b5b3d5 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:13.726742 1150 ts_tablet_manager.cc:1397] T f65ec84958eb488d8b0f812b59b5b3d5 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.077s user 0.010s sys 0.018s
I20250901 01:52:13.727689 1150 raft_consensus.cc:357] T f65ec84958eb488d8b0f812b59b5b3d5 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:13.727993 1150 raft_consensus.cc:383] T f65ec84958eb488d8b0f812b59b5b3d5 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:13.728116 1150 raft_consensus.cc:738] T f65ec84958eb488d8b0f812b59b5b3d5 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:13.728485 1150 consensus_queue.cc:260] T f65ec84958eb488d8b0f812b59b5b3d5 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
#26 0x7fe7929d4d0f in kudu::rpc::ServicePool::Init(int)::$_0::operator()() const /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/service_pool.cc:92:5
I20250901 01:52:13.729271 1150 ts_tablet_manager.cc:1428] T f65ec84958eb488d8b0f812b59b5b3d5 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250901 01:52:13.729660 1150 tablet_bootstrap.cc:492] T 34042ea5c1b645969c1de41fe587c343 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:13.732396 1150 tablet_bootstrap.cc:654] T 34042ea5c1b645969c1de41fe587c343 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:13.733569 1255 kserver.cc:163] Server-wide thread pool size limit: 3276
#27 0x7fe7929d4b38 in std::_Function_handler<void (), kudu::rpc::ServicePool::Init(int)::$_0>::_M_invoke(std::_Any_data const&) ../../../include/c++/7.5.0/bits/std_function.h:316:2
I20250901 01:52:13.741423 1306 ts_tablet_manager.cc:1905] T 9496ec6021784800aef62996b0dee593 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:13.743366 1332 ts_tablet_manager.cc:1918] T 7a8a3aca289c4735abeb9c95c414ab76 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:13.746922 1150 tablet_bootstrap.cc:492] T 34042ea5c1b645969c1de41fe587c343 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:13.747118 1150 ts_tablet_manager.cc:1397] T 34042ea5c1b645969c1de41fe587c343 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.000s
I20250901 01:52:13.748186 1150 raft_consensus.cc:357] T 34042ea5c1b645969c1de41fe587c343 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:13.748387 1150 raft_consensus.cc:383] T 34042ea5c1b645969c1de41fe587c343 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:13.748512 1150 raft_consensus.cc:738] T 34042ea5c1b645969c1de41fe587c343 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:13.748594 455 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_d3ec2cb111794bb48cac5240950ea847" } new_table_name: "default.Table_bbdd9e12d55540eb885b874ef8358435" modify_external_catalogs: true
I20250901 01:52:13.748754 1150 consensus_queue.cc:260] T 34042ea5c1b645969c1de41fe587c343 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:13.748715 1332 log.cc:1199] T 7a8a3aca289c4735abeb9c95c414ab76 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/7a8a3aca289c4735abeb9c95c414ab76
I20250901 01:52:13.749414 1150 ts_tablet_manager.cc:1428] T 34042ea5c1b645969c1de41fe587c343 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250901 01:52:13.749719 1150 tablet_bootstrap.cc:492] T 58c0e7a2e90e48528bc2bc7ec889f608 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:13.751737 1150 tablet_bootstrap.cc:654] T 58c0e7a2e90e48528bc2bc7ec889f608 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:13.753646 1332 ts_tablet_manager.cc:1939] T 7a8a3aca289c4735abeb9c95c414ab76 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:13.754716 441 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 7a8a3aca289c4735abeb9c95c414ab76 (table default.Table_8df8ec915ec84331933c6de7ed2983b3 [id=1d8f34cc14cb4e90a562ff4424b8577a]) successfully deleted
W20250901 01:52:13.757216 441 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 288e8029382f4154bc1e2103b3035a60 because the tablet was not found. No further retry: Not found: Tablet not found: 288e8029382f4154bc1e2103b3035a60
W20250901 01:52:13.758332 441 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 77405097e68344628f4eda667afaed21 because the tablet was not found. No further retry: Not found: Tablet not found: 77405097e68344628f4eda667afaed21
I20250901 01:52:13.758538 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 533d14f5903e49599a34ea353ffd8c00
I20250901 01:52:13.759187 454 catalog_manager.cc:6652] ReplaceTablet: tablet 533d14f5903e49599a34ea353ffd8c00 deleted and replaced by tablet a54af99bc10045e8b0b6868e5aed98b4
I20250901 01:52:13.759627 904 tablet_service.cc:1515] Processing DeleteTablet for tablet 533d14f5903e49599a34ea353ffd8c00 with delete_type TABLET_DATA_DELETED (replaced by tablet a54af99bc10045e8b0b6868e5aed98b4) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:13.760207 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_dc856980ff4c4aafb92cf1fd4f29e694"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:13.760968 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dc856980ff4c4aafb92cf1fd4f29e694 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:13.762097 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet 533d14f5903e49599a34ea353ffd8c00 with delete_type TABLET_DATA_DELETED (replaced by tablet a54af99bc10045e8b0b6868e5aed98b4) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:13.762424 1332 tablet_replica.cc:331] T 4f8e3fcaa19740d397f4d118ac4cbdb0 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:13.765028 1150 tablet_bootstrap.cc:492] T 58c0e7a2e90e48528bc2bc7ec889f608 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:13.765249 1150 ts_tablet_manager.cc:1397] T 58c0e7a2e90e48528bc2bc7ec889f608 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.003s
I20250901 01:52:13.766093 1150 raft_consensus.cc:357] T 58c0e7a2e90e48528bc2bc7ec889f608 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:13.766309 1150 raft_consensus.cc:383] T 58c0e7a2e90e48528bc2bc7ec889f608 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:13.766397 1150 raft_consensus.cc:738] T 58c0e7a2e90e48528bc2bc7ec889f608 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:13.766691 1150 consensus_queue.cc:260] T 58c0e7a2e90e48528bc2bc7ec889f608 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:13.767225 1150 ts_tablet_manager.cc:1428] T 58c0e7a2e90e48528bc2bc7ec889f608 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250901 01:52:13.767570 1150 tablet_bootstrap.cc:492] T a5f4b0a2d72a4b8095ae4af1fe7c0c5d P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:13.769076 1332 raft_consensus.cc:2241] T 4f8e3fcaa19740d397f4d118ac4cbdb0 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:13.769758 1150 tablet_bootstrap.cc:654] T a5f4b0a2d72a4b8095ae4af1fe7c0c5d P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:13.769979 1332 pending_rounds.cc:70] T 4f8e3fcaa19740d397f4d118ac4cbdb0 P fd27ab6665304aaea53adbd102dd4f58: Trying to abort 1 pending ops.
I20250901 01:52:13.770197 1332 pending_rounds.cc:77] T 4f8e3fcaa19740d397f4d118ac4cbdb0 P fd27ab6665304aaea53adbd102dd4f58: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7195408516370546688 op_type: NO_OP noop_request { }
I20250901 01:52:13.770568 1332 raft_consensus.cc:2887] T 4f8e3fcaa19740d397f4d118ac4cbdb0 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250901 01:52:13.771627 1306 ts_tablet_manager.cc:1918] T 9496ec6021784800aef62996b0dee593 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:13.771791 1306 log.cc:1199] T 9496ec6021784800aef62996b0dee593 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/9496ec6021784800aef62996b0dee593
I20250901 01:52:13.772020 1332 raft_consensus.cc:2270] T 4f8e3fcaa19740d397f4d118ac4cbdb0 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:13.772215 1306 ts_tablet_manager.cc:1939] T 9496ec6021784800aef62996b0dee593 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:13.773231 1332 ts_tablet_manager.cc:1905] T 4f8e3fcaa19740d397f4d118ac4cbdb0 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:13.775059 991 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250901 01:52:13.777086 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 9496ec6021784800aef62996b0dee593 (table default.Table_95c28a4758e148368604152622fe51be [id=bbcca84b801942d094d955c87cafacb4]) successfully deleted
I20250901 01:52:13.777284 1306 tablet_replica.cc:331] T 22691943a134400c8df4617240e5d3ef P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:13.777503 1306 raft_consensus.cc:2241] T 22691943a134400c8df4617240e5d3ef P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:13.777697 1306 raft_consensus.cc:2270] T 22691943a134400c8df4617240e5d3ef P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250901 01:52:13.777777 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 12c04e0c1f804179ae23509cdb24d395 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 12c04e0c1f804179ae23509cdb24d395 already in progress: creating tablet
W20250901 01:52:13.778074 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 4d27fb61943c45d4aca67689cc3a9ff3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4d27fb61943c45d4aca67689cc3a9ff3 already in progress: creating tablet
I20250901 01:52:13.781657 1306 ts_tablet_manager.cc:1905] T 22691943a134400c8df4617240e5d3ef P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:13.781577 454 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_3408ffb7316a4f1da2f045225e881c76" } new_table_name: "default.Table_942175dadf4b496195b2fa53f39f6ddc" modify_external_catalogs: true
I20250901 01:52:13.782051 1150 tablet_bootstrap.cc:492] T a5f4b0a2d72a4b8095ae4af1fe7c0c5d P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:13.782195 1150 ts_tablet_manager.cc:1397] T a5f4b0a2d72a4b8095ae4af1fe7c0c5d P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.003s
I20250901 01:52:13.782984 1150 raft_consensus.cc:357] T a5f4b0a2d72a4b8095ae4af1fe7c0c5d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:13.783195 1150 raft_consensus.cc:383] T a5f4b0a2d72a4b8095ae4af1fe7c0c5d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:13.783285 1150 raft_consensus.cc:738] T a5f4b0a2d72a4b8095ae4af1fe7c0c5d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:13.783528 1150 consensus_queue.cc:260] T a5f4b0a2d72a4b8095ae4af1fe7c0c5d P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:13.784103 1150 ts_tablet_manager.cc:1428] T a5f4b0a2d72a4b8095ae4af1fe7c0c5d P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250901 01:52:13.784380 1150 tablet_bootstrap.cc:492] T 1f18420261ae41459e1f25b927bd84a9 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:13.789971 1150 tablet_bootstrap.cc:654] T 1f18420261ae41459e1f25b927bd84a9 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:13.791574 454 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ddbc15685fca4c708329420109ad0c57 has no consensus state
I20250901 01:52:13.792505 454 master_service.cc:948] ReplaceTablet: received request to replace tablet d6a4040747fa4d588756506ee122d2dc from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:13.793900 453 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 7cc1efcb6bff4aba8015d2d3b7f4c923 (table default.Table_727db72cb2654b679929453b3a8fb00a [id=78b00856a9b747c19f390b4fdae8ddcf]). Expected version 2 got 0
I20250901 01:52:13.795640 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_8022c1172d8a4df8bc1295d193bb6542"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:13.796510 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8022c1172d8a4df8bc1295d193bb6542 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250901 01:52:13.802119 455 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 438cd35ff3f74c49b908d072560077db consensus state has no leader
I20250901 01:52:13.815539 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_142ccc8cc7414621bc0c31326d5aa0e9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:13.816802 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_142ccc8cc7414621bc0c31326d5aa0e9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250901 01:52:13.820964 859 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250901 01:52:13.825614 1306 ts_tablet_manager.cc:1918] T 22691943a134400c8df4617240e5d3ef P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:13.825824 1306 log.cc:1199] T 22691943a134400c8df4617240e5d3ef P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/22691943a134400c8df4617240e5d3ef
I20250901 01:52:13.826361 1306 ts_tablet_manager.cc:1939] T 22691943a134400c8df4617240e5d3ef P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:13.827090 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 22691943a134400c8df4617240e5d3ef (table default.Table_bbdd9e12d55540eb885b874ef8358435 [id=5a33cfdf60684867bb5f36d84ee80468]) successfully deleted
I20250901 01:52:13.827200 1306 tablet_replica.cc:331] T 58c0e7a2e90e48528bc2bc7ec889f608 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:13.827450 1306 raft_consensus.cc:2241] T 58c0e7a2e90e48528bc2bc7ec889f608 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:13.827642 1306 raft_consensus.cc:2270] T 58c0e7a2e90e48528bc2bc7ec889f608 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250901 01:52:13.827672 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 1f18420261ae41459e1f25b927bd84a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1f18420261ae41459e1f25b927bd84a9 already in progress: creating tablet
W20250901 01:52:13.829102 994 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250901 01:52:13.831599 991 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250901 01:52:13.832459 1306 ts_tablet_manager.cc:1905] T 58c0e7a2e90e48528bc2bc7ec889f608 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:13.839046 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_3b78b64e932d45458e89edf72fc45a6d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:13.839875 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3b78b64e932d45458e89edf72fc45a6d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250901 01:52:13.863903 991 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250901 01:52:13.864238 452 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet a7ff996482164776b468cf305989866b
I20250901 01:52:13.865386 904 tablet_service.cc:1515] Processing DeleteTablet for tablet a7ff996482164776b468cf305989866b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:13 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:13.865432 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_202b4c5ccc56474a8b2e7b9d7c4a28d6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:13.866170 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_202b4c5ccc56474a8b2e7b9d7c4a28d6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:13.867157 1306 ts_tablet_manager.cc:1918] T 58c0e7a2e90e48528bc2bc7ec889f608 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:13.867323 1306 log.cc:1199] T 58c0e7a2e90e48528bc2bc7ec889f608 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/58c0e7a2e90e48528bc2bc7ec889f608
I20250901 01:52:13.867771 1306 ts_tablet_manager.cc:1939] T 58c0e7a2e90e48528bc2bc7ec889f608 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:13.868219 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet a7ff996482164776b468cf305989866b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:13 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:13.868592 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 58c0e7a2e90e48528bc2bc7ec889f608 (table default.Table_409dad98d7eb455a8e16edfb4acedb34 [id=6f29cdc9974049368af2ae10beadb772]) successfully deleted
I20250901 01:52:13.874923 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet c9a4c80bec1d41458974a0c449aecaf7
I20250901 01:52:13.875505 451 catalog_manager.cc:6652] ReplaceTablet: tablet c9a4c80bec1d41458974a0c449aecaf7 deleted and replaced by tablet 121803e79ab64ec0b948b448646b9d0a
I20250901 01:52:13.875994 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_8ea6f8f2bd8848109b6a9f9e7f5b6e54"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:13.876427 904 tablet_service.cc:1515] Processing DeleteTablet for tablet c9a4c80bec1d41458974a0c449aecaf7 with delete_type TABLET_DATA_DELETED (replaced by tablet 121803e79ab64ec0b948b448646b9d0a) from {username='slave'} at 127.0.0.1:52846
W20250901 01:52:13.876700 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8ea6f8f2bd8848109b6a9f9e7f5b6e54 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:13.876948 1374 raft_consensus.cc:491] T 342b7fde3ede494b9eec6e361db255dc P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:13.877144 1374 raft_consensus.cc:513] T 342b7fde3ede494b9eec6e361db255dc P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } }
I20250901 01:52:13.877882 1374 leader_election.cc:290] T 342b7fde3ede494b9eec6e361db255dc P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561), 0cf599f74322478d91475df234e58965 (127.0.104.1:40731)
W20250901 01:52:13.879218 992 leader_election.cc:343] T 342b7fde3ede494b9eec6e361db255dc P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 342b7fde3ede494b9eec6e361db255dc
I20250901 01:52:13.879235 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet c9a4c80bec1d41458974a0c449aecaf7 with delete_type TABLET_DATA_DELETED (replaced by tablet 121803e79ab64ec0b948b448646b9d0a) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:13.884138 1031 tablet_service.cc:1468] Processing CreateTablet for tablet e6c10426f60f43e8bde69d9b1b9c8eb6 (DEFAULT_TABLE table=default.Table_3ffe49903e864e8e8cc92d82f4c14a3a [id=d3dbeb4258424133844e20b181ee44e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:13.884716 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6c10426f60f43e8bde69d9b1b9c8eb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:13.893677 904 tablet_service.cc:1468] Processing CreateTablet for tablet e6c10426f60f43e8bde69d9b1b9c8eb6 (DEFAULT_TABLE table=default.Table_3ffe49903e864e8e8cc92d82f4c14a3a [id=d3dbeb4258424133844e20b181ee44e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:13.894244 904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6c10426f60f43e8bde69d9b1b9c8eb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:13.899713 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_18b93109e1224ea59b5c93825b3b27d6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:13.868358 1306 tablet_replica.cc:331] T 6c15cd14831f422eb615038603c79ed0 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:13.900894 1306 raft_consensus.cc:2241] T 6c15cd14831f422eb615038603c79ed0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:13.901127 1306 raft_consensus.cc:2270] T 6c15cd14831f422eb615038603c79ed0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:13.901727 1306 ts_tablet_manager.cc:1905] T 6c15cd14831f422eb615038603c79ed0 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:13.909680 896 tablet_service.cc:1468] Processing CreateTablet for tablet 66a48a8ebe154b2cac89c234eaa378ff (DEFAULT_TABLE table=default.Table_ce16a1b94f714c2699ee6cd916bb0215 [id=551e2f9616d84026b96b6c22507f29de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:13.910166 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66a48a8ebe154b2cac89c234eaa378ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:13.910408 1031 tablet_service.cc:1468] Processing CreateTablet for tablet 66a48a8ebe154b2cac89c234eaa378ff (DEFAULT_TABLE table=default.Table_ce16a1b94f714c2699ee6cd916bb0215 [id=551e2f9616d84026b96b6c22507f29de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:13.910940 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66a48a8ebe154b2cac89c234eaa378ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:13.924042 1306 ts_tablet_manager.cc:1918] T 6c15cd14831f422eb615038603c79ed0 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:13.924201 1306 log.cc:1199] T 6c15cd14831f422eb615038603c79ed0 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/6c15cd14831f422eb615038603c79ed0
W20250901 01:52:13.924443 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_18b93109e1224ea59b5c93825b3b27d6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:13.927592 452 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_cecc206bb66146c7ad35d7731bc02859" } new_table_name: "default.Table_81b532507ef2460d9ff52022c060c07a" modify_external_catalogs: true
I20250901 01:52:13.934008 904 tablet_service.cc:1468] Processing CreateTablet for tablet ef58f477ebeb49639ed7d071e1a62c10 (DEFAULT_TABLE table=default.Table_7f507bbabca54260a4a5d9092c274e41 [id=6020a2081326445090e69ef48cad0d9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:13.934567 904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef58f477ebeb49639ed7d071e1a62c10. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:13.935559 894 tablet_service.cc:1468] Processing CreateTablet for tablet c95798f0daf4437db18a291c4464522e (DEFAULT_TABLE table=default.Table_a89cc8e48563460baabc34c821ceae35 [id=9c3930e9fcfe4b49ba4bea67a2b0ad07]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:13.935932 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c95798f0daf4437db18a291c4464522e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:13.937053 1293 raft_consensus.cc:491] T 84fe314502444bd88b038b87eead9cb5 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:13.938504 1293 raft_consensus.cc:513] T 84fe314502444bd88b038b87eead9cb5 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:13.939200 1293 leader_election.cc:290] T 84fe314502444bd88b038b87eead9cb5 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:13.940083 1034 tablet_service.cc:1468] Processing CreateTablet for tablet ef58f477ebeb49639ed7d071e1a62c10 (DEFAULT_TABLE table=default.Table_7f507bbabca54260a4a5d9092c274e41 [id=6020a2081326445090e69ef48cad0d9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:13.940639 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef58f477ebeb49639ed7d071e1a62c10. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:13.940831 898 tablet_service.cc:1468] Processing CreateTablet for tablet 629db80a8ab444e582628578155638fd (DEFAULT_TABLE table=default.Table_3f2af1af32b546d6b1eba5cc3195acd8 [id=c3865e058daa4cea9a89f5046fca712a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:13.941324 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 629db80a8ab444e582628578155638fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:13.942862 1035 tablet_service.cc:1468] Processing CreateTablet for tablet 629db80a8ab444e582628578155638fd (DEFAULT_TABLE table=default.Table_3f2af1af32b546d6b1eba5cc3195acd8 [id=c3865e058daa4cea9a89f5046fca712a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:13.943326 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 629db80a8ab444e582628578155638fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:13.942919 1033 tablet_service.cc:1468] Processing CreateTablet for tablet c95798f0daf4437db18a291c4464522e (DEFAULT_TABLE table=default.Table_a89cc8e48563460baabc34c821ceae35 [id=9c3930e9fcfe4b49ba4bea67a2b0ad07]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:13.948925 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c95798f0daf4437db18a291c4464522e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:13.953022 1058 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84fe314502444bd88b038b87eead9cb5" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:13.954102 859 leader_election.cc:343] T 84fe314502444bd88b038b87eead9cb5 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:13.958863 1151 tablet_bootstrap.cc:492] T b29e846dfae44f0e9ce3e5fea3763a4f P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:13.959103 1151 ts_tablet_manager.cc:1397] T b29e846dfae44f0e9ce3e5fea3763a4f P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.261s user 0.031s sys 0.033s
I20250901 01:52:13.965341 1151 raft_consensus.cc:357] T b29e846dfae44f0e9ce3e5fea3763a4f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:13.965690 1151 raft_consensus.cc:383] T b29e846dfae44f0e9ce3e5fea3763a4f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:13.965853 1151 raft_consensus.cc:738] T b29e846dfae44f0e9ce3e5fea3763a4f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:13.966218 1151 consensus_queue.cc:260] T b29e846dfae44f0e9ce3e5fea3763a4f P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:13.970999 1151 ts_tablet_manager.cc:1428] T b29e846dfae44f0e9ce3e5fea3763a4f P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250901 01:52:13.975509 1151 tablet_bootstrap.cc:492] T d945b310fe6344af9cc9dd68f84c4bdf P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:13.977830 1306 ts_tablet_manager.cc:1939] T 6c15cd14831f422eb615038603c79ed0 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:13.978711 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 6c15cd14831f422eb615038603c79ed0 (table default.Table_5e65c5f068b84995a2d7a76fa65dda0f [id=b9d81ddd38fe47838877b010be2b40d0]) successfully deleted
I20250901 01:52:13.979219 1306 tablet_replica.cc:331] T 0e698923812841b7a5d37d57da103c5c P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:13.979548 1306 raft_consensus.cc:2241] T 0e698923812841b7a5d37d57da103c5c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:13.979812 1306 raft_consensus.cc:2270] T 0e698923812841b7a5d37d57da103c5c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:13.980590 1306 ts_tablet_manager.cc:1905] T 0e698923812841b7a5d37d57da103c5c P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:13.977876 1151 tablet_bootstrap.cc:654] T d945b310fe6344af9cc9dd68f84c4bdf P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:13.982324 451 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_8df8ec915ec84331933c6de7ed2983b3" } new_table_name: "default.Table_68f31d33b83544cea7b959bc7c41f94b" modify_external_catalogs: true
I20250901 01:52:13.984190 1293 raft_consensus.cc:491] T 756cfe84a517459ca015aecf5cd81571 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:13.984345 1293 raft_consensus.cc:513] T 756cfe84a517459ca015aecf5cd81571 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:13.985038 1293 leader_election.cc:290] T 756cfe84a517459ca015aecf5cd81571 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:13.986227 446 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet f0affc7a78634607852a48458412761d
I20250901 01:52:13.986162 1058 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "756cfe84a517459ca015aecf5cd81571" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:13.987112 446 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_ac4ff85b5f09468d870ec5fd9b9abd93" } new_table_name: "default.Table_c9401a0bf1b84645bdfd72bc9bcbe5b6" modify_external_catalogs: true
W20250901 01:52:13.989167 859 leader_election.cc:343] T 756cfe84a517459ca015aecf5cd81571 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:13.990705 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_6eca408a8cf342e1bafcbfbf927e214b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:13.991346 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6eca408a8cf342e1bafcbfbf927e214b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:13.992817 1293 raft_consensus.cc:491] T 85322a864d154f299f2d670fdd2a7bfb P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:13.993052 1293 raft_consensus.cc:513] T 85322a864d154f299f2d670fdd2a7bfb P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:13.993841 1293 leader_election.cc:290] T 85322a864d154f299f2d670fdd2a7bfb P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:13.996138 898 tablet_service.cc:1515] Processing DeleteTablet for tablet f0affc7a78634607852a48458412761d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:13 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:13.997900 1058 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "85322a864d154f299f2d670fdd2a7bfb" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:13.998255 1033 tablet_service.cc:1515] Processing DeleteTablet for tablet 4cdd369214ef4f5ebabddbf28b2ae3f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:08 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:13.999586 1035 tablet_service.cc:1515] Processing DeleteTablet for tablet f0affc7a78634607852a48458412761d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:13 UTC) from {username='slave'} at 127.0.0.1:44246
W20250901 01:52:14.000792 859 leader_election.cc:343] T 85322a864d154f299f2d670fdd2a7bfb P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:14.004485 1306 ts_tablet_manager.cc:1918] T 0e698923812841b7a5d37d57da103c5c P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:14.004663 1306 log.cc:1199] T 0e698923812841b7a5d37d57da103c5c P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/0e698923812841b7a5d37d57da103c5c
I20250901 01:52:14.005040 1306 ts_tablet_manager.cc:1939] T 0e698923812841b7a5d37d57da103c5c P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:14.005699 1332 ts_tablet_manager.cc:1918] T 4f8e3fcaa19740d397f4d118ac4cbdb0 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:14.005836 1306 tablet_replica.cc:331] T d945b310fe6344af9cc9dd68f84c4bdf P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:14.005836 1332 log.cc:1199] T 4f8e3fcaa19740d397f4d118ac4cbdb0 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/4f8e3fcaa19740d397f4d118ac4cbdb0
I20250901 01:52:14.005985 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 0e698923812841b7a5d37d57da103c5c (table default.Table_68682089b6364db98892ca5d82ed1322 [id=83600209eb4649ec9f7842507aa8a1ff]) successfully deleted
I20250901 01:52:14.006060 1306 raft_consensus.cc:2241] T d945b310fe6344af9cc9dd68f84c4bdf P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:14.006227 1306 raft_consensus.cc:2270] T d945b310fe6344af9cc9dd68f84c4bdf P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:14.006300 1332 ts_tablet_manager.cc:1939] T 4f8e3fcaa19740d397f4d118ac4cbdb0 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:14.006790 1306 ts_tablet_manager.cc:1905] T d945b310fe6344af9cc9dd68f84c4bdf P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:14.006820 1332 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44246 (request call id 148) took 1075 ms. Trace:
I20250901 01:52:14.006938 1332 rpcz_store.cc:276] 0901 01:52:12.931269 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:12.931388 (+ 119us) service_pool.cc:225] Handling call
0901 01:52:13.762348 (+830960us) ts_tablet_manager.cc:1145] Deleting tablet 4f8e3fcaa19740d397f4d118ac4cbdb0
0901 01:52:13.762359 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:13.779617 (+ 17258us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:14.005523 (+225906us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:14.006780 (+ 1257us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5177,"tablet-delete.queue_time_us":830797}
I20250901 01:52:14.007405 1332 tablet_replica.cc:331] T 8e6e96b4de214cadba1ccd19390cffb0 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:14.007529 441 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 4f8e3fcaa19740d397f4d118ac4cbdb0 (table default.Table_e9b3e507b30344d3a70b1f97bd72b6e0 [id=1d0bc46740444f1fbf2129270e806259]) successfully deleted
I20250901 01:52:14.007644 1332 raft_consensus.cc:2241] T 8e6e96b4de214cadba1ccd19390cffb0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:14.007812 1332 raft_consensus.cc:2270] T 8e6e96b4de214cadba1ccd19390cffb0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:14.010596 898 tablet_service.cc:1468] Processing CreateTablet for tablet 1286c0ab85704837a99b6e5a62184d77 (DEFAULT_TABLE table=default.Table_ac4ff85b5f09468d870ec5fd9b9abd93 [id=be7660807d4745d19346469b223bea60]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:14.011050 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1286c0ab85704837a99b6e5a62184d77. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:14.015666 894 tablet_service.cc:1468] Processing CreateTablet for tablet 23af154a6f1c4e2e9352fb17c9024f10 (DEFAULT_TABLE table=default.Table_54d5a8d2e22840d6a43c4eeb44478d1d [id=ccedb9a9dab94413b88eb7c203a9b2ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:14.016021 899 tablet_service.cc:1468] Processing CreateTablet for tablet 40d74d7b5a3b4d1dae09eebd9e07a0c4 (DEFAULT_TABLE table=default.Table_749fcd4bf97d439c9fd49b138a52a3fd [id=24fffc7ebf5a47bdb37211910301acf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:14.016151 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23af154a6f1c4e2e9352fb17c9024f10. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:14.016553 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40d74d7b5a3b4d1dae09eebd9e07a0c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:14.016767 904 tablet_service.cc:1468] Processing CreateTablet for tablet 3a8231590ea643b0a1f83eec4d9a8191 (DEFAULT_TABLE table=default.Table_647007ba008f42bcb27440e4d85a1d90 [id=fd6d8e92d6e64a89a7d38aa761376955]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:14.017165 904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a8231590ea643b0a1f83eec4d9a8191. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:14.018762 896 tablet_service.cc:1468] Processing CreateTablet for tablet b73e362321284df1b18ef755dfe930a5 (DEFAULT_TABLE table=default.Table_cecc206bb66146c7ad35d7731bc02859 [id=8ef63407ade74b5688d236eed2d1561c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:14.019172 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b73e362321284df1b18ef755dfe930a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:14.021255 1035 tablet_service.cc:1468] Processing CreateTablet for tablet 1286c0ab85704837a99b6e5a62184d77 (DEFAULT_TABLE table=default.Table_ac4ff85b5f09468d870ec5fd9b9abd93 [id=be7660807d4745d19346469b223bea60]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:14.021651 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1286c0ab85704837a99b6e5a62184d77. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:14.029349 1150 tablet_bootstrap.cc:492] T 1f18420261ae41459e1f25b927bd84a9 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:14.029531 1150 ts_tablet_manager.cc:1397] T 1f18420261ae41459e1f25b927bd84a9 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.245s user 0.030s sys 0.094s
I20250901 01:52:14.030496 1150 raft_consensus.cc:357] T 1f18420261ae41459e1f25b927bd84a9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:14.030717 1150 raft_consensus.cc:383] T 1f18420261ae41459e1f25b927bd84a9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:14.030812 1150 raft_consensus.cc:738] T 1f18420261ae41459e1f25b927bd84a9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:14.031090 1150 consensus_queue.cc:260] T 1f18420261ae41459e1f25b927bd84a9 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:14.031634 1150 ts_tablet_manager.cc:1428] T 1f18420261ae41459e1f25b927bd84a9 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250901 01:52:14.031961 1150 tablet_bootstrap.cc:492] T 4410da9e95454ecf95faf80221b0a71c P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:14.033746 1150 tablet_bootstrap.cc:654] T 4410da9e95454ecf95faf80221b0a71c P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:14.033752 1033 tablet_service.cc:1468] Processing CreateTablet for tablet 23af154a6f1c4e2e9352fb17c9024f10 (DEFAULT_TABLE table=default.Table_54d5a8d2e22840d6a43c4eeb44478d1d [id=ccedb9a9dab94413b88eb7c203a9b2ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:14.034220 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23af154a6f1c4e2e9352fb17c9024f10. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:14.035491 1031 tablet_service.cc:1468] Processing CreateTablet for tablet b73e362321284df1b18ef755dfe930a5 (DEFAULT_TABLE table=default.Table_cecc206bb66146c7ad35d7731bc02859 [id=8ef63407ade74b5688d236eed2d1561c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:14.035893 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b73e362321284df1b18ef755dfe930a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:14.036707 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 40d74d7b5a3b4d1dae09eebd9e07a0c4 (DEFAULT_TABLE table=default.Table_749fcd4bf97d439c9fd49b138a52a3fd [id=24fffc7ebf5a47bdb37211910301acf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:14.037099 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40d74d7b5a3b4d1dae09eebd9e07a0c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:14.042739 894 tablet_service.cc:1468] Processing CreateTablet for tablet 9473f0ef978c4bfaa41fb7b0729e8863 (DEFAULT_TABLE table=default.Table_706ad9744bcd4923b9a627c93f3cfde7 [id=a0fdcd4db982472c9e18aaa4d9fe50c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:14.043229 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9473f0ef978c4bfaa41fb7b0729e8863. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:14.044232 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 9473f0ef978c4bfaa41fb7b0729e8863 (DEFAULT_TABLE table=default.Table_706ad9744bcd4923b9a627c93f3cfde7 [id=a0fdcd4db982472c9e18aaa4d9fe50c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:14.044665 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9473f0ef978c4bfaa41fb7b0729e8863. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:14.048741 1036 tablet_service.cc:1468] Processing CreateTablet for tablet f62d96c37cca4143a4b098b7c69979e6 (DEFAULT_TABLE table=default.Table_0877ac8202a24e9da650a589bc6dbfcc [id=1f65b3e09c9748f189ecb2b56c5d1132]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:14.049136 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f62d96c37cca4143a4b098b7c69979e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:14.051172 1150 tablet_bootstrap.cc:492] T 4410da9e95454ecf95faf80221b0a71c P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:14.051321 1150 ts_tablet_manager.cc:1397] T 4410da9e95454ecf95faf80221b0a71c P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
I20250901 01:52:14.052299 1150 raft_consensus.cc:357] T 4410da9e95454ecf95faf80221b0a71c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:14.052574 1150 raft_consensus.cc:383] T 4410da9e95454ecf95faf80221b0a71c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:14.052697 1150 raft_consensus.cc:738] T 4410da9e95454ecf95faf80221b0a71c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:14.053031 1150 consensus_queue.cc:260] T 4410da9e95454ecf95faf80221b0a71c P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:14.053699 1150 ts_tablet_manager.cc:1428] T 4410da9e95454ecf95faf80221b0a71c P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250901 01:52:14.054015 1150 tablet_bootstrap.cc:492] T 7442c6ff699141e1aff208ebdc9d192a P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:14.054517 894 tablet_service.cc:1468] Processing CreateTablet for tablet f62d96c37cca4143a4b098b7c69979e6 (DEFAULT_TABLE table=default.Table_0877ac8202a24e9da650a589bc6dbfcc [id=1f65b3e09c9748f189ecb2b56c5d1132]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:14.054977 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f62d96c37cca4143a4b098b7c69979e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:14.054939 1306 ts_tablet_manager.cc:1918] T d945b310fe6344af9cc9dd68f84c4bdf P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:14.055145 1306 log.cc:1199] T d945b310fe6344af9cc9dd68f84c4bdf P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/d945b310fe6344af9cc9dd68f84c4bdf
I20250901 01:52:14.055997 1306 ts_tablet_manager.cc:1939] T d945b310fe6344af9cc9dd68f84c4bdf P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:14.033752 1034 tablet_service.cc:1468] Processing CreateTablet for tablet 3a8231590ea643b0a1f83eec4d9a8191 (DEFAULT_TABLE table=default.Table_647007ba008f42bcb27440e4d85a1d90 [id=fd6d8e92d6e64a89a7d38aa761376955]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:14.061201 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a8231590ea643b0a1f83eec4d9a8191. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:14.061889 1151 tablet_bootstrap.cc:492] T d945b310fe6344af9cc9dd68f84c4bdf P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:14.062069 1151 ts_tablet_manager.cc:1397] T d945b310fe6344af9cc9dd68f84c4bdf P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.087s user 0.004s sys 0.000s
I20250901 01:52:14.062105 1150 tablet_bootstrap.cc:654] T 7442c6ff699141e1aff208ebdc9d192a P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:14.062968 1151 raft_consensus.cc:357] T d945b310fe6344af9cc9dd68f84c4bdf P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:14.063220 1151 raft_consensus.cc:383] T d945b310fe6344af9cc9dd68f84c4bdf P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:14.063301 1151 raft_consensus.cc:738] T d945b310fe6344af9cc9dd68f84c4bdf P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:14.063611 1306 tablet_replica.cc:331] T b29e846dfae44f0e9ce3e5fea3763a4f P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:14.063859 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet d945b310fe6344af9cc9dd68f84c4bdf (table default.Table_e2ea7aa90231448888c6cfead7e3d9c1 [id=c264ed0bf1d84bc9bd5c2bb0d1e8a45b]) successfully deleted
I20250901 01:52:14.063937 1306 raft_consensus.cc:2241] T b29e846dfae44f0e9ce3e5fea3763a4f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:14.064141 1306 raft_consensus.cc:2270] T b29e846dfae44f0e9ce3e5fea3763a4f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:14.064280 1151 consensus_queue.cc:260] T d945b310fe6344af9cc9dd68f84c4bdf P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:14.064850 1306 ts_tablet_manager.cc:1905] T b29e846dfae44f0e9ce3e5fea3763a4f P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:14.064903 1151 ts_tablet_manager.cc:1428] T d945b310fe6344af9cc9dd68f84c4bdf P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250901 01:52:14.065217 1151 tablet_bootstrap.cc:492] T 2a0d125423a043dcb11e67345599e468 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
W20250901 01:52:14.066239 702 kernel_stack_watchdog.cc:198] Thread 757 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 3207ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fe78f073980 (unknown) at ??:0
@ 0x92ce64 __sanitizer::BlockingMutex::Lock() at ??:?
@ 0x932e1b __sanitizer::ThreadRegistry::CreateThread() at ??:?
@ 0x91983b __asan::AsanThread::Create() at ??:?
@ 0x8921ac __interceptor_pthread_create at ??:?
@ 0x7fe78bc459c1 kudu::Thread::StartThread() at ??:0
@ 0xa63f97 kudu::Thread::Create() at ??:?
@ 0x7fe78bc6580b kudu::ThreadPool::CreateThread() at ??:0
@ 0x7fe78bc6122d kudu::ThreadPool::DoSubmit() at ??:0
@ 0x7fe78bc6655f kudu::ThreadPool::Submit() at ??:0
@ 0x7fe79b49d226 kudu::tserver::TSTabletManager::DeleteTabletAsync() at ??:0
@ 0x7fe79b3e7ed0 kudu::tserver::TabletServiceAdminImpl::DeleteTablet() at ??:0
@ 0x7fe7955e6507 kudu::tserver::TabletServerAdminServiceIf::TabletServerAdminServiceIf()::$_3::operator()() at ??:0
@ 0x7fe7955e6236 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fe7929cf2cf std::function<>::operator()() at ??:0
@ 0x7fe7929ce1af kudu::rpc::GeneratedServiceIf::Handle() at ??:0
I20250901 01:52:14.067163 1151 tablet_bootstrap.cc:654] T 2a0d125423a043dcb11e67345599e468 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:14.068573 1332 ts_tablet_manager.cc:1905] T 8e6e96b4de214cadba1ccd19390cffb0 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:14.072865 1306 ts_tablet_manager.cc:1918] T b29e846dfae44f0e9ce3e5fea3763a4f P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:14.073045 1306 log.cc:1199] T b29e846dfae44f0e9ce3e5fea3763a4f P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/b29e846dfae44f0e9ce3e5fea3763a4f
I20250901 01:52:14.073441 1306 ts_tablet_manager.cc:1939] T b29e846dfae44f0e9ce3e5fea3763a4f P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:14.074002 896 tablet_service.cc:1468] Processing CreateTablet for tablet 3fda93767e494a4684978c8522b829dd (DEFAULT_TABLE table=default.Table_655d2fe6f5f044d2ac3c542497704343 [id=998e1151707c436bae86cdac62918ad1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:14.074451 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fda93767e494a4684978c8522b829dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:14.075794 1038 tablet_service.cc:1468] Processing CreateTablet for tablet 3fda93767e494a4684978c8522b829dd (DEFAULT_TABLE table=default.Table_655d2fe6f5f044d2ac3c542497704343 [id=998e1151707c436bae86cdac62918ad1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:14.076180 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fda93767e494a4684978c8522b829dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:14.077087 903 tablet_service.cc:1468] Processing CreateTablet for tablet 438cd35ff3f74c49b908d072560077db (DEFAULT_TABLE table=default.Table_bbdd9e12d55540eb885b874ef8358435 [id=5a33cfdf60684867bb5f36d84ee80468]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:14.077404 903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 438cd35ff3f74c49b908d072560077db. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:14.078083 897 tablet_service.cc:1468] Processing CreateTablet for tablet a6445b29591548be9f9577db1ba0bfa5 (DEFAULT_TABLE table=default.Table_409dad98d7eb455a8e16edfb4acedb34 [id=6f29cdc9974049368af2ae10beadb772]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:14.078435 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6445b29591548be9f9577db1ba0bfa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:14.079427 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 9d886fa63f8f4996b087997b542e8bb1 (DEFAULT_TABLE table=default.Table_874154cd2b0c4afcb0fa684f312984db [id=a33b9b981bd64d7981829d663b0b48b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:14.079771 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d886fa63f8f4996b087997b542e8bb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:14.082448 1293 raft_consensus.cc:491] T 88a6ab29d00542aba54f0e4185fbf615 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:14.082616 1293 raft_consensus.cc:513] T 88a6ab29d00542aba54f0e4185fbf615 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:14.083446 1293 leader_election.cc:290] T 88a6ab29d00542aba54f0e4185fbf615 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:14.085155 1037 tablet_service.cc:1468] Processing CreateTablet for tablet 438cd35ff3f74c49b908d072560077db (DEFAULT_TABLE table=default.Table_bbdd9e12d55540eb885b874ef8358435 [id=5a33cfdf60684867bb5f36d84ee80468]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:14.085219 1058 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "88a6ab29d00542aba54f0e4185fbf615" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:14.085615 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 438cd35ff3f74c49b908d072560077db. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:14.085829 1032 tablet_service.cc:1468] Processing CreateTablet for tablet a6445b29591548be9f9577db1ba0bfa5 (DEFAULT_TABLE table=default.Table_409dad98d7eb455a8e16edfb4acedb34 [id=6f29cdc9974049368af2ae10beadb772]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:14.085943 902 tablet_service.cc:1468] Processing CreateTablet for tablet 9d886fa63f8f4996b087997b542e8bb1 (DEFAULT_TABLE table=default.Table_874154cd2b0c4afcb0fa684f312984db [id=a33b9b981bd64d7981829d663b0b48b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:14.086189 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6445b29591548be9f9577db1ba0bfa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:14.086397 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d886fa63f8f4996b087997b542e8bb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:14.096056 447 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet d60d571b2c2840a2b20629bb832d1309
I20250901 01:52:14.096730 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet f50f33de1b654663b9ef8a91e3436f11
I20250901 01:52:14.099414 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_f94fe76831a54c6e8c2b13c81c251b14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:14.100016 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f94fe76831a54c6e8c2b13c81c251b14 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:14.101066 447 master_service.cc:948] ReplaceTablet: received request to replace tablet a5f4b0a2d72a4b8095ae4af1fe7c0c5d from {username='slave'} at 127.0.0.1:57810
W20250901 01:52:14.102593 859 leader_election.cc:343] T 88a6ab29d00542aba54f0e4185fbf615 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:14.102620 1150 tablet_bootstrap.cc:492] T 7442c6ff699141e1aff208ebdc9d192a P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:14.102877 1150 ts_tablet_manager.cc:1397] T 7442c6ff699141e1aff208ebdc9d192a P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.049s user 0.004s sys 0.000s
I20250901 01:52:14.103083 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet f50f33de1b654663b9ef8a91e3436f11 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:13 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:14.104548 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet d60d571b2c2840a2b20629bb832d1309 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:13 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:14.104342 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet b29e846dfae44f0e9ce3e5fea3763a4f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:09 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:14.104945 896 tablet_service.cc:1515] Processing DeleteTablet for tablet d60d571b2c2840a2b20629bb832d1309 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:13 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:14.105225 902 tablet_service.cc:1515] Processing DeleteTablet for tablet f50f33de1b654663b9ef8a91e3436f11 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:13 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:14.105855 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet b29e846dfae44f0e9ce3e5fea3763a4f (table default.Table_1e00e9f110c7483bb1a077829c64f85b [id=4ef5558babf64de7930fadf065ef6866]) successfully deleted
I20250901 01:52:14.106204 1306 tablet_replica.cc:331] T 360755d30b5d47ba98c9c7e23ff5ab7e P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:14.106447 1306 raft_consensus.cc:2241] T 360755d30b5d47ba98c9c7e23ff5ab7e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250901 01:52:14.106446 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 60285f2ba037433999f5455fb9cc7acd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 60285f2ba037433999f5455fb9cc7acd already in progress: creating tablet
I20250901 01:52:14.106621 1306 raft_consensus.cc:2270] T 360755d30b5d47ba98c9c7e23ff5ab7e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:14.107144 1306 ts_tablet_manager.cc:1905] T 360755d30b5d47ba98c9c7e23ff5ab7e P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:14.103680 1150 raft_consensus.cc:357] T 7442c6ff699141e1aff208ebdc9d192a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:14.108068 1150 raft_consensus.cc:383] T 7442c6ff699141e1aff208ebdc9d192a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:14.108161 1150 raft_consensus.cc:738] T 7442c6ff699141e1aff208ebdc9d192a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:14.108424 1150 consensus_queue.cc:260] T 7442c6ff699141e1aff208ebdc9d192a P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:14.108969 1150 ts_tablet_manager.cc:1428] T 7442c6ff699141e1aff208ebdc9d192a P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250901 01:52:14.109336 1150 tablet_bootstrap.cc:492] T 920925ee9b014e14a79ceb7796ff7828 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:14.111826 1150 tablet_bootstrap.cc:654] T 920925ee9b014e14a79ceb7796ff7828 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:14.114544 1306 ts_tablet_manager.cc:1918] T 360755d30b5d47ba98c9c7e23ff5ab7e P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:14.114682 1306 log.cc:1199] T 360755d30b5d47ba98c9c7e23ff5ab7e P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/360755d30b5d47ba98c9c7e23ff5ab7e
I20250901 01:52:14.115123 1306 ts_tablet_manager.cc:1939] T 360755d30b5d47ba98c9c7e23ff5ab7e P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:14.115859 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 360755d30b5d47ba98c9c7e23ff5ab7e (table default.Table_64b711e409d7434c8b176bf1a3ef8c86 [id=a762733ef34449a5a7a3898464cd8045]) successfully deleted
I20250901 01:52:14.115942 1306 tablet_replica.cc:331] T 84fe314502444bd88b038b87eead9cb5 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:14.116199 1306 raft_consensus.cc:2241] T 84fe314502444bd88b038b87eead9cb5 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:14.116423 1306 raft_consensus.cc:2270] T 84fe314502444bd88b038b87eead9cb5 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:14.117087 1306 ts_tablet_manager.cc:1905] T 84fe314502444bd88b038b87eead9cb5 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:14.120108 902 tablet_service.cc:1468] Processing CreateTablet for tablet 831e42de95854541a49ae6b3e039d3e9 (DEFAULT_TABLE table=default.Table_113664a018e84769a7ff61aa35a38618 [id=68214167814a4ecbab3ec100e83f4a64]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:14.120520 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 831e42de95854541a49ae6b3e039d3e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:14.120568 1038 tablet_service.cc:1468] Processing CreateTablet for tablet 831e42de95854541a49ae6b3e039d3e9 (DEFAULT_TABLE table=default.Table_113664a018e84769a7ff61aa35a38618 [id=68214167814a4ecbab3ec100e83f4a64]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:14.120846 1150 tablet_bootstrap.cc:492] T 920925ee9b014e14a79ceb7796ff7828 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:14.120981 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 831e42de95854541a49ae6b3e039d3e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:14.120992 1150 ts_tablet_manager.cc:1397] T 920925ee9b014e14a79ceb7796ff7828 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.004s
I20250901 01:52:14.122282 1150 raft_consensus.cc:357] T 920925ee9b014e14a79ceb7796ff7828 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:14.122471 1150 raft_consensus.cc:383] T 920925ee9b014e14a79ceb7796ff7828 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:14.122556 1150 raft_consensus.cc:738] T 920925ee9b014e14a79ceb7796ff7828 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:14.122815 1150 consensus_queue.cc:260] T 920925ee9b014e14a79ceb7796ff7828 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:14.123342 1150 ts_tablet_manager.cc:1428] T 920925ee9b014e14a79ceb7796ff7828 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250901 01:52:14.123597 1150 tablet_bootstrap.cc:492] T c27bb23c150f4565a1d0ba8f231cc386 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:14.125370 1028 tablet_service.cc:1468] Processing CreateTablet for tablet f06a77ecc841462396a8d5026c7327f4 (DEFAULT_TABLE table=default.Table_e2ea7aa90231448888c6cfead7e3d9c1 [id=c264ed0bf1d84bc9bd5c2bb0d1e8a45b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:14.125854 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f06a77ecc841462396a8d5026c7327f4. 1 dirs total, 0 dirs full, 0 dirs failed
W20250901 01:52:14.123653 442 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.61:42375, user_credentials={real_user=slave}} blocked reactor thread for 114485us
I20250901 01:52:14.129230 1026 tablet_service.cc:1468] Processing CreateTablet for tablet ddbc15685fca4c708329420109ad0c57 (DEFAULT_TABLE table=default.Table_942175dadf4b496195b2fa53f39f6ddc [id=2bd332b452f84f25af4225af8e6c6740]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:14.130506 1332 ts_tablet_manager.cc:1918] T 8e6e96b4de214cadba1ccd19390cffb0 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
#28 0xa774dc in std::function<void ()>::operator()() const ../../../include/c++/7.5.0/bits/std_function.h:706:14
I20250901 01:52:14.140211 897 tablet_service.cc:1468] Processing CreateTablet for tablet ddbc15685fca4c708329420109ad0c57 (DEFAULT_TABLE table=default.Table_942175dadf4b496195b2fa53f39f6ddc [id=2bd332b452f84f25af4225af8e6c6740]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:14.141222 896 tablet_service.cc:1468] Processing CreateTablet for tablet f06a77ecc841462396a8d5026c7327f4 (DEFAULT_TABLE table=default.Table_e2ea7aa90231448888c6cfead7e3d9c1 [id=c264ed0bf1d84bc9bd5c2bb0d1e8a45b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:14.142520 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddbc15685fca4c708329420109ad0c57. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:14.143797 894 tablet_service.cc:1468] Processing CreateTablet for tablet ded33e0950b544509af11739353c620d (DEFAULT_TABLE table=default.Table_71659da6afd8410fafb7a26532d52c9e [id=01c391f6c76943c9af685a74348eaf41]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:14.130772 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddbc15685fca4c708329420109ad0c57. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:14.144665 1332 log.cc:1199] T 8e6e96b4de214cadba1ccd19390cffb0 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/8e6e96b4de214cadba1ccd19390cffb0
I20250901 01:52:14.145063 1332 ts_tablet_manager.cc:1939] T 8e6e96b4de214cadba1ccd19390cffb0 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:14.148262 1332 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44246 (request call id 149) took 1216 ms. Trace:
I20250901 01:52:14.148393 1332 rpcz_store.cc:276] 0901 01:52:12.931449 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:12.931516 (+ 67us) service_pool.cc:225] Handling call
0901 01:52:14.007327 (+1075811us) ts_tablet_manager.cc:1145] Deleting tablet 8e6e96b4de214cadba1ccd19390cffb0
0901 01:52:14.007340 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:14.071778 (+ 64438us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:14.130369 (+ 58591us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:14.148193 (+ 17824us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1075594}
I20250901 01:52:14.144515 1151 tablet_bootstrap.cc:492] T 2a0d125423a043dcb11e67345599e468 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:14.151306 1151 ts_tablet_manager.cc:1397] T 2a0d125423a043dcb11e67345599e468 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.086s user 0.004s sys 0.000s
I20250901 01:52:14.152521 1151 raft_consensus.cc:357] T 2a0d125423a043dcb11e67345599e468 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:14.152791 1151 raft_consensus.cc:383] T 2a0d125423a043dcb11e67345599e468 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:14.152961 1151 raft_consensus.cc:738] T 2a0d125423a043dcb11e67345599e468 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:14.153257 1151 consensus_queue.cc:260] T 2a0d125423a043dcb11e67345599e468 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:14.154837 1151 ts_tablet_manager.cc:1428] T 2a0d125423a043dcb11e67345599e468 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250901 01:52:14.155172 1151 tablet_bootstrap.cc:492] T 4fd0e4f9d86c4362af96ce9964b72328 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:14.157022 1151 tablet_bootstrap.cc:654] T 4fd0e4f9d86c4362af96ce9964b72328 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:14.158244 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f06a77ecc841462396a8d5026c7327f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:14.143553 1035 tablet_service.cc:1468] Processing CreateTablet for tablet ded33e0950b544509af11739353c620d (DEFAULT_TABLE table=default.Table_71659da6afd8410fafb7a26532d52c9e [id=01c391f6c76943c9af685a74348eaf41]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:14.158787 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ded33e0950b544509af11739353c620d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:14.166235 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ded33e0950b544509af11739353c620d. 1 dirs total, 0 dirs full, 0 dirs failed
W20250901 01:52:14.168792 992 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250901 01:52:14.170825 1306 ts_tablet_manager.cc:1918] T 84fe314502444bd88b038b87eead9cb5 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:14.171566 441 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 8e6e96b4de214cadba1ccd19390cffb0 (table default.Table_da2c70a8f2fe49a0a8184e88c0a9e0ea [id=c1b1e9e3876244f68d31c63ce7bdebf1]) successfully deleted
I20250901 01:52:14.172740 1332 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44246 (request call id 150) took 1239 ms. Trace:
I20250901 01:52:14.173111 1306 log.cc:1199] T 84fe314502444bd88b038b87eead9cb5 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/84fe314502444bd88b038b87eead9cb5
I20250901 01:52:14.174047 1306 ts_tablet_manager.cc:1939] T 84fe314502444bd88b038b87eead9cb5 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:14.174751 1150 tablet_bootstrap.cc:654] T c27bb23c150f4565a1d0ba8f231cc386 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:14.174818 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 84fe314502444bd88b038b87eead9cb5 (table default.Table_6a5316d194864096bdfdf11e382f69d2 [id=c57384464f9f44f797e98c309b6866f5]) successfully deleted
I20250901 01:52:14.175062 1306 tablet_replica.cc:331] T 88a6ab29d00542aba54f0e4185fbf615 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:14.175274 1306 raft_consensus.cc:2241] T 88a6ab29d00542aba54f0e4185fbf615 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:14.172941 1332 rpcz_store.cc:276] 0901 01:52:12.931556 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:12.931594 (+ 38us) service_pool.cc:225] Handling call
0901 01:52:14.171228 (+1239634us) ts_tablet_manager.cc:1145] Deleting tablet bf302e76746b4f7bbe4469914f83720f
0901 01:52:14.171241 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:14.171325 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1239482}
I20250901 01:52:14.178238 1306 raft_consensus.cc:2270] T 88a6ab29d00542aba54f0e4185fbf615 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250901 01:52:14.178655 441 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet bf302e76746b4f7bbe4469914f83720f because the tablet was not found. No further retry: Not found: Tablet not found: bf302e76746b4f7bbe4469914f83720f
I20250901 01:52:14.178860 1306 ts_tablet_manager.cc:1905] T 88a6ab29d00542aba54f0e4185fbf615 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:14.182010 1332 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44246 (request call id 146) took 1247 ms. Trace:
I20250901 01:52:14.184669 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet ef58f477ebeb49639ed7d071e1a62c10
I20250901 01:52:14.185400 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet d6a4040747fa4d588756506ee122d2dc
I20250901 01:52:14.185865 454 catalog_manager.cc:6652] ReplaceTablet: tablet d6a4040747fa4d588756506ee122d2dc deleted and replaced by tablet d0c3c363dda444519a17dca63ad9caa7
I20250901 01:52:14.186110 454 master_service.cc:948] ReplaceTablet: received request to replace tablet a7ff996482164776b468cf305989866b from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:14.186694 455 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet f65ec84958eb488d8b0f812b59b5b3d5 (table default.Table_a3ff88e352884abdba91e6d03acbae22 [id=d1cd5288671540c0a95f7a037363014e]). Expected version 2 got 0
I20250901 01:52:14.187418 896 tablet_service.cc:1515] Processing DeleteTablet for tablet d6a4040747fa4d588756506ee122d2dc with delete_type TABLET_DATA_DELETED (replaced by tablet d0c3c363dda444519a17dca63ad9caa7) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:14.188966 453 master_service.cc:948] ReplaceTablet: received request to replace tablet b633b461dbf34e13bc6e1cc08e6d20c1 from {username='slave'} at 127.0.0.1:57810
W20250901 01:52:14.189518 452 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b73e362321284df1b18ef755dfe930a5 consensus state has no leader
I20250901 01:52:14.189590 897 tablet_service.cc:1515] Processing DeleteTablet for tablet ef58f477ebeb49639ed7d071e1a62c10 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:13 UTC) from {username='slave'} at 127.0.0.1:52846
W20250901 01:52:14.190323 451 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 85322a864d154f299f2d670fdd2a7bfb consensus state has no leader
W20250901 01:52:14.190941 446 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1286c0ab85704837a99b6e5a62184d77 consensus state has no leader
I20250901 01:52:14.191423 1306 ts_tablet_manager.cc:1918] T 88a6ab29d00542aba54f0e4185fbf615 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:14.191568 1306 log.cc:1199] T 88a6ab29d00542aba54f0e4185fbf615 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/88a6ab29d00542aba54f0e4185fbf615
I20250901 01:52:14.192046 1306 ts_tablet_manager.cc:1939] T 88a6ab29d00542aba54f0e4185fbf615 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:14.192060 1293 raft_consensus.cc:491] T f65ec84958eb488d8b0f812b59b5b3d5 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:14.192776 1293 raft_consensus.cc:513] T f65ec84958eb488d8b0f812b59b5b3d5 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:14.193620 1293 leader_election.cc:290] T f65ec84958eb488d8b0f812b59b5b3d5 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:14.199265 897 tablet_service.cc:1515] Processing DeleteTablet for tablet a5f4b0a2d72a4b8095ae4af1fe7c0c5d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:11 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:14.199671 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 58c0e7a2e90e48528bc2bc7ec889f608 with delete_type TABLET_DATA_DELETED (replaced by tablet a6445b29591548be9f9577db1ba0bfa5) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:14.206221 1306 tablet_replica.cc:331] T 31390f38258c465cb1798615a1fcc790 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:14.206252 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 88a6ab29d00542aba54f0e4185fbf615 (table default.Table_e9b3e507b30344d3a70b1f97bd72b6e0 [id=1d0bc46740444f1fbf2129270e806259]) successfully deleted
I20250901 01:52:14.206522 1306 raft_consensus.cc:2241] T 31390f38258c465cb1798615a1fcc790 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:14.207438 1306 raft_consensus.cc:2270] T 31390f38258c465cb1798615a1fcc790 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:14.208189 1306 ts_tablet_manager.cc:1905] T 31390f38258c465cb1798615a1fcc790 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:14.209982 1150 tablet_bootstrap.cc:492] T c27bb23c150f4565a1d0ba8f231cc386 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:14.210167 1150 ts_tablet_manager.cc:1397] T c27bb23c150f4565a1d0ba8f231cc386 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.087s user 0.003s sys 0.009s
I20250901 01:52:14.211087 1150 raft_consensus.cc:357] T c27bb23c150f4565a1d0ba8f231cc386 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:14.211367 1150 raft_consensus.cc:383] T c27bb23c150f4565a1d0ba8f231cc386 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:14.211457 1150 raft_consensus.cc:738] T c27bb23c150f4565a1d0ba8f231cc386 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:14.211807 1150 consensus_queue.cc:260] T c27bb23c150f4565a1d0ba8f231cc386 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:14.211951 1058 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f65ec84958eb488d8b0f812b59b5b3d5" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:14.212432 1150 ts_tablet_manager.cc:1428] T c27bb23c150f4565a1d0ba8f231cc386 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250901 01:52:14.213178 1150 tablet_bootstrap.cc:492] T 3b18a7084727490380d26b3485e4e799 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
W20250901 01:52:14.213523 859 leader_election.cc:343] T f65ec84958eb488d8b0f812b59b5b3d5 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:14.212244 1332 rpcz_store.cc:276] 0901 01:52:12.931046 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:12.931631 (+ 585us) service_pool.cc:225] Handling call
0901 01:52:14.178663 (+1247032us) ts_tablet_manager.cc:1145] Deleting tablet 3eaf8baac9f1485f944cd9e74b8b5569
0901 01:52:14.178677 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:14.178818 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1246780}
I20250901 01:52:14.215150 1332 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44246 (request call id 143) took 1284 ms. Trace:
I20250901 01:52:14.215247 1332 rpcz_store.cc:276] 0901 01:52:12.930627 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:12.931224 (+ 597us) service_pool.cc:225] Handling call
0901 01:52:14.215013 (+1283789us) ts_tablet_manager.cc:1145] Deleting tablet c445c300d71643609071f0f36528a9c4
0901 01:52:14.215022 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:14.215090 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1282976}
I20250901 01:52:14.215369 1151 tablet_bootstrap.cc:492] T 4fd0e4f9d86c4362af96ce9964b72328 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:14.215569 1332 tablet_replica.cc:331] T a4f5685260304d589dd467953d9bebfc P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:14.215778 1332 raft_consensus.cc:2241] T a4f5685260304d589dd467953d9bebfc P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:14.215797 1151 ts_tablet_manager.cc:1397] T 4fd0e4f9d86c4362af96ce9964b72328 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.061s user 0.004s sys 0.000s
I20250901 01:52:14.216102 1150 tablet_bootstrap.cc:654] T 3b18a7084727490380d26b3485e4e799 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:14.213655 1034 tablet_service.cc:1515] Processing DeleteTablet for tablet d6a4040747fa4d588756506ee122d2dc with delete_type TABLET_DATA_DELETED (replaced by tablet d0c3c363dda444519a17dca63ad9caa7) from {username='slave'} at 127.0.0.1:44246
W20250901 01:52:14.216276 441 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 3eaf8baac9f1485f944cd9e74b8b5569 because the tablet was not found. No further retry: Not found: Tablet not found: 3eaf8baac9f1485f944cd9e74b8b5569
W20250901 01:52:14.216912 441 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet c445c300d71643609071f0f36528a9c4 because the tablet was not found. No further retry: Not found: Tablet not found: c445c300d71643609071f0f36528a9c4
I20250901 01:52:14.217785 1151 raft_consensus.cc:357] T 4fd0e4f9d86c4362af96ce9964b72328 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:14.218979 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_4ce20bf6ad224378861e083f6a480edf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:14.219595 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4ce20bf6ad224378861e083f6a480edf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:14.222172 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_51c18d561b1342839bcbaf8fbacf5917"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:14.222813 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_51c18d561b1342839bcbaf8fbacf5917 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:14.225633 449 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_8022c1172d8a4df8bc1295d193bb6542" } modify_external_catalogs: true
I20250901 01:52:14.225864 449 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_8022c1172d8a4df8bc1295d193bb6542" } modify_external_catalogs: true
I20250901 01:52:14.226641 446 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_71659da6afd8410fafb7a26532d52c9e" } new_table_name: "default.Table_93afd37289414655980b22934695bd55" modify_external_catalogs: true
I20250901 01:52:14.227032 1150 tablet_bootstrap.cc:492] T 3b18a7084727490380d26b3485e4e799 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:14.227207 1150 ts_tablet_manager.cc:1397] T 3b18a7084727490380d26b3485e4e799 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.002s
I20250901 01:52:14.228122 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_aba07c28a3e248e4b102730106d6a2ba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:14.228189 1150 raft_consensus.cc:357] T 3b18a7084727490380d26b3485e4e799 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:14.228389 1150 raft_consensus.cc:383] T 3b18a7084727490380d26b3485e4e799 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250901 01:52:14.228797 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aba07c28a3e248e4b102730106d6a2ba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:14.228880 1150 raft_consensus.cc:738] T 3b18a7084727490380d26b3485e4e799 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:14.229306 1150 consensus_queue.cc:260] T 3b18a7084727490380d26b3485e4e799 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:14.230001 1150 ts_tablet_manager.cc:1428] T 3b18a7084727490380d26b3485e4e799 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250901 01:52:14.232160 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_c3570db2dfd14451aa98edf48f7b2a5e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:14.232651 451 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_0ad58ab77e93494e8936a824b95886d3" } new_table_name: "default.Table_dd8e6f9348c440bca1569cf68eec5beb" modify_external_catalogs: true
I20250901 01:52:14.234360 1306 ts_tablet_manager.cc:1918] T 31390f38258c465cb1798615a1fcc790 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:14.234514 1306 log.cc:1199] T 31390f38258c465cb1798615a1fcc790 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/31390f38258c465cb1798615a1fcc790
I20250901 01:52:14.234854 1306 ts_tablet_manager.cc:1939] T 31390f38258c465cb1798615a1fcc790 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:14.235874 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_36f5f8854ed14d1ea48eb18865d00659"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:14.237272 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 31390f38258c465cb1798615a1fcc790 (table default.Table_1a83c4b1e2734d038d0309401d0ab08e [id=c1902e6ac21b4a8a8af96d647f85e288]) successfully deleted
I20250901 01:52:14.237429 1150 tablet_bootstrap.cc:492] T ceb724d859d24d569598e18389dae3b3 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
W20250901 01:52:14.238536 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c3570db2dfd14451aa98edf48f7b2a5e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:14.240456 1034 tablet_service.cc:1515] Processing DeleteTablet for tablet 2a0d125423a043dcb11e67345599e468 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:09 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:14.241825 1150 tablet_bootstrap.cc:654] T ceb724d859d24d569598e18389dae3b3 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:14.241825 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_36f5f8854ed14d1ea48eb18865d00659 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:14.245311 1293 raft_consensus.cc:491] T 7cc1efcb6bff4aba8015d2d3b7f4c923 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:14.245502 1293 raft_consensus.cc:513] T 7cc1efcb6bff4aba8015d2d3b7f4c923 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:14.246747 1293 leader_election.cc:290] T 7cc1efcb6bff4aba8015d2d3b7f4c923 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:14.249071 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 533d14f5903e49599a34ea353ffd8c00 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 533d14f5903e49599a34ea353ffd8c00 already in progress: creating tablet
W20250901 01:52:14.249429 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet a7ff996482164776b468cf305989866b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a7ff996482164776b468cf305989866b already in progress: creating tablet
W20250901 01:52:14.249679 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet c9a4c80bec1d41458974a0c449aecaf7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9a4c80bec1d41458974a0c449aecaf7 already in progress: creating tablet
I20250901 01:52:14.249610 1058 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7cc1efcb6bff4aba8015d2d3b7f4c923" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:14.249794 1306 tablet_replica.cc:331] T f0affc7a78634607852a48458412761d P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:14.250052 1306 raft_consensus.cc:2241] T f0affc7a78634607852a48458412761d P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
W20250901 01:52:14.250236 859 leader_election.cc:343] T 7cc1efcb6bff4aba8015d2d3b7f4c923 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:14.250294 1306 raft_consensus.cc:2270] T f0affc7a78634607852a48458412761d P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:14.250995 1306 ts_tablet_manager.cc:1905] T f0affc7a78634607852a48458412761d P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:14.252056 1151 raft_consensus.cc:383] T 4fd0e4f9d86c4362af96ce9964b72328 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:14.252173 1151 raft_consensus.cc:738] T 4fd0e4f9d86c4362af96ce9964b72328 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:14.252460 1151 consensus_queue.cc:260] T 4fd0e4f9d86c4362af96ce9964b72328 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:14.253152 1151 ts_tablet_manager.cc:1428] T 4fd0e4f9d86c4362af96ce9964b72328 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.037s user 0.002s sys 0.000s
I20250901 01:52:14.253481 1151 tablet_bootstrap.cc:492] T 467c2bf4a8cb4937b6f627bea28d8ddb P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:14.254510 1150 tablet_bootstrap.cc:492] T ceb724d859d24d569598e18389dae3b3 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:14.254681 1150 ts_tablet_manager.cc:1397] T ceb724d859d24d569598e18389dae3b3 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.003s
I20250901 01:52:14.255462 1151 tablet_bootstrap.cc:654] T 467c2bf4a8cb4937b6f627bea28d8ddb P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:14.255512 1150 raft_consensus.cc:357] T ceb724d859d24d569598e18389dae3b3 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:14.255713 1150 raft_consensus.cc:383] T ceb724d859d24d569598e18389dae3b3 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:14.255805 1150 raft_consensus.cc:738] T ceb724d859d24d569598e18389dae3b3 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:14.256055 1150 consensus_queue.cc:260] T ceb724d859d24d569598e18389dae3b3 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:14.256592 1150 ts_tablet_manager.cc:1428] T ceb724d859d24d569598e18389dae3b3 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250901 01:52:14.256915 1517 raft_consensus.cc:491] T a5f4b0a2d72a4b8095ae4af1fe7c0c5d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:14.257308 1150 tablet_bootstrap.cc:492] T 2f28451a3ecc481a958588466f7fec77 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:14.257143 1517 raft_consensus.cc:513] T a5f4b0a2d72a4b8095ae4af1fe7c0c5d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:14.258313 1517 leader_election.cc:290] T a5f4b0a2d72a4b8095ae4af1fe7c0c5d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:14.260375 1150 tablet_bootstrap.cc:654] T 2f28451a3ecc481a958588466f7fec77 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:14.260713 1058 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a5f4b0a2d72a4b8095ae4af1fe7c0c5d" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:14.262359 859 leader_election.cc:343] T a5f4b0a2d72a4b8095ae4af1fe7c0c5d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:14.215978 1332 raft_consensus.cc:2270] T a4f5685260304d589dd467953d9bebfc P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:14.267818 1332 ts_tablet_manager.cc:1905] T a4f5685260304d589dd467953d9bebfc P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:14.268708 1374 raft_consensus.cc:491] T 4d333b670df84f308f0cbf6da636b768 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:14.268857 1374 raft_consensus.cc:513] T 4d333b670df84f308f0cbf6da636b768 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } }
I20250901 01:52:14.269587 1374 leader_election.cc:290] T 4d333b670df84f308f0cbf6da636b768 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561), 0cf599f74322478d91475df234e58965 (127.0.104.1:40731)
W20250901 01:52:14.271030 992 leader_election.cc:343] T 4d333b670df84f308f0cbf6da636b768 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 4d333b670df84f308f0cbf6da636b768
W20250901 01:52:14.275107 702 kernel_stack_watchdog.cc:198] Thread 716 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 4520ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fe78f073980 (unknown) at ??:0
@ 0x92ce64 __sanitizer::BlockingMutex::Lock() at ??:?
@ 0x932e1b __sanitizer::ThreadRegistry::CreateThread() at ??:?
@ 0x91983b __asan::AsanThread::Create() at ??:?
@ 0x8921ac __interceptor_pthread_create at ??:?
@ 0x7fe78bc459c1 kudu::Thread::StartThread() at ??:0
@ 0xa63f97 kudu::Thread::Create() at ??:?
@ 0x7fe78bc6580b kudu::ThreadPool::CreateThread() at ??:0
@ 0x7fe78bc6122d kudu::ThreadPool::DoSubmit() at ??:0
@ 0x7fe78bc6655f kudu::ThreadPool::Submit() at ??:0
@ 0x7fe792947bb2 kudu::rpc::ReactorThread::StartConnectionNegotiation() at ??:0
@ 0x7fe7929492d1 kudu::rpc::ReactorThread::FindOrStartConnection() at ??:0
@ 0x7fe792948ad2 kudu::rpc::ReactorThread::AssignOutboundCall() at ??:0
@ 0x7fe79295842e kudu::rpc::AssignOutboundCallTask::Run() at ??:0
@ 0x7fe792946bcd kudu::rpc::ReactorThread::AsyncHandler() at ??:0
@ 0x7fe792958f45 ev::base<>::method_thunk<>() at ??:0
I20250901 01:52:14.276608 1306 ts_tablet_manager.cc:1918] T f0affc7a78634607852a48458412761d P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:14.276751 1306 log.cc:1199] T f0affc7a78634607852a48458412761d P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/f0affc7a78634607852a48458412761d
I20250901 01:52:14.277148 1306 ts_tablet_manager.cc:1939] T f0affc7a78634607852a48458412761d P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:14.278172 1306 tablet_replica.cc:331] T d6a4040747fa4d588756506ee122d2dc P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:14.278430 1306 raft_consensus.cc:2241] T d6a4040747fa4d588756506ee122d2dc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:14.279294 455 master_service.cc:948] ReplaceTablet: received request to replace tablet 82d163681fda47f38d79863bd28f576d from {username='slave'} at 127.0.0.1:57810
W20250901 01:52:14.279773 451 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fcb7b5b7d63d4f0b906fea97cb548de2 consensus state has no leader
W20250901 01:52:14.280371 446 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ded33e0950b544509af11739353c620d consensus state has no leader
I20250901 01:52:14.284735 1035 tablet_service.cc:1515] Processing DeleteTablet for tablet ef58f477ebeb49639ed7d071e1a62c10 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:13 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:14.286294 1306 raft_consensus.cc:2270] T d6a4040747fa4d588756506ee122d2dc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:14.287144 1306 ts_tablet_manager.cc:1905] T d6a4040747fa4d588756506ee122d2dc P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:14.292291 450 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_117c6c43b552476dbb8b7b5ccdce47b7" } modify_external_catalogs: true
I20250901 01:52:14.292562 450 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_117c6c43b552476dbb8b7b5ccdce47b7" } modify_external_catalogs: true
I20250901 01:52:14.293470 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_271fd01373bf49f2ab8b4faf82db7393"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:14.294673 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_b1ae493cd53a40e2b5d26a0bfb3a9ef5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:14.299059 1150 tablet_bootstrap.cc:492] T 2f28451a3ecc481a958588466f7fec77 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:14.299145 452 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_eacb38fbc98046dfac27344cfbed0a7c" } modify_external_catalogs: true
I20250901 01:52:14.299379 1150 ts_tablet_manager.cc:1397] T 2f28451a3ecc481a958588466f7fec77 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.043s user 0.009s sys 0.002s
I20250901 01:52:14.299388 452 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_eacb38fbc98046dfac27344cfbed0a7c" } modify_external_catalogs: true
W20250901 01:52:14.300299 439 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 1286c0ab85704837a99b6e5a62184d77 (table default.Table_c9401a0bf1b84645bdfd72bc9bcbe5b6 [id=be7660807d4745d19346469b223bea60], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 1286c0ab85704837a99b6e5a62184d77 consensus state has no leader
I20250901 01:52:14.300483 1150 raft_consensus.cc:357] T 2f28451a3ecc481a958588466f7fec77 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:14.300717 1150 raft_consensus.cc:383] T 2f28451a3ecc481a958588466f7fec77 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:14.300819 1150 raft_consensus.cc:738] T 2f28451a3ecc481a958588466f7fec77 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:14.301108 1150 consensus_queue.cc:260] T 2f28451a3ecc481a958588466f7fec77 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:14.301967 1150 ts_tablet_manager.cc:1428] T 2f28451a3ecc481a958588466f7fec77 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250901 01:52:14.302241 1150 tablet_bootstrap.cc:492] T 8381305e4c4148808803e255b8a43a90 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:14.302212 454 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_79a6a0f0c6944313b9987f23b6a09557" } modify_external_catalogs: true
I20250901 01:52:14.302397 454 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_79a6a0f0c6944313b9987f23b6a09557" } modify_external_catalogs: true
I20250901 01:52:14.303987 1150 tablet_bootstrap.cc:654] T 8381305e4c4148808803e255b8a43a90 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:14.317317 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_271fd01373bf49f2ab8b4faf82db7393 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:14.318507 1306 ts_tablet_manager.cc:1918] T d6a4040747fa4d588756506ee122d2dc P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:14.318725 1306 log.cc:1199] T d6a4040747fa4d588756506ee122d2dc P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/d6a4040747fa4d588756506ee122d2dc
I20250901 01:52:14.319181 1306 ts_tablet_manager.cc:1939] T d6a4040747fa4d588756506ee122d2dc P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:14.278337 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet f0affc7a78634607852a48458412761d (table default.Table_b7bffcbb78614d3cad2b1d375d7a97cc [id=535cf5f83c56443d93935ce80ff87efb]) successfully deleted
W20250901 01:52:14.319799 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet d60d571b2c2840a2b20629bb832d1309 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d60d571b2c2840a2b20629bb832d1309 already in progress: creating tablet
W20250901 01:52:14.320235 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet f50f33de1b654663b9ef8a91e3436f11 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f50f33de1b654663b9ef8a91e3436f11 already in progress: creating tablet
I20250901 01:52:14.320622 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet d6a4040747fa4d588756506ee122d2dc (table default.Table_6400e401888045348632f924f636d011 [id=8c1b263824ec4c15a5ec424364c78895]) successfully deleted
W20250901 01:52:14.321115 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b1ae493cd53a40e2b5d26a0bfb3a9ef5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:14.321919 1150 tablet_bootstrap.cc:492] T 8381305e4c4148808803e255b8a43a90 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:14.322142 1150 ts_tablet_manager.cc:1397] T 8381305e4c4148808803e255b8a43a90 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.005s
I20250901 01:52:14.323143 1150 raft_consensus.cc:357] T 8381305e4c4148808803e255b8a43a90 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:14.323381 1150 raft_consensus.cc:383] T 8381305e4c4148808803e255b8a43a90 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:14.323475 1150 raft_consensus.cc:738] T 8381305e4c4148808803e255b8a43a90 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:14.323760 1150 consensus_queue.cc:260] T 8381305e4c4148808803e255b8a43a90 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:14.324328 1150 ts_tablet_manager.cc:1428] T 8381305e4c4148808803e255b8a43a90 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250901 01:52:14.324786 1150 tablet_bootstrap.cc:492] T 3df908c3096a4df7bddd8ac88b49cfe0 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:14.327557 1150 tablet_bootstrap.cc:654] T 3df908c3096a4df7bddd8ac88b49cfe0 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:14.329118 1306 tablet_replica.cc:331] T a5f4b0a2d72a4b8095ae4af1fe7c0c5d P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
W20250901 01:52:14.329094 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet ef58f477ebeb49639ed7d071e1a62c10 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ef58f477ebeb49639ed7d071e1a62c10 already in progress: creating tablet
I20250901 01:52:14.329371 1306 raft_consensus.cc:2241] T a5f4b0a2d72a4b8095ae4af1fe7c0c5d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:14.329627 1306 raft_consensus.cc:2270] T a5f4b0a2d72a4b8095ae4af1fe7c0c5d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250901 01:52:14.330050 992 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250901 01:52:14.330296 1306 ts_tablet_manager.cc:1905] T a5f4b0a2d72a4b8095ae4af1fe7c0c5d P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:14.333163 1151 tablet_bootstrap.cc:492] T 467c2bf4a8cb4937b6f627bea28d8ddb P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:14.339649 1151 ts_tablet_manager.cc:1397] T 467c2bf4a8cb4937b6f627bea28d8ddb P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.086s user 0.003s sys 0.000s
I20250901 01:52:14.342067 1151 raft_consensus.cc:357] T 467c2bf4a8cb4937b6f627bea28d8ddb P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:14.346838 1151 raft_consensus.cc:383] T 467c2bf4a8cb4937b6f627bea28d8ddb P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:14.348397 1151 raft_consensus.cc:738] T 467c2bf4a8cb4937b6f627bea28d8ddb P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:14.351441 1151 consensus_queue.cc:260] T 467c2bf4a8cb4937b6f627bea28d8ddb P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:14.351701 1306 ts_tablet_manager.cc:1918] T a5f4b0a2d72a4b8095ae4af1fe7c0c5d P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:14.351876 1306 log.cc:1199] T a5f4b0a2d72a4b8095ae4af1fe7c0c5d P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/a5f4b0a2d72a4b8095ae4af1fe7c0c5d
I20250901 01:52:14.352254 1306 ts_tablet_manager.cc:1939] T a5f4b0a2d72a4b8095ae4af1fe7c0c5d P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:14.354933 1150 tablet_bootstrap.cc:492] T 3df908c3096a4df7bddd8ac88b49cfe0 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:14.355154 1150 ts_tablet_manager.cc:1397] T 3df908c3096a4df7bddd8ac88b49cfe0 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.002s
I20250901 01:52:14.356225 1150 raft_consensus.cc:357] T 3df908c3096a4df7bddd8ac88b49cfe0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:14.356666 1150 raft_consensus.cc:383] T 3df908c3096a4df7bddd8ac88b49cfe0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:14.356817 1150 raft_consensus.cc:738] T 3df908c3096a4df7bddd8ac88b49cfe0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:14.357205 1150 consensus_queue.cc:260] T 3df908c3096a4df7bddd8ac88b49cfe0 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:14.357810 1150 ts_tablet_manager.cc:1428] T 3df908c3096a4df7bddd8ac88b49cfe0 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250901 01:52:14.358075 1150 tablet_bootstrap.cc:492] T d60d571b2c2840a2b20629bb832d1309 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:14.360095 1150 tablet_bootstrap.cc:654] T d60d571b2c2840a2b20629bb832d1309 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:14.361893 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet a5f4b0a2d72a4b8095ae4af1fe7c0c5d (table default.Table_457273ab524846e6954d3e60f698e178 [id=a2546d228d4d42bb9829c56443a4c867]) successfully deleted
W20250901 01:52:14.362673 440 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 58c0e7a2e90e48528bc2bc7ec889f608 because the tablet was not found. No further retry: Not found: Tablet not found: 58c0e7a2e90e48528bc2bc7ec889f608
I20250901 01:52:14.362877 1332 ts_tablet_manager.cc:1918] T a4f5685260304d589dd467953d9bebfc P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:14.363037 1332 log.cc:1199] T a4f5685260304d589dd467953d9bebfc P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/a4f5685260304d589dd467953d9bebfc
I20250901 01:52:14.363459 1332 ts_tablet_manager.cc:1939] T a4f5685260304d589dd467953d9bebfc P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:14.363754 1517 raft_consensus.cc:491] T 920925ee9b014e14a79ceb7796ff7828 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:14.363966 1332 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44246 (request call id 161) took 1266 ms. Trace:
I20250901 01:52:14.363935 1517 raft_consensus.cc:513] T 920925ee9b014e14a79ceb7796ff7828 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:14.364086 1332 rpcz_store.cc:276] 0901 01:52:13.097782 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:13.097864 (+ 82us) service_pool.cc:225] Handling call
0901 01:52:14.215479 (+1117615us) ts_tablet_manager.cc:1145] Deleting tablet a4f5685260304d589dd467953d9bebfc
0901 01:52:14.215490 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:14.331254 (+115764us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:14.362713 (+ 31459us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:14.363930 (+ 1217us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1117094}
I20250901 01:52:14.364655 1517 leader_election.cc:290] T 920925ee9b014e14a79ceb7796ff7828 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:14.367199 1058 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "920925ee9b014e14a79ceb7796ff7828" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:14.369181 1150 tablet_bootstrap.cc:492] T d60d571b2c2840a2b20629bb832d1309 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:14.369364 1150 ts_tablet_manager.cc:1397] T d60d571b2c2840a2b20629bb832d1309 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
W20250901 01:52:14.369891 859 leader_election.cc:343] T 920925ee9b014e14a79ceb7796ff7828 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:14.366426 1151 ts_tablet_manager.cc:1428] T 467c2bf4a8cb4937b6f627bea28d8ddb P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.025s user 0.002s sys 0.000s
I20250901 01:52:14.373792 1150 raft_consensus.cc:357] T d60d571b2c2840a2b20629bb832d1309 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:14.374028 1150 raft_consensus.cc:383] T d60d571b2c2840a2b20629bb832d1309 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:14.374107 1150 raft_consensus.cc:738] T d60d571b2c2840a2b20629bb832d1309 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:14.374421 1150 consensus_queue.cc:260] T d60d571b2c2840a2b20629bb832d1309 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:14.375015 1150 ts_tablet_manager.cc:1428] T d60d571b2c2840a2b20629bb832d1309 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250901 01:52:14.375298 1150 tablet_bootstrap.cc:492] T 186ec810d02e4748aad15228b5866ac1 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:14.377290 1150 tablet_bootstrap.cc:654] T 186ec810d02e4748aad15228b5866ac1 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:14.378029 1332 tablet_replica.cc:331] T 447ffe73741640ceb73f35676232a094 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:14.388168 447 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 4fd0e4f9d86c4362af96ce9964b72328 (table default.Table_d4d399ae87234df7a710191d0ecf02d0 [id=f48fd53522234290b6c0373fd7b02570]). Expected version 1 got 0
I20250901 01:52:14.391855 1150 tablet_bootstrap.cc:492] T 186ec810d02e4748aad15228b5866ac1 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:14.392014 1150 ts_tablet_manager.cc:1397] T 186ec810d02e4748aad15228b5866ac1 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.004s
I20250901 01:52:14.392555 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 121803e79ab64ec0b948b448646b9d0a
I20250901 01:52:14.392870 1150 raft_consensus.cc:357] T 186ec810d02e4748aad15228b5866ac1 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:14.393512 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 121803e79ab64ec0b948b448646b9d0a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:14 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:14.393592 1150 raft_consensus.cc:383] T 186ec810d02e4748aad15228b5866ac1 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250901 01:52:14.394124 440 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 121803e79ab64ec0b948b448646b9d0a because the tablet was not found. No further retry: Not found: Tablet not found: 121803e79ab64ec0b948b448646b9d0a
I20250901 01:52:14.394492 1150 raft_consensus.cc:738] T 186ec810d02e4748aad15228b5866ac1 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:14.394765 1150 consensus_queue.cc:260] T 186ec810d02e4748aad15228b5866ac1 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:14.395421 1150 ts_tablet_manager.cc:1428] T 186ec810d02e4748aad15228b5866ac1 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250901 01:52:14.400722 451 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_706ad9744bcd4923b9a627c93f3cfde7" } new_table_name: "default.Table_ae4267b6728a4e60a333464621e3dc8f" modify_external_catalogs: true
I20250901 01:52:14.402282 446 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_51c18d561b1342839bcbaf8fbacf5917" } new_table_name: "default.Table_34a777af60f64f5ba5a17c0875d03f38" modify_external_catalogs: true
I20250901 01:52:14.403959 447 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_063de0c73851415196ba2e77742edee4" } modify_external_catalogs: true
I20250901 01:52:14.404166 447 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_063de0c73851415196ba2e77742edee4" } modify_external_catalogs: true
I20250901 01:52:14.395826 1150 tablet_bootstrap.cc:492] T fcb7b5b7d63d4f0b906fea97cb548de2 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:14.405436 450 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_b1ae493cd53a40e2b5d26a0bfb3a9ef5" } new_table_name: "default.Table_ee242c34410e4c6887405d04a4b26124" modify_external_catalogs: true
I20250901 01:52:14.406905 1150 tablet_bootstrap.cc:654] T fcb7b5b7d63d4f0b906fea97cb548de2 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:14.420200 446 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet eac50cbae691416e903cb2cbc907f96d has no consensus state
W20250901 01:52:14.420878 451 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9473f0ef978c4bfaa41fb7b0729e8863 consensus state has no leader
I20250901 01:52:14.421458 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_e098ab0a5d4045cfb18b7182ffda1b99"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:14.422271 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e098ab0a5d4045cfb18b7182ffda1b99 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250901 01:52:14.424261 702 kernel_stack_watchdog.cc:198] Thread 715 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 1385ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fe78f073980 (unknown) at ??:0
@ 0x92ce64 __sanitizer::BlockingMutex::Lock() at ??:?
@ 0x932e1b __sanitizer::ThreadRegistry::CreateThread() at ??:?
@ 0x91983b __asan::AsanThread::Create() at ??:?
@ 0x8921ac __interceptor_pthread_create at ??:?
@ 0x7fe78bc459c1 kudu::Thread::StartThread() at ??:0
@ 0xa63f97 kudu::Thread::Create() at ??:?
@ 0x7fe78bc6580b kudu::ThreadPool::CreateThread() at ??:0
@ 0x7fe78bc6122d kudu::ThreadPool::DoSubmit() at ??:0
@ 0x7fe78bc6067f kudu::ThreadPoolToken::Submit() at ??:0
@ 0x7fe79767b5ea kudu::consensus::RaftConsensus::ReportFailureDetected() at ??:0
@ 0x7fe7976b6d6b kudu::consensus::RaftConsensus::Start()::$_1::operator()() at ??:0
@ 0x7fe7976b6a7e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa774dd std::function<>::operator()() at ??:?
@ 0x7fe792933239 kudu::rpc::PeriodicTimer::Callback() at ??:0
@ 0x7fe79293433f kudu::rpc::PeriodicTimer::Callback()::$_0::operator()() at ??:0
I20250901 01:52:14.426301 446 master_service.cc:948] ReplaceTablet: received request to replace tablet 66a48a8ebe154b2cac89c234eaa378ff from {username='slave'} at 127.0.0.1:57810
W20250901 01:52:14.434507 450 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8dd587304e6342edbe96c9897b40e5c9 has no consensus state
I20250901 01:52:14.434881 450 master_service.cc:948] ReplaceTablet: received request to replace tablet 5a8c21aee9f84216b6f90538a5bf4c17 from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:14.437304 1150 tablet_bootstrap.cc:492] T fcb7b5b7d63d4f0b906fea97cb548de2 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:14.437543 1150 ts_tablet_manager.cc:1397] T fcb7b5b7d63d4f0b906fea97cb548de2 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.042s user 0.004s sys 0.003s
I20250901 01:52:14.444877 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_0fc5eac5c02943af9b6801ac9cee55a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:14.445670 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0fc5eac5c02943af9b6801ac9cee55a3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:14.460948 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_9d8c1fb24bc24631a2ba2b34a395f24a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:14.461781 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9d8c1fb24bc24631a2ba2b34a395f24a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:14.485113 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 5a8c21aee9f84216b6f90538a5bf4c17
I20250901 01:52:14.486191 450 catalog_manager.cc:6652] ReplaceTablet: tablet 5a8c21aee9f84216b6f90538a5bf4c17 deleted and replaced by tablet 42c1156e8dd84c6fa1396b46de13eb6f
I20250901 01:52:14.486626 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 5a8c21aee9f84216b6f90538a5bf4c17 with delete_type TABLET_DATA_DELETED (replaced by tablet 42c1156e8dd84c6fa1396b46de13eb6f) from {username='slave'} at 127.0.0.1:52846
W20250901 01:52:14.487314 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 5a8c21aee9f84216b6f90538a5bf4c17 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5a8c21aee9f84216b6f90538a5bf4c17 already in progress: creating tablet
I20250901 01:52:14.492834 448 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 920925ee9b014e14a79ceb7796ff7828 (table default.Table_537be4a82c0348f6952b911b167d02ad [id=8199118617cb4707bc3ae9b1363750f7]). Expected version 1 got 0
I20250901 01:52:14.493220 448 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 4410da9e95454ecf95faf80221b0a71c (table default.Table_420b0979d6fd42da9b320456f4ccc9ce [id=d2004563c4374790954327f05c305cc6]). Expected version 1 got 0
I20250901 01:52:14.494879 448 master_service.cc:948] ReplaceTablet: received request to replace tablet 2685fb7258264463b5f1ec99cd7d67ad from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:14.495050 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 1f18420261ae41459e1f25b927bd84a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:13 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:14.495371 1306 tablet_replica.cc:331] T 1f18420261ae41459e1f25b927bd84a9 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:14.495604 1306 raft_consensus.cc:2241] T 1f18420261ae41459e1f25b927bd84a9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:14.512198 446 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 66a48a8ebe154b2cac89c234eaa378ff
I20250901 01:52:14.513212 446 catalog_manager.cc:6652] ReplaceTablet: tablet 66a48a8ebe154b2cac89c234eaa378ff deleted and replaced by tablet 15e4d972e8254cf899c3e0b71b6e2bd9
I20250901 01:52:14.513499 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 66a48a8ebe154b2cac89c234eaa378ff with delete_type TABLET_DATA_DELETED (replaced by tablet 15e4d972e8254cf899c3e0b71b6e2bd9) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:14.513603 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_b0e916c76a7a4766aa0887cc8691488f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:14.514271 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b0e916c76a7a4766aa0887cc8691488f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:14.521966 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 82d163681fda47f38d79863bd28f576d
I20250901 01:52:14.522576 455 catalog_manager.cc:6652] ReplaceTablet: tablet 82d163681fda47f38d79863bd28f576d deleted and replaced by tablet bf8f0237233a49219cd1c41cf808bdba
I20250901 01:52:14.522975 455 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_ac8bd7697e9c4195b4a16a6a44e9594b" } new_table_name: "default.Table_2f09538b6fbe401f909105decb56e1cb" modify_external_catalogs: true
I20250901 01:52:14.525149 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 82d163681fda47f38d79863bd28f576d with delete_type TABLET_DATA_DELETED (replaced by tablet bf8f0237233a49219cd1c41cf808bdba) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:14.384497 1151 tablet_bootstrap.cc:492] T 2685fb7258264463b5f1ec99cd7d67ad P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:14.531836 1151 tablet_bootstrap.cc:654] T 2685fb7258264463b5f1ec99cd7d67ad P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:14.393800 1038 tablet_service.cc:1515] Processing DeleteTablet for tablet 121803e79ab64ec0b948b448646b9d0a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:14 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:14.532518 653 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 121803e79ab64ec0b948b448646b9d0a
I20250901 01:52:14.533586 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 121803e79ab64ec0b948b448646b9d0a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:14 UTC) from {username='slave'} at 127.0.0.1:52846
W20250901 01:52:14.425062 994 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250901 01:52:14.486425 1035 tablet_service.cc:1515] Processing DeleteTablet for tablet 5a8c21aee9f84216b6f90538a5bf4c17 with delete_type TABLET_DATA_DELETED (replaced by tablet 42c1156e8dd84c6fa1396b46de13eb6f) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:14.513530 1034 tablet_service.cc:1515] Processing DeleteTablet for tablet 66a48a8ebe154b2cac89c234eaa378ff with delete_type TABLET_DATA_DELETED (replaced by tablet 15e4d972e8254cf899c3e0b71b6e2bd9) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:14.523046 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 82d163681fda47f38d79863bd28f576d with delete_type TABLET_DATA_DELETED (replaced by tablet bf8f0237233a49219cd1c41cf808bdba) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:14.535892 1035 tablet_service.cc:1515] Processing DeleteTablet for tablet 121803e79ab64ec0b948b448646b9d0a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:14 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:14.536234 1034 tablet_service.cc:1468] Processing CreateTablet for tablet e25923b2daf9429683b2d44a75a56189 (DEFAULT_TABLE table=default.Table_202b4c5ccc56474a8b2e7b9d7c4a28d6 [id=746845bd84c6444b8a1e5683eaba4a68]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:14.536254 894 tablet_service.cc:1468] Processing CreateTablet for tablet e25923b2daf9429683b2d44a75a56189 (DEFAULT_TABLE table=default.Table_202b4c5ccc56474a8b2e7b9d7c4a28d6 [id=746845bd84c6444b8a1e5683eaba4a68]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:14.536684 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e25923b2daf9429683b2d44a75a56189. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:14.536684 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e25923b2daf9429683b2d44a75a56189. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:14.543299 1332 raft_consensus.cc:2241] T 447ffe73741640ceb73f35676232a094 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:14.543565 1332 pending_rounds.cc:70] T 447ffe73741640ceb73f35676232a094 P fd27ab6665304aaea53adbd102dd4f58: Trying to abort 1 pending ops.
I20250901 01:52:14.543656 1332 pending_rounds.cc:77] T 447ffe73741640ceb73f35676232a094 P fd27ab6665304aaea53adbd102dd4f58: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7195408520477831168 op_type: NO_OP noop_request { }
I20250901 01:52:14.543766 1332 raft_consensus.cc:2887] T 447ffe73741640ceb73f35676232a094 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250901 01:52:14.543871 1332 raft_consensus.cc:2270] T 447ffe73741640ceb73f35676232a094 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:14.544451 1332 ts_tablet_manager.cc:1905] T 447ffe73741640ceb73f35676232a094 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:14.549526 1151 tablet_bootstrap.cc:492] T 2685fb7258264463b5f1ec99cd7d67ad P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:14.549697 1151 ts_tablet_manager.cc:1397] T 2685fb7258264463b5f1ec99cd7d67ad P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.165s user 0.000s sys 0.004s
I20250901 01:52:14.550594 1151 raft_consensus.cc:357] T 2685fb7258264463b5f1ec99cd7d67ad P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:14.550824 1151 raft_consensus.cc:383] T 2685fb7258264463b5f1ec99cd7d67ad P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:14.550915 1151 raft_consensus.cc:738] T 2685fb7258264463b5f1ec99cd7d67ad P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:14.551182 1151 consensus_queue.cc:260] T 2685fb7258264463b5f1ec99cd7d67ad P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:14.551440 1517 raft_consensus.cc:491] T 8381305e4c4148808803e255b8a43a90 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:14.551601 1306 raft_consensus.cc:2270] T 1f18420261ae41459e1f25b927bd84a9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:14.551602 1517 raft_consensus.cc:513] T 8381305e4c4148808803e255b8a43a90 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:14.552284 1306 ts_tablet_manager.cc:1905] T 1f18420261ae41459e1f25b927bd84a9 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:14.552610 1517 leader_election.cc:290] T 8381305e4c4148808803e255b8a43a90 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:14.553071 1151 ts_tablet_manager.cc:1428] T 2685fb7258264463b5f1ec99cd7d67ad P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20250901 01:52:14.553488 1151 tablet_bootstrap.cc:492] T 360755d30b5d47ba98c9c7e23ff5ab7e P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:14.555972 1151 tablet_bootstrap.cc:654] T 360755d30b5d47ba98c9c7e23ff5ab7e P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:14.562271 1374 raft_consensus.cc:491] T 4cdd369214ef4f5ebabddbf28b2ae3f7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:14.562494 1374 raft_consensus.cc:513] T 4cdd369214ef4f5ebabddbf28b2ae3f7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:14.563030 1374 leader_election.cc:290] T 4cdd369214ef4f5ebabddbf28b2ae3f7 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:14.563973 992 leader_election.cc:343] T 4cdd369214ef4f5ebabddbf28b2ae3f7 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 4cdd369214ef4f5ebabddbf28b2ae3f7
I20250901 01:52:14.564518 1306 ts_tablet_manager.cc:1918] T 1f18420261ae41459e1f25b927bd84a9 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:14.564680 1306 log.cc:1199] T 1f18420261ae41459e1f25b927bd84a9 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/1f18420261ae41459e1f25b927bd84a9
I20250901 01:52:14.565085 1306 ts_tablet_manager.cc:1939] T 1f18420261ae41459e1f25b927bd84a9 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:14.565822 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 1f18420261ae41459e1f25b927bd84a9 (table default.Table_a2bf411424c74109a7c2baa28a4db8b9 [id=7c904bfb6cc94518873d6806d1fda600]) successfully deleted
W20250901 01:52:14.566210 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 66a48a8ebe154b2cac89c234eaa378ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 66a48a8ebe154b2cac89c234eaa378ff already in progress: creating tablet
W20250901 01:52:14.566431 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 82d163681fda47f38d79863bd28f576d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 82d163681fda47f38d79863bd28f576d already in progress: creating tablet
W20250901 01:52:14.566743 440 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 121803e79ab64ec0b948b448646b9d0a because the tablet was not found. No further retry: Not found: Tablet not found: 121803e79ab64ec0b948b448646b9d0a
I20250901 01:52:14.567095 1150 raft_consensus.cc:357] T fcb7b5b7d63d4f0b906fea97cb548de2 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:14.567310 1150 raft_consensus.cc:383] T fcb7b5b7d63d4f0b906fea97cb548de2 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:14.567409 1150 raft_consensus.cc:738] T fcb7b5b7d63d4f0b906fea97cb548de2 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:14.569471 1058 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8381305e4c4148808803e255b8a43a90" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:14.570078 859 leader_election.cc:343] T 8381305e4c4148808803e255b8a43a90 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:14.570995 1150 consensus_queue.cc:260] T fcb7b5b7d63d4f0b906fea97cb548de2 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:14.574769 1150 ts_tablet_manager.cc:1428] T fcb7b5b7d63d4f0b906fea97cb548de2 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.134s user 0.001s sys 0.001s
I20250901 01:52:14.575088 1150 tablet_bootstrap.cc:492] T 5adf6b492640417ca071c94702b48cb9 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:14.575606 1034 tablet_service.cc:1468] Processing CreateTablet for tablet c763d252598745a088344aa0eb5fa3d3 (DEFAULT_TABLE table=default.Table_142ccc8cc7414621bc0c31326d5aa0e9 [id=6a01b6c6977546be9c6128ca5f200a92]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:14.576099 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c763d252598745a088344aa0eb5fa3d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:14.576208 894 tablet_service.cc:1468] Processing CreateTablet for tablet c763d252598745a088344aa0eb5fa3d3 (DEFAULT_TABLE table=default.Table_142ccc8cc7414621bc0c31326d5aa0e9 [id=6a01b6c6977546be9c6128ca5f200a92]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:14.576664 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c763d252598745a088344aa0eb5fa3d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:14.590001 1150 tablet_bootstrap.cc:654] T 5adf6b492640417ca071c94702b48cb9 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:14.592201 897 tablet_service.cc:1468] Processing CreateTablet for tablet 5d794471dbed4477b9d52fd411fa4d10 (DEFAULT_TABLE table=default.Table_dc856980ff4c4aafb92cf1fd4f29e694 [id=917ac6e5eaf54d019896a1d1ae2f6622]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:14.592815 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d794471dbed4477b9d52fd411fa4d10. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:14.593564 1034 tablet_service.cc:1468] Processing CreateTablet for tablet 5d794471dbed4477b9d52fd411fa4d10 (DEFAULT_TABLE table=default.Table_dc856980ff4c4aafb92cf1fd4f29e694 [id=917ac6e5eaf54d019896a1d1ae2f6622]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:14.593986 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d794471dbed4477b9d52fd411fa4d10. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:14.596135 1151 tablet_bootstrap.cc:492] T 360755d30b5d47ba98c9c7e23ff5ab7e P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:14.596292 1151 ts_tablet_manager.cc:1397] T 360755d30b5d47ba98c9c7e23ff5ab7e P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.043s user 0.009s sys 0.003s
I20250901 01:52:14.597530 1151 raft_consensus.cc:357] T 360755d30b5d47ba98c9c7e23ff5ab7e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:14.597738 1151 raft_consensus.cc:383] T 360755d30b5d47ba98c9c7e23ff5ab7e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:14.597829 1151 raft_consensus.cc:738] T 360755d30b5d47ba98c9c7e23ff5ab7e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:14.598276 1151 consensus_queue.cc:260] T 360755d30b5d47ba98c9c7e23ff5ab7e P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:14.599956 1151 ts_tablet_manager.cc:1428] T 360755d30b5d47ba98c9c7e23ff5ab7e P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20250901 01:52:14.600306 1151 tablet_bootstrap.cc:492] T 0e698923812841b7a5d37d57da103c5c P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:14.605079 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 2685fb7258264463b5f1ec99cd7d67ad
I20250901 01:52:14.605635 448 catalog_manager.cc:6652] ReplaceTablet: tablet 2685fb7258264463b5f1ec99cd7d67ad deleted and replaced by tablet 25ce0fe8c9544f828694184d19d13df9
I20250901 01:52:14.605993 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_3552b4561f10466b9b77f797ca003e53"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:14.606643 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3552b4561f10466b9b77f797ca003e53 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:14.607220 1151 tablet_bootstrap.cc:654] T 0e698923812841b7a5d37d57da103c5c P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:14.608203 455 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 670d90a37b934579b353bb4e4820127b consensus state has no leader
I20250901 01:52:14.609218 1035 tablet_service.cc:1515] Processing DeleteTablet for tablet 2685fb7258264463b5f1ec99cd7d67ad with delete_type TABLET_DATA_DELETED (replaced by tablet 25ce0fe8c9544f828694184d19d13df9) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:14.610427 455 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_d4d399ae87234df7a710191d0ecf02d0" } new_table_name: "default.Table_2d3d2227e6a24bd2a86f457e553e698b" modify_external_catalogs: true
I20250901 01:52:14.617022 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_fe014107d0ee4fb8bdfca5356093854b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:14.617910 1151 tablet_bootstrap.cc:492] T 0e698923812841b7a5d37d57da103c5c P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:14.618112 1151 ts_tablet_manager.cc:1397] T 0e698923812841b7a5d37d57da103c5c P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
W20250901 01:52:14.618235 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fe014107d0ee4fb8bdfca5356093854b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250901 01:52:14.623461 442 consensus_peers.cc:489] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227 -> Peer 7ce2288f784044eea010664e3def133f (127.0.104.60:44231): Couldn't send request to peer 7ce2288f784044eea010664e3def133f. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.60:44231: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250901 01:52:14.623834 897 tablet_service.cc:1515] Processing DeleteTablet for tablet 2685fb7258264463b5f1ec99cd7d67ad with delete_type TABLET_DATA_DELETED (replaced by tablet 25ce0fe8c9544f828694184d19d13df9) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:14.624136 1306 tablet_replica.cc:331] T 2685fb7258264463b5f1ec99cd7d67ad P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:14.624342 1306 raft_consensus.cc:2241] T 2685fb7258264463b5f1ec99cd7d67ad P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:14.624565 1306 raft_consensus.cc:2270] T 2685fb7258264463b5f1ec99cd7d67ad P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:14.624991 451 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 467c2bf4a8cb4937b6f627bea28d8ddb (table default.Table_a0e83c6959d64160a5fdec67e0e49ba3 [id=f3735a7cf9634c11abc2c8121e2ed930]). Expected version 1 got 0
I20250901 01:52:14.625217 1306 ts_tablet_manager.cc:1905] T 2685fb7258264463b5f1ec99cd7d67ad P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:14.620086 1151 raft_consensus.cc:357] T 0e698923812841b7a5d37d57da103c5c P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:14.627578 1151 raft_consensus.cc:383] T 0e698923812841b7a5d37d57da103c5c P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:14.627667 1151 raft_consensus.cc:738] T 0e698923812841b7a5d37d57da103c5c P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:14.627895 1151 consensus_queue.cc:260] T 0e698923812841b7a5d37d57da103c5c P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:14.630386 897 tablet_service.cc:1468] Processing CreateTablet for tablet 92f0ad59fbbb435ea872c669a52f854a (DEFAULT_TABLE table=default.Table_4aa01c0e60364804a5e99d552ca3956c [id=59306e9184ca4715b1cd675609807eca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:14.630401 1035 tablet_service.cc:1468] Processing CreateTablet for tablet f8a5886ddb644a86b76df44aab309057 (DEFAULT_TABLE table=default.Table_79a6a0f0c6944313b9987f23b6a09557 [id=3f3723927dc04cf99499f68d39db8038]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:14.630798 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8a5886ddb644a86b76df44aab309057. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:14.630821 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92f0ad59fbbb435ea872c669a52f854a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:14.631064 1151 ts_tablet_manager.cc:1428] T 0e698923812841b7a5d37d57da103c5c P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.013s user 0.002s sys 0.001s
I20250901 01:52:14.632025 1151 tablet_bootstrap.cc:492] T 85322a864d154f299f2d670fdd2a7bfb P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:14.633314 896 tablet_service.cc:1468] Processing CreateTablet for tablet f8a5886ddb644a86b76df44aab309057 (DEFAULT_TABLE table=default.Table_79a6a0f0c6944313b9987f23b6a09557 [id=3f3723927dc04cf99499f68d39db8038]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:14.633684 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8a5886ddb644a86b76df44aab309057. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:14.639132 1151 tablet_bootstrap.cc:654] T 85322a864d154f299f2d670fdd2a7bfb P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:14.645545 1150 tablet_bootstrap.cc:492] T 5adf6b492640417ca071c94702b48cb9 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:14.645713 1150 ts_tablet_manager.cc:1397] T 5adf6b492640417ca071c94702b48cb9 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.071s user 0.002s sys 0.002s
I20250901 01:52:14.646505 1150 raft_consensus.cc:357] T 5adf6b492640417ca071c94702b48cb9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:14.646685 1150 raft_consensus.cc:383] T 5adf6b492640417ca071c94702b48cb9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:14.646775 1150 raft_consensus.cc:738] T 5adf6b492640417ca071c94702b48cb9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:14.647060 1150 consensus_queue.cc:260] T 5adf6b492640417ca071c94702b48cb9 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:14.647668 1150 ts_tablet_manager.cc:1428] T 5adf6b492640417ca071c94702b48cb9 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250901 01:52:14.647965 1150 tablet_bootstrap.cc:492] T a59ac57aebe24860aa02f53ce0485910 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:14.649901 1150 tablet_bootstrap.cc:654] T a59ac57aebe24860aa02f53ce0485910 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:14.650305 453 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet b633b461dbf34e13bc6e1cc08e6d20c1
I20250901 01:52:14.650843 453 catalog_manager.cc:6652] ReplaceTablet: tablet b633b461dbf34e13bc6e1cc08e6d20c1 deleted and replaced by tablet 4855448edb4b40c184838aa7f81c112b
I20250901 01:52:14.653038 1035 tablet_service.cc:1515] Processing DeleteTablet for tablet b633b461dbf34e13bc6e1cc08e6d20c1 with delete_type TABLET_DATA_DELETED (replaced by tablet 4855448edb4b40c184838aa7f81c112b) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:14.653580 896 tablet_service.cc:1515] Processing DeleteTablet for tablet b633b461dbf34e13bc6e1cc08e6d20c1 with delete_type TABLET_DATA_DELETED (replaced by tablet 4855448edb4b40c184838aa7f81c112b) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:14.654088 1306 ts_tablet_manager.cc:1918] T 2685fb7258264463b5f1ec99cd7d67ad P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:14.654217 1306 log.cc:1199] T 2685fb7258264463b5f1ec99cd7d67ad P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/2685fb7258264463b5f1ec99cd7d67ad
I20250901 01:52:14.654623 1306 ts_tablet_manager.cc:1939] T 2685fb7258264463b5f1ec99cd7d67ad P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:14.655336 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 2685fb7258264463b5f1ec99cd7d67ad (table default.Table_dcacb44eed4241d3acbae153ebaad663 [id=0241aa3ab1984557a151461005c6b97c]) successfully deleted
I20250901 01:52:14.665162 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_5e32028e03b04431b57a5d3a653afda4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:14.665956 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5e32028e03b04431b57a5d3a653afda4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:14.667912 1035 tablet_service.cc:1515] Processing DeleteTablet for tablet 360755d30b5d47ba98c9c7e23ff5ab7e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:10 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:14.668424 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_583e53c2e86544e9be68eb27ab6068a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:14.669132 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_583e53c2e86544e9be68eb27ab6068a3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250901 01:52:14.673451 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet b633b461dbf34e13bc6e1cc08e6d20c1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b633b461dbf34e13bc6e1cc08e6d20c1 already in progress: creating tablet
W20250901 01:52:14.676111 702 kernel_stack_watchdog.cc:198] Thread 1149 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 5176ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fe78f073980 (unknown) at ??:0
@ 0x92cb07 __sanitizer::internal_sched_yield() at ??:?
@ 0x8921bd __interceptor_pthread_create at ??:?
@ 0x7fe78bc459c1 kudu::Thread::StartThread() at ??:0
@ 0xa63f97 kudu::Thread::Create() at ??:?
@ 0x7fe78bc6580b kudu::ThreadPool::CreateThread() at ??:0
@ 0x7fe78bc6122d kudu::ThreadPool::DoSubmit() at ??:0
@ 0x7fe78bc6655f kudu::ThreadPool::Submit() at ??:0
@ 0x7fe79701088d kudu::log::SegmentAllocator::AsyncAllocateSegmentUnlocked() at ??:0
@ 0x7fe79700f87f kudu::log::SegmentAllocator::AllocateSegmentAndRollOver() at ??:0
@ 0x7fe79700f43e kudu::log::SegmentAllocator::Init() at ??:0
@ 0x7fe7970174b5 kudu::log::Log::Init() at ??:0
@ 0x7fe797016811 kudu::log::Log::Open() at ??:0
@ 0x7fe79895a5e1 kudu::tablet::TabletBootstrap::OpenNewLog() at ??:0
@ 0x7fe79895694c kudu::tablet::TabletBootstrap::RunBootstrap() at ??:0
@ 0x7fe7989550cf kudu::tablet::TabletBootstrap::Bootstrap() at ??:0
I20250901 01:52:14.677021 449 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet b6daeff0070a4db08e90020752d21c29
W20250901 01:52:14.677516 992 consensus_peers.cc:489] T f0affc7a78634607852a48458412761d P fd27ab6665304aaea53adbd102dd4f58 -> Peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Couldn't send request to peer 00c282f779c1477d920f24dc0dc16628. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f0affc7a78634607852a48458412761d. This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:14.680256 1035 tablet_service.cc:1515] Processing DeleteTablet for tablet b6daeff0070a4db08e90020752d21c29 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:14 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:14.681136 896 tablet_service.cc:1515] Processing DeleteTablet for tablet b6daeff0070a4db08e90020752d21c29 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:14 UTC) from {username='slave'} at 127.0.0.1:52846
W20250901 01:52:14.681696 440 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet b6daeff0070a4db08e90020752d21c29 because the tablet was not found. No further retry: Not found: Tablet not found: b6daeff0070a4db08e90020752d21c29
I20250901 01:52:14.682713 1034 tablet_service.cc:1468] Processing CreateTablet for tablet 92f0ad59fbbb435ea872c669a52f854a (DEFAULT_TABLE table=default.Table_4aa01c0e60364804a5e99d552ca3956c [id=59306e9184ca4715b1cd675609807eca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:14.683131 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92f0ad59fbbb435ea872c669a52f854a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:14.699452 1151 tablet_bootstrap.cc:492] T 85322a864d154f299f2d670fdd2a7bfb P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:14.699620 1151 ts_tablet_manager.cc:1397] T 85322a864d154f299f2d670fdd2a7bfb P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.068s user 0.002s sys 0.008s
I20250901 01:52:14.700493 1151 raft_consensus.cc:357] T 85322a864d154f299f2d670fdd2a7bfb P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:14.700737 1151 raft_consensus.cc:383] T 85322a864d154f299f2d670fdd2a7bfb P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:14.700868 1151 raft_consensus.cc:738] T 85322a864d154f299f2d670fdd2a7bfb P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:14.701166 1151 consensus_queue.cc:260] T 85322a864d154f299f2d670fdd2a7bfb P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:14.702936 1151 ts_tablet_manager.cc:1428] T 85322a864d154f299f2d670fdd2a7bfb P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250901 01:52:14.702960 1035 tablet_service.cc:1468] Processing CreateTablet for tablet 75e7a90815e449c39f386901a504143c (DEFAULT_TABLE table=default.Table_ca246ea8f871412d949125b3714d8b67 [id=85d86bb68a4b43c4a8d08dd9ce12a4fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:14.705736 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 670d90a37b934579b353bb4e4820127b (DEFAULT_TABLE table=default.Table_2f09538b6fbe401f909105decb56e1cb [id=a90d1f10531042c3bb7a196b8be61c5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:14.706523 896 tablet_service.cc:1468] Processing CreateTablet for tablet 670d90a37b934579b353bb4e4820127b (DEFAULT_TABLE table=default.Table_2f09538b6fbe401f909105decb56e1cb [id=a90d1f10531042c3bb7a196b8be61c5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:14.707093 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 670d90a37b934579b353bb4e4820127b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:14.713474 894 tablet_service.cc:1468] Processing CreateTablet for tablet 75e7a90815e449c39f386901a504143c (DEFAULT_TABLE table=default.Table_ca246ea8f871412d949125b3714d8b67 [id=85d86bb68a4b43c4a8d08dd9ce12a4fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:14.714159 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75e7a90815e449c39f386901a504143c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:14.715754 1374 raft_consensus.cc:491] T 85322a864d154f299f2d670fdd2a7bfb P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:14.715924 1374 raft_consensus.cc:513] T 85322a864d154f299f2d670fdd2a7bfb P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:14.716782 1374 leader_election.cc:290] T 85322a864d154f299f2d670fdd2a7bfb P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:14.718719 923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "85322a864d154f299f2d670fdd2a7bfb" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:14.719046 923 raft_consensus.cc:2466] T 85322a864d154f299f2d670fdd2a7bfb P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:14.719640 992 leader_election.cc:304] T 85322a864d154f299f2d670fdd2a7bfb P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:14.720026 1374 raft_consensus.cc:2802] T 85322a864d154f299f2d670fdd2a7bfb P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:14.720158 1374 raft_consensus.cc:491] T 85322a864d154f299f2d670fdd2a7bfb P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:14.720252 1374 raft_consensus.cc:3058] T 85322a864d154f299f2d670fdd2a7bfb P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:14.723932 1374 raft_consensus.cc:513] T 85322a864d154f299f2d670fdd2a7bfb P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:14.725396 1540 raft_consensus.cc:491] T 34042ea5c1b645969c1de41fe587c343 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:14.725426 1374 leader_election.cc:290] T 85322a864d154f299f2d670fdd2a7bfb P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:14.725638 1540 raft_consensus.cc:513] T 34042ea5c1b645969c1de41fe587c343 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:14.726758 1540 leader_election.cc:290] T 34042ea5c1b645969c1de41fe587c343 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:14.728742 923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "85322a864d154f299f2d670fdd2a7bfb" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:14.728963 923 raft_consensus.cc:3058] T 85322a864d154f299f2d670fdd2a7bfb P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:14.731818 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 670d90a37b934579b353bb4e4820127b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:14.733058 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_6f470109ee524f10a48d1080ec15a199"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:14.733167 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "34042ea5c1b645969c1de41fe587c343" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:14.733669 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6f470109ee524f10a48d1080ec15a199 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:14.736675 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75e7a90815e449c39f386901a504143c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:14.736757 923 raft_consensus.cc:2466] T 85322a864d154f299f2d670fdd2a7bfb P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:14.740864 992 leader_election.cc:304] T 85322a864d154f299f2d670fdd2a7bfb P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:14.741389 1038 tablet_service.cc:1515] Processing DeleteTablet for tablet 0e698923812841b7a5d37d57da103c5c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:10 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:14.741994 1374 raft_consensus.cc:2802] T 85322a864d154f299f2d670fdd2a7bfb P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:14.742915 1151 tablet_bootstrap.cc:492] T 88a6ab29d00542aba54f0e4185fbf615 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
W20250901 01:52:14.743383 859 leader_election.cc:343] T 34042ea5c1b645969c1de41fe587c343 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:14.744031 1540 raft_consensus.cc:491] T d60d571b2c2840a2b20629bb832d1309 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:14.744216 1540 raft_consensus.cc:513] T d60d571b2c2840a2b20629bb832d1309 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:14.744975 1540 leader_election.cc:290] T d60d571b2c2840a2b20629bb832d1309 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:14.745337 1151 tablet_bootstrap.cc:654] T 88a6ab29d00542aba54f0e4185fbf615 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:14.745970 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d60d571b2c2840a2b20629bb832d1309" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:14.746477 859 leader_election.cc:343] T d60d571b2c2840a2b20629bb832d1309 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
W20250901 01:52:14.748692 991 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250901 01:52:14.756752 450 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 3b18a7084727490380d26b3485e4e799 (table default.Table_da2c70a8f2fe49a0a8184e88c0a9e0ea [id=c1b1e9e3876244f68d31c63ce7bdebf1]). Expected version 1 got 0
I20250901 01:52:14.757149 447 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet ceb724d859d24d569598e18389dae3b3
I20250901 01:52:14.759522 902 tablet_service.cc:1515] Processing DeleteTablet for tablet ceb724d859d24d569598e18389dae3b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:14 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:14.759825 1306 tablet_replica.cc:331] T ceb724d859d24d569598e18389dae3b3 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:14.760061 1306 raft_consensus.cc:2241] T ceb724d859d24d569598e18389dae3b3 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:14.760293 1306 raft_consensus.cc:2270] T ceb724d859d24d569598e18389dae3b3 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:14.761018 1306 ts_tablet_manager.cc:1905] T ceb724d859d24d569598e18389dae3b3 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:14.761783 902 tablet_service.cc:1515] Processing DeleteTablet for tablet d60d571b2c2840a2b20629bb832d1309 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:13 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:14.762531 902 tablet_service.cc:1515] Processing DeleteTablet for tablet 3df908c3096a4df7bddd8ac88b49cfe0 with delete_type TABLET_DATA_DELETED (replaced by tablet b009cc95c819431ea9827a3e453b8da3) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:14.763248 902 tablet_service.cc:1515] Processing DeleteTablet for tablet 2f28451a3ecc481a958588466f7fec77 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:12 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:14.764962 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_9bf84b7a07be4f57a58101480fe6d2c0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:14.765635 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9bf84b7a07be4f57a58101480fe6d2c0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:14.767577 1306 ts_tablet_manager.cc:1918] T ceb724d859d24d569598e18389dae3b3 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:14.767724 1306 log.cc:1199] T ceb724d859d24d569598e18389dae3b3 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/ceb724d859d24d569598e18389dae3b3
I20250901 01:52:14.768122 1306 ts_tablet_manager.cc:1939] T ceb724d859d24d569598e18389dae3b3 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:14.768703 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet ceb724d859d24d569598e18389dae3b3 (table default.Table_063de0c73851415196ba2e77742edee4 [id=696debd71f87437a880d4ad17fdf5ad8]) successfully deleted
I20250901 01:52:14.768779 1306 tablet_replica.cc:331] T d60d571b2c2840a2b20629bb832d1309 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:14.768955 1306 raft_consensus.cc:2241] T d60d571b2c2840a2b20629bb832d1309 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:14.769181 1306 raft_consensus.cc:2270] T d60d571b2c2840a2b20629bb832d1309 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:14.769342 450 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_dfd01d6b61ac43538c37f74330579cbd" } modify_external_catalogs: true
I20250901 01:52:14.769616 450 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_dfd01d6b61ac43538c37f74330579cbd" } modify_external_catalogs: true
I20250901 01:52:14.770010 1306 ts_tablet_manager.cc:1905] T d60d571b2c2840a2b20629bb832d1309 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:14.775322 1034 tablet_service.cc:1468] Processing CreateTablet for tablet b2e8b2cd10154de08ddd30c214592a4a (DEFAULT_TABLE table=default.Table_fad16c8446b049ec9b4a90aba2ef14de [id=fd72e46c84634b6c9ea74970480ecc62]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:14.775884 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2e8b2cd10154de08ddd30c214592a4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:14.776854 902 tablet_service.cc:1468] Processing CreateTablet for tablet b2e8b2cd10154de08ddd30c214592a4a (DEFAULT_TABLE table=default.Table_fad16c8446b049ec9b4a90aba2ef14de [id=fd72e46c84634b6c9ea74970480ecc62]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:14.777244 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2e8b2cd10154de08ddd30c214592a4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:14.783916 1374 raft_consensus.cc:695] T 85322a864d154f299f2d670fdd2a7bfb P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:14.784309 1374 consensus_queue.cc:237] T 85322a864d154f299f2d670fdd2a7bfb P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:14.784961 1035 tablet_service.cc:1515] Processing DeleteTablet for tablet ceb724d859d24d569598e18389dae3b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:14 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:14.786347 1151 tablet_bootstrap.cc:492] T 88a6ab29d00542aba54f0e4185fbf615 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:14.786572 1151 ts_tablet_manager.cc:1397] T 88a6ab29d00542aba54f0e4185fbf615 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.044s user 0.005s sys 0.000s
I20250901 01:52:14.787355 1151 raft_consensus.cc:357] T 88a6ab29d00542aba54f0e4185fbf615 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:14.787627 1151 raft_consensus.cc:383] T 88a6ab29d00542aba54f0e4185fbf615 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:14.787737 1151 raft_consensus.cc:738] T 88a6ab29d00542aba54f0e4185fbf615 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:14.788054 1151 consensus_queue.cc:260] T 88a6ab29d00542aba54f0e4185fbf615 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:14.790508 1552 raft_consensus.cc:491] T b29e846dfae44f0e9ce3e5fea3763a4f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:14.790514 902 tablet_service.cc:1468] Processing CreateTablet for tablet 9127f9c5239e445ba86012951b7c75d9 (DEFAULT_TABLE table=default.Table_da8cc5e7d1e24421b170ecca2fc2147b [id=ab670bb02c6e4ed491e7831ad73cdfc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:14.790700 1552 raft_consensus.cc:513] T b29e846dfae44f0e9ce3e5fea3763a4f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:14.791064 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9127f9c5239e445ba86012951b7c75d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:14.793112 1552 leader_election.cc:290] T b29e846dfae44f0e9ce3e5fea3763a4f P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:14.793568 992 leader_election.cc:343] T b29e846dfae44f0e9ce3e5fea3763a4f P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: b29e846dfae44f0e9ce3e5fea3763a4f
I20250901 01:52:14.796706 1035 tablet_service.cc:1468] Processing CreateTablet for tablet 9127f9c5239e445ba86012951b7c75d9 (DEFAULT_TABLE table=default.Table_da8cc5e7d1e24421b170ecca2fc2147b [id=ab670bb02c6e4ed491e7831ad73cdfc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:14.797243 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9127f9c5239e445ba86012951b7c75d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:14.803746 1151 ts_tablet_manager.cc:1428] T 88a6ab29d00542aba54f0e4185fbf615 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20250901 01:52:14.808640 1151 tablet_bootstrap.cc:492] T d6a4040747fa4d588756506ee122d2dc P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:14.810263 1332 ts_tablet_manager.cc:1918] T 447ffe73741640ceb73f35676232a094 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:14.814502 1332 log.cc:1199] T 447ffe73741640ceb73f35676232a094 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/447ffe73741640ceb73f35676232a094
I20250901 01:52:14.815038 1332 ts_tablet_manager.cc:1939] T 447ffe73741640ceb73f35676232a094 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:14.815560 1332 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44246 (request call id 162) took 1712 ms. Trace:
I20250901 01:52:14.815765 1332 rpcz_store.cc:276] 0901 01:52:13.103248 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:13.103311 (+ 63us) service_pool.cc:225] Handling call
0901 01:52:14.377944 (+1274633us) ts_tablet_manager.cc:1145] Deleting tablet 447ffe73741640ceb73f35676232a094
0901 01:52:14.377959 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:14.657126 (+279167us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:14.810101 (+152975us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:14.815524 (+ 5423us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13260,"tablet-delete.queue_time_us":1260727}
I20250901 01:52:14.816576 1332 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44246 (request call id 159) took 1761 ms. Trace:
I20250901 01:52:14.816608 441 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 447ffe73741640ceb73f35676232a094 (table default.Table_ac8663b3415c4c668e298a84b2deb8c3 [id=33cd13ca007c4308957250255f985e70]) successfully deleted
I20250901 01:52:14.816799 1332 rpcz_store.cc:276] 0901 01:52:13.055238 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:13.055386 (+ 148us) service_pool.cc:225] Handling call
0901 01:52:14.816370 (+1760984us) ts_tablet_manager.cc:1145] Deleting tablet 9496ec6021784800aef62996b0dee593
0901 01:52:14.816379 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:14.816540 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":768,"tablet-delete.queue_time_us":1701693}
W20250901 01:52:14.817384 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 9496ec6021784800aef62996b0dee593 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9496ec6021784800aef62996b0dee593 already in progress: creating tablet
I20250901 01:52:14.817867 1332 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44246 (request call id 160) took 1762 ms. Trace:
I20250901 01:52:14.818073 1332 rpcz_store.cc:276] 0901 01:52:13.055436 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:13.055640 (+ 204us) service_pool.cc:225] Handling call
0901 01:52:14.817722 (+1762082us) ts_tablet_manager.cc:1145] Deleting tablet 154574e17f214cc0ad56e6e6be3af899
0901 01:52:14.817734 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:14.817834 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1702985}
W20250901 01:52:14.819242 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 154574e17f214cc0ad56e6e6be3af899 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 154574e17f214cc0ad56e6e6be3af899 already in progress: creating tablet
I20250901 01:52:14.819648 1332 tablet_replica.cc:331] T 342b7fde3ede494b9eec6e361db255dc P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:14.824497 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet f8a5886ddb644a86b76df44aab309057
I20250901 01:52:14.827261 897 tablet_service.cc:1515] Processing DeleteTablet for tablet f8a5886ddb644a86b76df44aab309057 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:14 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:14.811213 1151 tablet_bootstrap.cc:654] T d6a4040747fa4d588756506ee122d2dc P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:14.830018 454 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet fcb7b5b7d63d4f0b906fea97cb548de2 (table default.Table_dd8e6f9348c440bca1569cf68eec5beb [id=4d9ea39016ca4bf4859f4f80ea4e9ed6]). Expected version 2 got 0
I20250901 01:52:14.830245 454 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 5adf6b492640417ca071c94702b48cb9 (table default.Table_947ec0e1a97a40b59be1e234bcaf4e0d [id=5d14706774d345c48ca3575feabb17b0]). Expected version 1 got 0
W20250901 01:52:14.831105 454 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 5adf6b492640417ca071c94702b48cb9 (table default.Table_947ec0e1a97a40b59be1e234bcaf4e0d [id=5d14706774d345c48ca3575feabb17b0], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 5adf6b492640417ca071c94702b48cb9 consensus state has no leader
I20250901 01:52:14.832989 454 catalog_manager.cc:5582] T 85322a864d154f299f2d670fdd2a7bfb P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } }
I20250901 01:52:14.833448 454 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 85322a864d154f299f2d670fdd2a7bfb (table default.Table_68f31d33b83544cea7b959bc7c41f94b [id=1d8f34cc14cb4e90a562ff4424b8577a]). Expected version 4 got 0
I20250901 01:52:14.835533 452 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet d77aadd7291647a0bae7c421f244c252
I20250901 01:52:14.836241 452 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_fad16c8446b049ec9b4a90aba2ef14de" } new_table_name: "default.Table_6cf5e8ab687249788f151b78ba59351a" modify_external_catalogs: true
I20250901 01:52:14.840346 1306 ts_tablet_manager.cc:1918] T d60d571b2c2840a2b20629bb832d1309 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:14.843138 1306 log.cc:1199] T d60d571b2c2840a2b20629bb832d1309 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/d60d571b2c2840a2b20629bb832d1309
I20250901 01:52:14.844458 1306 ts_tablet_manager.cc:1939] T d60d571b2c2840a2b20629bb832d1309 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:14.845280 899 tablet_service.cc:1468] Processing CreateTablet for tablet bb0cf5a6a27045a08df6845d7473184d (DEFAULT_TABLE table=default.Table_b23a4778615d4597950ce1709ab00d37 [id=bf59a9514b7d49c3b3c1e9c6064ef369]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:14.847286 897 tablet_service.cc:1515] Processing DeleteTablet for tablet d77aadd7291647a0bae7c421f244c252 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:14 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:14.847498 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb0cf5a6a27045a08df6845d7473184d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:14.847611 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet d60d571b2c2840a2b20629bb832d1309 (table default.Table_79c2dbec23ad41159b38d5177072d78c [id=d4c09fc79416433b9555ed44abf26bfd]) successfully deleted
I20250901 01:52:14.849453 1306 tablet_replica.cc:331] T 3df908c3096a4df7bddd8ac88b49cfe0 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:14.849678 1306 raft_consensus.cc:2241] T 3df908c3096a4df7bddd8ac88b49cfe0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:14.845458 904 tablet_service.cc:1468] Processing CreateTablet for tablet fae2779fac164b9f8052c3666db803b8 (DEFAULT_TABLE table=default.Table_7736ed4b5a0d4b1a816500c525e68ad3 [id=7191ff77d44c4c86aa44473814570d09]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:14.850782 904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fae2779fac164b9f8052c3666db803b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:14.852451 1150 tablet_bootstrap.cc:492] T a59ac57aebe24860aa02f53ce0485910 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:14.852644 1150 ts_tablet_manager.cc:1397] T a59ac57aebe24860aa02f53ce0485910 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.205s user 0.009s sys 0.011s
I20250901 01:52:14.853574 1150 raft_consensus.cc:357] T a59ac57aebe24860aa02f53ce0485910 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:14.854311 1306 raft_consensus.cc:2270] T 3df908c3096a4df7bddd8ac88b49cfe0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:14.855626 1306 ts_tablet_manager.cc:1905] T 3df908c3096a4df7bddd8ac88b49cfe0 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:14.859639 1150 raft_consensus.cc:383] T a59ac57aebe24860aa02f53ce0485910 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:14.859760 1150 raft_consensus.cc:738] T a59ac57aebe24860aa02f53ce0485910 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:14.861374 1150 consensus_queue.cc:260] T a59ac57aebe24860aa02f53ce0485910 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:14.868659 1150 ts_tablet_manager.cc:1428] T a59ac57aebe24860aa02f53ce0485910 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.016s user 0.000s sys 0.002s
I20250901 01:52:14.869063 1540 raft_consensus.cc:491] T 3b18a7084727490380d26b3485e4e799 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:14.869201 1540 raft_consensus.cc:513] T 3b18a7084727490380d26b3485e4e799 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:14.869824 1540 leader_election.cc:290] T 3b18a7084727490380d26b3485e4e799 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:14.869897 904 tablet_service.cc:1468] Processing CreateTablet for tablet d58f69fad01f403eab74a47d3695ed83 (DEFAULT_TABLE table=default.Table_95884295301d476fb3e03054727f9cf1 [id=1aa18a2513db4e99a72572846527663c]), partition=RANGE (key) PARTITION UNBOUNDED
W20250901 01:52:14.870280 859 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250901 01:52:14.870493 904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d58f69fad01f403eab74a47d3695ed83. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:14.870391 899 tablet_service.cc:1468] Processing CreateTablet for tablet a54af99bc10045e8b0b6868e5aed98b4 (DEFAULT_TABLE table=default.Table_12f764717f384ee4b906dec11b1212f8 [id=5fdd352d5e5e47a3a2bd8fd608a263dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:14.871492 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a54af99bc10045e8b0b6868e5aed98b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:14.874432 1306 ts_tablet_manager.cc:1918] T 3df908c3096a4df7bddd8ac88b49cfe0 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:14.874605 1306 log.cc:1199] T 3df908c3096a4df7bddd8ac88b49cfe0 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/3df908c3096a4df7bddd8ac88b49cfe0
I20250901 01:52:14.874693 1150 tablet_bootstrap.cc:492] T dd694cfa09564d3da5d279c0861a4c32 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:14.875003 1306 ts_tablet_manager.cc:1939] T 3df908c3096a4df7bddd8ac88b49cfe0 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:14.875571 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 3df908c3096a4df7bddd8ac88b49cfe0 (table default.Table_7bb211d400ca44d48416f325df223fa6 [id=cc26c89db2484a6fad919231116285f2]) successfully deleted
I20250901 01:52:14.876655 1150 tablet_bootstrap.cc:654] T dd694cfa09564d3da5d279c0861a4c32 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:14.877269 1306 tablet_replica.cc:331] T 2f28451a3ecc481a958588466f7fec77 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:14.877748 1306 raft_consensus.cc:2241] T 2f28451a3ecc481a958588466f7fec77 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:14.878283 1306 raft_consensus.cc:2270] T 2f28451a3ecc481a958588466f7fec77 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:14.879719 1306 ts_tablet_manager.cc:1905] T 2f28451a3ecc481a958588466f7fec77 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:14.887719 448 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_8a4599f08716440e947a2d356d6363a5" } new_table_name: "default.Table_22d8d38d0b624912a0d8968fb764420e" modify_external_catalogs: true
I20250901 01:52:14.891783 1540 raft_consensus.cc:491] T 7442c6ff699141e1aff208ebdc9d192a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:14.891973 1540 raft_consensus.cc:513] T 7442c6ff699141e1aff208ebdc9d192a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:14.892755 1540 leader_election.cc:290] T 7442c6ff699141e1aff208ebdc9d192a P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:14.893491 1306 ts_tablet_manager.cc:1918] T 2f28451a3ecc481a958588466f7fec77 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:14.893633 1306 log.cc:1199] T 2f28451a3ecc481a958588466f7fec77 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/2f28451a3ecc481a958588466f7fec77
I20250901 01:52:14.894105 1306 ts_tablet_manager.cc:1939] T 2f28451a3ecc481a958588466f7fec77 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
W20250901 01:52:14.888274 455 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4fd0e4f9d86c4362af96ce9964b72328 consensus state has no leader
I20250901 01:52:14.896068 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_edd1006860a84e209149771e23bb8834"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:14.896855 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_edd1006860a84e209149771e23bb8834 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:14.898828 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 2f28451a3ecc481a958588466f7fec77 (table default.Table_12450bd55edc4f7d96797422cfae2c50 [id=a919133c60784a4ca1bec96b7f410029]) successfully deleted
W20250901 01:52:14.899132 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet f8a5886ddb644a86b76df44aab309057 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f8a5886ddb644a86b76df44aab309057 already in progress: creating tablet
W20250901 01:52:14.899363 440 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet d77aadd7291647a0bae7c421f244c252 because the tablet was not found. No further retry: Not found: Tablet not found: d77aadd7291647a0bae7c421f244c252
I20250901 01:52:14.888904 446 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_727db72cb2654b679929453b3a8fb00a" } new_table_name: "default.Table_5610feff8a344dc9be4bc1f0ad8ddd51" modify_external_catalogs: true
I20250901 01:52:14.904649 902 tablet_service.cc:1468] Processing CreateTablet for tablet 822382a1781b4304bce29355ce83730e (DEFAULT_TABLE table=default.Table_b300f31bdab14e24ad40f265ac4de311 [id=4a54bc023da7481fbb0ce573be49f4c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:14.905403 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 822382a1781b4304bce29355ce83730e. 1 dirs total, 0 dirs full, 0 dirs failed
W20250901 01:52:14.908807 991 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250901 01:52:14.908826 994 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250901 01:52:14.909258 1374 raft_consensus.cc:491] T 88a6ab29d00542aba54f0e4185fbf615 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:14.909444 1374 raft_consensus.cc:513] T 88a6ab29d00542aba54f0e4185fbf615 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:14.909843 1058 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7442c6ff699141e1aff208ebdc9d192a" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:14.910246 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet d77aadd7291647a0bae7c421f244c252 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:14 UTC) from {username='slave'} at 127.0.0.1:44246
W20250901 01:52:14.910718 859 leader_election.cc:343] T 7442c6ff699141e1aff208ebdc9d192a P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:14.910725 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3b18a7084727490380d26b3485e4e799" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:14.910949 1033 tablet_service.cc:1468] Processing CreateTablet for tablet fae2779fac164b9f8052c3666db803b8 (DEFAULT_TABLE table=default.Table_7736ed4b5a0d4b1a816500c525e68ad3 [id=7191ff77d44c4c86aa44473814570d09]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:14.911168 1034 tablet_service.cc:1515] Processing DeleteTablet for tablet f8a5886ddb644a86b76df44aab309057 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:14 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:14.910727 1038 tablet_service.cc:1468] Processing CreateTablet for tablet bb0cf5a6a27045a08df6845d7473184d (DEFAULT_TABLE table=default.Table_b23a4778615d4597950ce1709ab00d37 [id=bf59a9514b7d49c3b3c1e9c6064ef369]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:14.911422 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fae2779fac164b9f8052c3666db803b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:14.911645 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb0cf5a6a27045a08df6845d7473184d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:14.911937 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 822382a1781b4304bce29355ce83730e (DEFAULT_TABLE table=default.Table_b300f31bdab14e24ad40f265ac4de311 [id=4a54bc023da7481fbb0ce573be49f4c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:14.912153 1540 raft_consensus.cc:491] T 186ec810d02e4748aad15228b5866ac1 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:14.912262 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 822382a1781b4304bce29355ce83730e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:14.912320 1540 raft_consensus.cc:513] T 186ec810d02e4748aad15228b5866ac1 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:14.912691 1032 tablet_service.cc:1468] Processing CreateTablet for tablet a54af99bc10045e8b0b6868e5aed98b4 (DEFAULT_TABLE table=default.Table_12f764717f384ee4b906dec11b1212f8 [id=5fdd352d5e5e47a3a2bd8fd608a263dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:14.913012 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a54af99bc10045e8b0b6868e5aed98b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:14.913146 1540 leader_election.cc:290] T 186ec810d02e4748aad15228b5866ac1 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:14.916703 1026 tablet_service.cc:1468] Processing CreateTablet for tablet d58f69fad01f403eab74a47d3695ed83 (DEFAULT_TABLE table=default.Table_95884295301d476fb3e03054727f9cf1 [id=1aa18a2513db4e99a72572846527663c]), partition=RANGE (key) PARTITION UNBOUNDED
W20250901 01:52:14.916785 859 leader_election.cc:343] T 3b18a7084727490380d26b3485e4e799 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:14.917091 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d58f69fad01f403eab74a47d3695ed83. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:14.917014 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "186ec810d02e4748aad15228b5866ac1" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:14.923889 859 leader_election.cc:343] T 186ec810d02e4748aad15228b5866ac1 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:14.940393 1332 raft_consensus.cc:2241] T 342b7fde3ede494b9eec6e361db255dc P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:14.942190 1374 leader_election.cc:290] T 88a6ab29d00542aba54f0e4185fbf615 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:14.942270 992 leader_election.cc:343] T 88a6ab29d00542aba54f0e4185fbf615 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 88a6ab29d00542aba54f0e4185fbf615
I20250901 01:52:14.940989 1332 raft_consensus.cc:2270] T 342b7fde3ede494b9eec6e361db255dc P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:14.945051 1332 ts_tablet_manager.cc:1905] T 342b7fde3ede494b9eec6e361db255dc P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:14.945746 1374 raft_consensus.cc:491] T 4fd0e4f9d86c4362af96ce9964b72328 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:14.947489 1374 raft_consensus.cc:513] T 4fd0e4f9d86c4362af96ce9964b72328 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:14.948940 1374 leader_election.cc:290] T 4fd0e4f9d86c4362af96ce9964b72328 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:14.949105 923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4fd0e4f9d86c4362af96ce9964b72328" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:14.949450 923 raft_consensus.cc:2466] T 4fd0e4f9d86c4362af96ce9964b72328 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:14.949975 992 leader_election.cc:304] T 4fd0e4f9d86c4362af96ce9964b72328 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:14.950443 1374 raft_consensus.cc:2802] T 4fd0e4f9d86c4362af96ce9964b72328 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:14.950649 1374 raft_consensus.cc:491] T 4fd0e4f9d86c4362af96ce9964b72328 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:14.950805 1374 raft_consensus.cc:3058] T 4fd0e4f9d86c4362af96ce9964b72328 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:14.951653 1552 raft_consensus.cc:491] T d945b310fe6344af9cc9dd68f84c4bdf P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:14.951803 1552 raft_consensus.cc:513] T d945b310fe6344af9cc9dd68f84c4bdf P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:14.952384 1552 leader_election.cc:290] T d945b310fe6344af9cc9dd68f84c4bdf P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:14.953944 992 leader_election.cc:343] T d945b310fe6344af9cc9dd68f84c4bdf P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: d945b310fe6344af9cc9dd68f84c4bdf
I20250901 01:52:14.955996 1374 raft_consensus.cc:513] T 4fd0e4f9d86c4362af96ce9964b72328 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:14.957490 1374 leader_election.cc:290] T 4fd0e4f9d86c4362af96ce9964b72328 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:14.957494 923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4fd0e4f9d86c4362af96ce9964b72328" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:14.957799 923 raft_consensus.cc:3058] T 4fd0e4f9d86c4362af96ce9964b72328 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:14.960997 1374 raft_consensus.cc:491] T 360755d30b5d47ba98c9c7e23ff5ab7e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:14.961175 1374 raft_consensus.cc:513] T 360755d30b5d47ba98c9c7e23ff5ab7e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:14.961620 923 raft_consensus.cc:2466] T 4fd0e4f9d86c4362af96ce9964b72328 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:14.962186 992 leader_election.cc:304] T 4fd0e4f9d86c4362af96ce9964b72328 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:14.962489 1552 raft_consensus.cc:2802] T 4fd0e4f9d86c4362af96ce9964b72328 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
W20250901 01:52:14.972204 992 leader_election.cc:343] T 360755d30b5d47ba98c9c7e23ff5ab7e P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 360755d30b5d47ba98c9c7e23ff5ab7e
#29 0x7fe78bc46cc8 in kudu::Thread::SuperviseThread(void*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:694:3
I20250901 01:52:14.975512 1374 leader_election.cc:290] T 360755d30b5d47ba98c9c7e23ff5ab7e P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
#30 0x7fe78f0686da in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76da)
I20250901 01:52:14.980589 902 tablet_service.cc:1468] Processing CreateTablet for tablet 72ba689f9230411995dc96e2b45601df (DEFAULT_TABLE table=default.Table_8ea6f8f2bd8848109b6a9f9e7f5b6e54 [id=351a05134f4948039c920cafbe050914]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:14.981055 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72ba689f9230411995dc96e2b45601df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:14.981424 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_50ae3908a7944f70949c54be8fbf1564"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:14.982004 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_50ae3908a7944f70949c54be8fbf1564 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:14.983441 451 master_service.cc:948] ReplaceTablet: received request to replace tablet 9d886fa63f8f4996b087997b542e8bb1 from {username='slave'} at 127.0.0.1:57810
#31 0x7fe787d9a71e in clone (/lib/x86_64-linux-gnu/libc.so.6+0x12171e)
0x6190000f0088 is located 8 bytes inside of 1048-byte region [0x6190000f0080,0x6190000f0498)
freed by thread T138 (tablet-delete [) here:
I20250901 01:52:14.987600 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 72ba689f9230411995dc96e2b45601df (DEFAULT_TABLE table=default.Table_8ea6f8f2bd8848109b6a9f9e7f5b6e54 [id=351a05134f4948039c920cafbe050914]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:14.988345 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72ba689f9230411995dc96e2b45601df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:14.991070 902 tablet_service.cc:1468] Processing CreateTablet for tablet 121803e79ab64ec0b948b448646b9d0a (DEFAULT_TABLE table=default.Table_117c6c43b552476dbb8b7b5ccdce47b7 [id=a4c947d85200437395c66bb4f7e607af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:14.991524 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 121803e79ab64ec0b948b448646b9d0a. 1 dirs total, 0 dirs full, 0 dirs failed
#0 0x94d060 in operator delete(void*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/asan/asan_new_delete.cpp:160
I20250901 01:52:14.995754 1332 ts_tablet_manager.cc:1918] T 342b7fde3ede494b9eec6e361db255dc P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:14.995957 1332 log.cc:1199] T 342b7fde3ede494b9eec6e361db255dc P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/342b7fde3ede494b9eec6e361db255dc
I20250901 01:52:14.996068 1033 tablet_service.cc:1468] Processing CreateTablet for tablet 121803e79ab64ec0b948b448646b9d0a (DEFAULT_TABLE table=default.Table_117c6c43b552476dbb8b7b5ccdce47b7 [id=a4c947d85200437395c66bb4f7e607af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:14.996490 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 121803e79ab64ec0b948b448646b9d0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:14.996668 449 master_service.cc:948] ReplaceTablet: received request to replace tablet c763d252598745a088344aa0eb5fa3d3 from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:14.999476 1151 tablet_bootstrap.cc:492] T d6a4040747fa4d588756506ee122d2dc P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:14.999673 1151 ts_tablet_manager.cc:1397] T d6a4040747fa4d588756506ee122d2dc P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.195s user 0.004s sys 0.010s
I20250901 01:52:15.000826 1151 raft_consensus.cc:357] T d6a4040747fa4d588756506ee122d2dc P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:15.001068 1151 raft_consensus.cc:383] T d6a4040747fa4d588756506ee122d2dc P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:15.001176 1151 raft_consensus.cc:738] T d6a4040747fa4d588756506ee122d2dc P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:15.001470 1151 consensus_queue.cc:260] T d6a4040747fa4d588756506ee122d2dc P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:15.002225 1151 ts_tablet_manager.cc:1428] T d6a4040747fa4d588756506ee122d2dc P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250901 01:52:15.002600 1151 tablet_bootstrap.cc:492] T 756cfe84a517459ca015aecf5cd81571 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:15.003067 1293 raft_consensus.cc:491] T 4410da9e95454ecf95faf80221b0a71c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:15.003229 1293 raft_consensus.cc:513] T 4410da9e95454ecf95faf80221b0a71c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:15.003861 1293 leader_election.cc:290] T 4410da9e95454ecf95faf80221b0a71c P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:15.005987 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4410da9e95454ecf95faf80221b0a71c" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:15.006863 859 leader_election.cc:343] T 4410da9e95454ecf95faf80221b0a71c P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:15.007241 1540 raft_consensus.cc:491] T 5adf6b492640417ca071c94702b48cb9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:15.007390 1540 raft_consensus.cc:513] T 5adf6b492640417ca071c94702b48cb9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:15.007968 1540 leader_election.cc:290] T 5adf6b492640417ca071c94702b48cb9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:15.008718 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5adf6b492640417ca071c94702b48cb9" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
#1 0x7fe7989847ac in std::_Sp_counted_ptr<kudu::tablet::Tablet*, (__gnu_cxx::_Lock_policy)2>::_M_dispose() ../../../include/c++/7.5.0/bits/shared_ptr_base.h:376:9
W20250901 01:52:15.009246 859 leader_election.cc:343] T 5adf6b492640417ca071c94702b48cb9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
#2 0x96bf76 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() ../../../include/c++/7.5.0/bits/shared_ptr_base.h:154:6
#3 0x96be51 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() ../../../include/c++/7.5.0/bits/shared_ptr_base.h:684:11
I20250901 01:52:15.014297 1552 raft_consensus.cc:695] T 4fd0e4f9d86c4362af96ce9964b72328 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:15.014842 1552 consensus_queue.cc:237] T 4fd0e4f9d86c4362af96ce9964b72328 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:15.015842 1026 tablet_service.cc:1468] Processing CreateTablet for tablet d77aadd7291647a0bae7c421f244c252 (DEFAULT_TABLE table=default.Table_eacb38fbc98046dfac27344cfbed0a7c [id=015239c5e105476ca8bf6198560f0feb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:15.016381 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d77aadd7291647a0bae7c421f244c252. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:15.017232 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_746f677d34894fc0a027945dd2c04b53"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:15.017930 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_746f677d34894fc0a027945dd2c04b53 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:15.025985 1564 raft_consensus.cc:491] T 2685fb7258264463b5f1ec99cd7d67ad P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:15.026293 1564 raft_consensus.cc:513] T 2685fb7258264463b5f1ec99cd7d67ad P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:15.029568 992 leader_election.cc:343] T 2685fb7258264463b5f1ec99cd7d67ad P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 2685fb7258264463b5f1ec99cd7d67ad
I20250901 01:52:15.031042 1564 leader_election.cc:290] T 2685fb7258264463b5f1ec99cd7d67ad P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:15.031419 899 tablet_service.cc:1468] Processing CreateTablet for tablet b6daeff0070a4db08e90020752d21c29 (DEFAULT_TABLE table=default.Table_8022c1172d8a4df8bc1295d193bb6542 [id=f278d7b5596944238ef56dfb5de097dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:15.031847 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6daeff0070a4db08e90020752d21c29. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:15.032555 1035 tablet_service.cc:1468] Processing CreateTablet for tablet 3c1fd0820d8a4995abe027f685da7f7b (DEFAULT_TABLE table=default.Table_3b78b64e932d45458e89edf72fc45a6d [id=b3914f33e856466e8479d0a2263ec56f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:15.032814 904 tablet_service.cc:1468] Processing CreateTablet for tablet 3c1fd0820d8a4995abe027f685da7f7b (DEFAULT_TABLE table=default.Table_3b78b64e932d45458e89edf72fc45a6d [id=b3914f33e856466e8479d0a2263ec56f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:15.032954 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c1fd0820d8a4995abe027f685da7f7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:15.033164 904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c1fd0820d8a4995abe027f685da7f7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:15.036635 450 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet bf8f0237233a49219cd1c41cf808bdba
I20250901 01:52:15.037554 450 master_service.cc:948] ReplaceTablet: received request to replace tablet 920925ee9b014e14a79ceb7796ff7828 from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:15.046015 1332 ts_tablet_manager.cc:1939] T 342b7fde3ede494b9eec6e361db255dc P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:15.046622 1332 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44246 (request call id 163) took 1805 ms. Trace:
I20250901 01:52:15.046734 1332 rpcz_store.cc:276] 0901 01:52:13.241131 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:13.241233 (+ 102us) service_pool.cc:225] Handling call
0901 01:52:14.819569 (+1578336us) ts_tablet_manager.cc:1145] Deleting tablet 342b7fde3ede494b9eec6e361db255dc
0901 01:52:14.819581 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:14.950586 (+131005us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:14.995579 (+ 44993us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:15.046555 (+ 50976us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1577902}
I20250901 01:52:15.047242 1332 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44246 (request call id 164) took 1758 ms. Trace:
I20250901 01:52:15.047250 441 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 342b7fde3ede494b9eec6e361db255dc (table default.Table_e6bab04ba03d4d9fa029e550a887e303 [id=405abb093fe44077b374dc89acb88328]) successfully deleted
I20250901 01:52:15.047336 1332 rpcz_store.cc:276] 0901 01:52:13.288956 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:13.289017 (+ 61us) service_pool.cc:225] Handling call
0901 01:52:15.047072 (+1758055us) ts_tablet_manager.cc:1145] Deleting tablet 12c04e0c1f804179ae23509cdb24d395
0901 01:52:15.047083 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:15.047190 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1757822}
I20250901 01:52:15.047756 1332 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44246 (request call id 167) took 1655 ms. Trace:
W20250901 01:52:15.047839 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 12c04e0c1f804179ae23509cdb24d395 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 12c04e0c1f804179ae23509cdb24d395 already in progress: creating tablet
I20250901 01:52:15.047848 1332 rpcz_store.cc:276] 0901 01:52:13.398603 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:13.398660 (+ 57us) service_pool.cc:225] Handling call
0901 01:52:15.047583 (+1648923us) ts_tablet_manager.cc:1145] Deleting tablet 4d27fb61943c45d4aca67689cc3a9ff3
0901 01:52:15.047592 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:15.047710 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1648663}
I20250901 01:52:15.048203 1332 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44246 (request call id 168) took 1649 ms. Trace:
W20250901 01:52:15.048336 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 4d27fb61943c45d4aca67689cc3a9ff3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4d27fb61943c45d4aca67689cc3a9ff3 already in progress: creating tablet
W20250901 01:52:15.048843 991 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250901 01:52:15.048962 1026 tablet_service.cc:1468] Processing CreateTablet for tablet b6daeff0070a4db08e90020752d21c29 (DEFAULT_TABLE table=default.Table_8022c1172d8a4df8bc1295d193bb6542 [id=f278d7b5596944238ef56dfb5de097dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:15.049757 902 tablet_service.cc:1468] Processing CreateTablet for tablet d77aadd7291647a0bae7c421f244c252 (DEFAULT_TABLE table=default.Table_eacb38fbc98046dfac27344cfbed0a7c [id=015239c5e105476ca8bf6198560f0feb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:15.050115 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6daeff0070a4db08e90020752d21c29. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:15.061214 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d77aadd7291647a0bae7c421f244c252. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:15.070709 1255 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.60:44231
I20250901 01:52:15.070770 1570 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.60:44231 every 8 connection(s)
I20250901 01:52:15.072898 1150 tablet_bootstrap.cc:492] T dd694cfa09564d3da5d279c0861a4c32 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:15.073139 1150 ts_tablet_manager.cc:1397] T dd694cfa09564d3da5d279c0861a4c32 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.199s user 0.009s sys 0.009s
I20250901 01:52:15.074141 1150 raft_consensus.cc:357] T dd694cfa09564d3da5d279c0861a4c32 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:15.074440 1150 raft_consensus.cc:383] T dd694cfa09564d3da5d279c0861a4c32 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:15.074528 1150 raft_consensus.cc:738] T dd694cfa09564d3da5d279c0861a4c32 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:15.074823 1150 consensus_queue.cc:260] T dd694cfa09564d3da5d279c0861a4c32 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:15.075408 1150 ts_tablet_manager.cc:1428] T dd694cfa09564d3da5d279c0861a4c32 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250901 01:52:15.075695 1150 tablet_bootstrap.cc:492] T a7ff996482164776b468cf305989866b P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:15.077076 454 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_3f2af1af32b546d6b1eba5cc3195acd8" } new_table_name: "default.Table_f0e56dba1e8a4ed49a760e204c1ff96e" modify_external_catalogs: true
W20250901 01:52:15.083590 452 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b2e8b2cd10154de08ddd30c214592a4a consensus state has no leader
I20250901 01:52:15.085350 1255 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/master-2/data/info.pb
W20250901 01:52:15.091677 448 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 98cca86164db4252b1b49de88747822a consensus state has no leader
I20250901 01:52:15.093039 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskddMNdm/build/asan/bin/kudu as pid 1255
I20250901 01:52:15.103644 1552 raft_consensus.cc:491] T 2a0d125423a043dcb11e67345599e468 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:15.104369 1552 raft_consensus.cc:513] T 2a0d125423a043dcb11e67345599e468 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:15.107781 1552 leader_election.cc:290] T 2a0d125423a043dcb11e67345599e468 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:15.048301 1332 rpcz_store.cc:276] 0901 01:52:13.399097 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:13.399139 (+ 42us) service_pool.cc:225] Handling call
0901 01:52:15.048045 (+1648906us) ts_tablet_manager.cc:1145] Deleting tablet 22691943a134400c8df4617240e5d3ef
0901 01:52:15.048053 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:15.048140 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1648752}
W20250901 01:52:15.109850 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 22691943a134400c8df4617240e5d3ef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 22691943a134400c8df4617240e5d3ef already in progress: creating tablet
I20250901 01:52:15.111410 1332 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44246 (request call id 169) took 1711 ms. Trace:
I20250901 01:52:15.115857 1572 sys_catalog.cc:263] Verifying existing consensus state
I20250901 01:52:15.115609 1332 rpcz_store.cc:276] 0901 01:52:13.399428 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:13.399467 (+ 39us) service_pool.cc:225] Handling call
0901 01:52:15.111181 (+1711714us) ts_tablet_manager.cc:1145] Deleting tablet 1f18420261ae41459e1f25b927bd84a9
0901 01:52:15.111195 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:15.111324 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1711518}
I20250901 01:52:15.119112 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_c5c427e0288c4f879bd65c0f006c0f5d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:15.119957 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c5c427e0288c4f879bd65c0f006c0f5d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250901 01:52:15.123867 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 1f18420261ae41459e1f25b927bd84a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1f18420261ae41459e1f25b927bd84a9 already in progress: creating tablet
I20250901 01:52:15.124166 1332 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44246 (request call id 170) took 1724 ms. Trace:
I20250901 01:52:15.125664 1332 rpcz_store.cc:276] 0901 01:52:13.399763 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:13.399802 (+ 39us) service_pool.cc:225] Handling call
0901 01:52:15.123976 (+1724174us) ts_tablet_manager.cc:1145] Deleting tablet 58c0e7a2e90e48528bc2bc7ec889f608
0901 01:52:15.123989 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:15.124097 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1723980}
W20250901 01:52:15.126281 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 58c0e7a2e90e48528bc2bc7ec889f608 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 58c0e7a2e90e48528bc2bc7ec889f608 already in progress: creating tablet
I20250901 01:52:15.126343 1332 tablet_replica.cc:331] T d945b310fe6344af9cc9dd68f84c4bdf P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:15.126559 1332 raft_consensus.cc:2241] T d945b310fe6344af9cc9dd68f84c4bdf P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:15.126866 1332 raft_consensus.cc:2270] T d945b310fe6344af9cc9dd68f84c4bdf P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
#4 0xaf4f31 in std::__shared_ptr<kudu::tablet::Tablet, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() ../../../include/c++/7.5.0/bits/shared_ptr_base.h:1123:31
#5 0x7fe7989db416 in std::__shared_ptr<kudu::tablet::Tablet, (__gnu_cxx::_Lock_policy)2>::reset() ../../../include/c++/7.5.0/bits/shared_ptr_base.h:1235:9
#6 0x7fe7989bc188 in kudu::tablet::TabletReplica::Stop() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tablet/tablet_replica.cc:372:13
I20250901 01:52:15.128203 1332 ts_tablet_manager.cc:1905] T d945b310fe6344af9cc9dd68f84c4bdf P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:15.128459 1572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Bootstrap starting.
I20250901 01:52:15.130931 1151 tablet_bootstrap.cc:654] T 756cfe84a517459ca015aecf5cd81571 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:15.138062 455 master_service.cc:948] ReplaceTablet: received request to replace tablet 2f28451a3ecc481a958588466f7fec77 from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:15.142639 455 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_f94fe76831a54c6e8c2b13c81c251b14" } modify_external_catalogs: true
I20250901 01:52:15.142849 455 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_f94fe76831a54c6e8c2b13c81c251b14" } modify_external_catalogs: true
W20250901 01:52:15.150624 446 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7cc1efcb6bff4aba8015d2d3b7f4c923 consensus state has no leader
I20250901 01:52:15.156733 446 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_81b532507ef2460d9ff52022c060c07a" } modify_external_catalogs: true
I20250901 01:52:15.157013 446 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_81b532507ef2460d9ff52022c060c07a" } modify_external_catalogs: true
I20250901 01:52:15.174101 1552 raft_consensus.cc:491] T 467c2bf4a8cb4937b6f627bea28d8ddb P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:15.174371 1552 raft_consensus.cc:513] T 467c2bf4a8cb4937b6f627bea28d8ddb P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:15.175558 1552 leader_election.cc:290] T 467c2bf4a8cb4937b6f627bea28d8ddb P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:15.182821 1540 raft_consensus.cc:491] T c27bb23c150f4565a1d0ba8f231cc386 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:15.183079 1540 raft_consensus.cc:513] T c27bb23c150f4565a1d0ba8f231cc386 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:15.183868 1540 leader_election.cc:290] T c27bb23c150f4565a1d0ba8f231cc386 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:15.184659 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c27bb23c150f4565a1d0ba8f231cc386" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:15.185706 922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "467c2bf4a8cb4937b6f627bea28d8ddb" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:15.185770 904 tablet_service.cc:1515] Processing DeleteTablet for tablet dd694cfa09564d3da5d279c0861a4c32 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:12 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:15.185925 922 raft_consensus.cc:2466] T 467c2bf4a8cb4937b6f627bea28d8ddb P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
W20250901 01:52:15.186879 992 leader_election.cc:343] T 2a0d125423a043dcb11e67345599e468 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 2a0d125423a043dcb11e67345599e468
W20250901 01:52:15.187145 859 leader_election.cc:343] T c27bb23c150f4565a1d0ba8f231cc386 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:15.187186 992 leader_election.cc:304] T 467c2bf4a8cb4937b6f627bea28d8ddb P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:15.187495 1552 raft_consensus.cc:2802] T 467c2bf4a8cb4937b6f627bea28d8ddb P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:15.187687 1552 raft_consensus.cc:491] T 467c2bf4a8cb4937b6f627bea28d8ddb P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:15.187824 1552 raft_consensus.cc:3058] T 467c2bf4a8cb4937b6f627bea28d8ddb P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:15.188647 1306 tablet_replica.cc:331] T dd694cfa09564d3da5d279c0861a4c32 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:15.188885 1306 raft_consensus.cc:2241] T dd694cfa09564d3da5d279c0861a4c32 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:15.189056 1306 raft_consensus.cc:2270] T dd694cfa09564d3da5d279c0861a4c32 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:15.189755 1306 ts_tablet_manager.cc:1905] T dd694cfa09564d3da5d279c0861a4c32 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:15.191557 1150 tablet_bootstrap.cc:654] T a7ff996482164776b468cf305989866b P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:15.198669 1306 ts_tablet_manager.cc:1918] T dd694cfa09564d3da5d279c0861a4c32 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:15.198916 1306 log.cc:1199] T dd694cfa09564d3da5d279c0861a4c32 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/dd694cfa09564d3da5d279c0861a4c32
I20250901 01:52:15.199432 1306 ts_tablet_manager.cc:1939] T dd694cfa09564d3da5d279c0861a4c32 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:15.200369 969 heartbeater.cc:344] Connected to a master server at 127.0.104.60:44231
I20250901 01:52:15.200682 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet dd694cfa09564d3da5d279c0861a4c32 (table default.Table_7183dbd577624461af0f861fa4f63d6d [id=a785d19b66e546b5901b6d7e93106161]) successfully deleted
I20250901 01:52:15.203025 1552 raft_consensus.cc:513] T 467c2bf4a8cb4937b6f627bea28d8ddb P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:15.203438 1572 log.cc:826] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Log is configured to *not* fsync() on all Append() calls
I20250901 01:52:15.204591 921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "467c2bf4a8cb4937b6f627bea28d8ddb" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:15.204785 921 raft_consensus.cc:3058] T 467c2bf4a8cb4937b6f627bea28d8ddb P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:15.207643 921 raft_consensus.cc:2466] T 467c2bf4a8cb4937b6f627bea28d8ddb P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:15.207724 1151 tablet_bootstrap.cc:492] T 756cfe84a517459ca015aecf5cd81571 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:15.207891 1151 ts_tablet_manager.cc:1397] T 756cfe84a517459ca015aecf5cd81571 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.205s user 0.001s sys 0.005s
I20250901 01:52:15.208675 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 9d886fa63f8f4996b087997b542e8bb1
I20250901 01:52:15.209206 451 catalog_manager.cc:6652] ReplaceTablet: tablet 9d886fa63f8f4996b087997b542e8bb1 deleted and replaced by tablet eaa521c52a2f4408b59770bd2a05907c
I20250901 01:52:15.209484 451 master_service.cc:948] ReplaceTablet: received request to replace tablet 920925ee9b014e14a79ceb7796ff7828 from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:15.212210 453 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_655d2fe6f5f044d2ac3c542497704343" } modify_external_catalogs: true
I20250901 01:52:15.212419 453 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_655d2fe6f5f044d2ac3c542497704343" } modify_external_catalogs: true
I20250901 01:52:15.212478 902 tablet_service.cc:1515] Processing DeleteTablet for tablet 9d886fa63f8f4996b087997b542e8bb1 with delete_type TABLET_DATA_DELETED (replaced by tablet eaa521c52a2f4408b59770bd2a05907c) from {username='slave'} at 127.0.0.1:52846
W20250901 01:52:15.216758 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 9d886fa63f8f4996b087997b542e8bb1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9d886fa63f8f4996b087997b542e8bb1 already in progress: creating tablet
I20250901 01:52:15.216967 902 tablet_service.cc:1515] Processing DeleteTablet for tablet dd694cfa09564d3da5d279c0861a4c32 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:12 UTC) from {username='slave'} at 127.0.0.1:52846
W20250901 01:52:15.218752 440 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet dd694cfa09564d3da5d279c0861a4c32 because the tablet was not found. No further retry: Not found: Tablet not found: dd694cfa09564d3da5d279c0861a4c32
W20250901 01:52:15.214371 992 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250901 01:52:15.223999 992 leader_election.cc:304] T 467c2bf4a8cb4937b6f627bea28d8ddb P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:15.224247 1552 leader_election.cc:290] T 467c2bf4a8cb4937b6f627bea28d8ddb P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:15.224345 1564 raft_consensus.cc:2802] T 467c2bf4a8cb4937b6f627bea28d8ddb P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:15.224633 1564 raft_consensus.cc:695] T 467c2bf4a8cb4937b6f627bea28d8ddb P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:15.224884 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 9d886fa63f8f4996b087997b542e8bb1 with delete_type TABLET_DATA_DELETED (replaced by tablet eaa521c52a2f4408b59770bd2a05907c) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:15.224951 1564 consensus_queue.cc:237] T 467c2bf4a8cb4937b6f627bea28d8ddb P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:15.235864 1103 heartbeater.cc:344] Connected to a master server at 127.0.104.60:44231
W20250901 01:52:15.238680 992 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250901 01:52:15.239279 1151 raft_consensus.cc:357] T 756cfe84a517459ca015aecf5cd81571 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:15.239567 1151 raft_consensus.cc:383] T 756cfe84a517459ca015aecf5cd81571 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:15.239650 1151 raft_consensus.cc:738] T 756cfe84a517459ca015aecf5cd81571 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:15.240643 1151 consensus_queue.cc:260] T 756cfe84a517459ca015aecf5cd81571 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:15.250584 1151 ts_tablet_manager.cc:1428] T 756cfe84a517459ca015aecf5cd81571 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.025s user 0.000s sys 0.002s
I20250901 01:52:15.259658 1151 tablet_bootstrap.cc:492] T 9496ec6021784800aef62996b0dee593 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:15.266312 1151 tablet_bootstrap.cc:654] T 9496ec6021784800aef62996b0dee593 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:15.268206 449 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet c763d252598745a088344aa0eb5fa3d3
I20250901 01:52:15.268841 449 catalog_manager.cc:6652] ReplaceTablet: tablet c763d252598745a088344aa0eb5fa3d3 deleted and replaced by tablet a8483e7212d94daba7e5401256c33aef
I20250901 01:52:15.270025 902 tablet_service.cc:1515] Processing DeleteTablet for tablet c763d252598745a088344aa0eb5fa3d3 with delete_type TABLET_DATA_DELETED (replaced by tablet a8483e7212d94daba7e5401256c33aef) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:15.270952 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet c763d252598745a088344aa0eb5fa3d3 with delete_type TABLET_DATA_DELETED (replaced by tablet a8483e7212d94daba7e5401256c33aef) from {username='slave'} at 127.0.0.1:44246
W20250901 01:52:15.271968 992 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250901 01:52:15.274302 449 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_54d5a8d2e22840d6a43c4eeb44478d1d" } modify_external_catalogs: true
I20250901 01:52:15.274533 449 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_54d5a8d2e22840d6a43c4eeb44478d1d" } modify_external_catalogs: true
W20250901 01:52:15.277293 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet c763d252598745a088344aa0eb5fa3d3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c763d252598745a088344aa0eb5fa3d3 already in progress: creating tablet
#7 0x7fe79b49e2b7 in kudu::tserver::TSTabletManager::DeleteTablet(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, kudu::tablet::TabletDataState, std::optional<long> const&, kudu::tserver::TabletServerErrorPB_Code*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tserver/ts_tablet_manager.cc:1194:12
#8 0x7fe79b4d0344 in kudu::tserver::DeleteTabletRunnable::Run() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tserver/ts_tablet_manager.cc:1101:36
#9 0x7fe79b4b1da0 in kudu::tserver::TSTabletManager::DeleteTabletAsync(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, kudu::tablet::TabletDataState, std::optional<long> const&, std::function<void (kudu::Status const&, kudu::tserver::TabletServerErrorPB_Code)> const&)::$_16::operator()() const /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tserver/ts_tablet_manager.cc:1121:67
#10 0x7fe79b4b1b2d in std::_Function_handler<void (), kudu::tserver::TSTabletManager::DeleteTabletAsync(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, kudu::tablet::TabletDataState, std::optional<long> const&, std::function<void (kudu::Status const&, kudu::tserver::TabletServerErrorPB_Code)> const&)::$_16>::_M_invoke(std::_Any_data const&) ../../../include/c++/7.5.0/bits/std_function.h:316:2
#11 0xa774dc in std::function<void ()>::operator()() const ../../../include/c++/7.5.0/bits/std_function.h:706:14
I20250901 01:52:15.292279 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_612c3ccdf69d4dbd95aa76bdf404fc54"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:15.292989 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_612c3ccdf69d4dbd95aa76bdf404fc54 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:15.296792 1150 tablet_bootstrap.cc:492] T a7ff996482164776b468cf305989866b P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:15.297039 1150 ts_tablet_manager.cc:1397] T a7ff996482164776b468cf305989866b P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.221s user 0.002s sys 0.013s
I20250901 01:52:15.297984 1150 raft_consensus.cc:357] T a7ff996482164776b468cf305989866b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:15.298229 1150 raft_consensus.cc:383] T a7ff996482164776b468cf305989866b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:15.298316 1150 raft_consensus.cc:738] T a7ff996482164776b468cf305989866b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:15.298744 1150 consensus_queue.cc:260] T a7ff996482164776b468cf305989866b P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:15.302212 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 920925ee9b014e14a79ceb7796ff7828
I20250901 01:52:15.302762 450 catalog_manager.cc:6652] ReplaceTablet: tablet 920925ee9b014e14a79ceb7796ff7828 deleted and replaced by tablet 081e2630e284498aa819f85bf8326315
I20250901 01:52:15.304549 902 tablet_service.cc:1515] Processing DeleteTablet for tablet 920925ee9b014e14a79ceb7796ff7828 with delete_type TABLET_DATA_DELETED (replaced by tablet 081e2630e284498aa819f85bf8326315) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:15.306028 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 920925ee9b014e14a79ceb7796ff7828 with delete_type TABLET_DATA_DELETED (replaced by tablet 081e2630e284498aa819f85bf8326315) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:15.308449 1150 ts_tablet_manager.cc:1428] T a7ff996482164776b468cf305989866b P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.011s user 0.000s sys 0.003s
I20250901 01:52:15.309620 1306 tablet_replica.cc:331] T 920925ee9b014e14a79ceb7796ff7828 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:15.309906 1306 raft_consensus.cc:2241] T 920925ee9b014e14a79ceb7796ff7828 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:15.310181 1306 raft_consensus.cc:2270] T 920925ee9b014e14a79ceb7796ff7828 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:15.310897 1306 ts_tablet_manager.cc:1905] T 920925ee9b014e14a79ceb7796ff7828 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:15.314977 1150 tablet_bootstrap.cc:492] T 154574e17f214cc0ad56e6e6be3af899 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:15.315372 1552 raft_consensus.cc:491] T 0e698923812841b7a5d37d57da103c5c P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:15.315632 1552 raft_consensus.cc:513] T 0e698923812841b7a5d37d57da103c5c P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:15.317437 1552 leader_election.cc:290] T 0e698923812841b7a5d37d57da103c5c P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:15.317957 992 leader_election.cc:343] T 0e698923812841b7a5d37d57da103c5c P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 0e698923812841b7a5d37d57da103c5c
I20250901 01:52:15.318994 451 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_e9b3e507b30344d3a70b1f97bd72b6e0" } new_table_name: "default.Table_1159dbb70cc940df8b509db6e70c350a" modify_external_catalogs: true
#12 0x7fe78bc67b5f in kudu::ThreadPool::DispatchThread() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/threadpool.cc:733:7
I20250901 01:52:15.320879 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_54591f7e2f0f4649a25e6beac813f588"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:15.321560 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_54591f7e2f0f4649a25e6beac813f588 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
#13 0x7fe78bc6ae2b in kudu::ThreadPool::CreateThread()::$_1::operator()() const /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/threadpool.cc:815:48
#14 0x7fe78bc6aca8 in std::_Function_handler<void (), kudu::ThreadPool::CreateThread()::$_1>::_M_invoke(std::_Any_data const&) ../../../include/c++/7.5.0/bits/std_function.h:316:2
#15 0xa774dc in std::function<void ()>::operator()() const ../../../include/c++/7.5.0/bits/std_function.h:706:14
#16 0x7fe78bc46cc8 in kudu::Thread::SuperviseThread(void*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:694:3
#17 0x7fe78f0686da in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76da)
previously allocated by thread T135 (tablet-open [wo) here:
#0 0x94c668 in operator new(unsigned long) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/asan/asan_new_delete.cpp:99
#1 0x7fe79895839f in kudu::tablet::TabletBootstrap::OpenTablet(bool*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tablet/tablet_bootstrap.cc:693:29
#2 0x7fe79895675e in kudu::tablet::TabletBootstrap::RunBootstrap(std::shared_ptr<kudu::tablet::Tablet>*, scoped_refptr<kudu::log::Log>*, kudu::consensus::ConsensusBootstrapInfo*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tablet/tablet_bootstrap.cc:644:3
#3 0x7fe7989550ce in kudu::tablet::TabletBootstrap::Bootstrap(std::shared_ptr<kudu::tablet::Tablet>*, scoped_refptr<kudu::log::Log>*, kudu::consensus::ConsensusBootstrapInfo*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tablet/tablet_bootstrap.cc:579:29
#4 0x7fe7989549ac in kudu::tablet::BootstrapTablet(scoped_refptr<kudu::tablet::TabletMetadata>, kudu::consensus::RaftConfigPB, kudu::clock::Clock*, std::shared_ptr<kudu::MemTracker>, scoped_refptr<kudu::rpc::ResultTracker>, kudu::MetricRegistry*, kudu::FileCache*, scoped_refptr<kudu::tablet::TabletReplica>, scoped_refptr<kudu::log::LogAnchorRegistry>, std::shared_ptr<kudu::tablet::Tablet>*, scoped_refptr<kudu::log::Log>*, kudu::consensus::ConsensusBootstrapInfo*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tablet/tablet_bootstrap.cc:521:3
#5 0x7fe79b4998b2 in kudu::tserver::TSTabletManager::OpenTablet(scoped_refptr<kudu::tablet::TabletReplica> const&, scoped_refptr<kudu::tserver::TransitionInProgressDeleter> const&, std::atomic<int>*, std::atomic<int>*, kudu::Timer*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tserver/ts_tablet_manager.cc:1406:9
#6 0x7fe79b4b067a in kudu::tserver::TSTabletManager::CreateNewTablet(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, kudu::Partition const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, kudu::Schema const&, kudu::PartitionSchema const&, kudu::consensus::RaftConfigPB, std::optional<kudu::TableExtraConfigPB>, std::optional<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::optional<kudu::TableTypePB>, scoped_refptr<kudu::tablet::TabletReplica>*)::$_11::operator()() const /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tserver/ts_tablet_manager.cc:703:3
#7 0x7fe79b4b03bd in std::_Function_handler<void (), kudu::tserver::TSTabletManager::CreateNewTablet(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, kudu::Partition const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, kudu::Schema const&, kudu::PartitionSchema const&, kudu::consensus::RaftConfigPB, std::optional<kudu::TableExtraConfigPB>, std::optional<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::optional<kudu::TableTypePB>, scoped_refptr<kudu::tablet::TabletReplica>*)::$_11>::_M_invoke(std::_Any_data const&) ../../../include/c++/7.5.0/bits/std_function.h:316:2
#8 0xa774dc in std::function<void ()>::operator()() const ../../../include/c++/7.5.0/bits/std_function.h:706:14
#9 0x7fe78bc67b5f in kudu::ThreadPool::DispatchThread() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/threadpool.cc:733:7
#10 0x7fe78bc6ae2b in kudu::ThreadPool::CreateThread()::$_1::operator()() const /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/threadpool.cc:815:48
#11 0x7fe78bc6aca8 in std::_Function_handler<void (), kudu::ThreadPool::CreateThread()::$_1>::_M_invoke(std::_Any_data const&) ../../../include/c++/7.5.0/bits/std_function.h:316:2
#12 0xa774dc in std::function<void ()>::operator()() const ../../../include/c++/7.5.0/bits/std_function.h:706:14
#13 0x7fe78bc46cc8 in kudu::Thread::SuperviseThread(void*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:694:3
#14 0x7fe78f0686da in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76da)
Thread T56 (rpc worker-756) created by T0 here:
I20250901 01:52:15.328243 1151 tablet_bootstrap.cc:492] T 9496ec6021784800aef62996b0dee593 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:15.329300 1151 ts_tablet_manager.cc:1397] T 9496ec6021784800aef62996b0dee593 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.071s user 0.002s sys 0.009s
I20250901 01:52:15.329838 1150 tablet_bootstrap.cc:654] T 154574e17f214cc0ad56e6e6be3af899 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:15.331435 994 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250901 01:52:15.336987 454 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 629db80a8ab444e582628578155638fd consensus state has no leader
I20250901 01:52:15.335389 1151 raft_consensus.cc:357] T 9496ec6021784800aef62996b0dee593 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:15.337682 1151 raft_consensus.cc:383] T 9496ec6021784800aef62996b0dee593 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:15.338472 1151 raft_consensus.cc:738] T 9496ec6021784800aef62996b0dee593 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:15.343192 1151 consensus_queue.cc:260] T 9496ec6021784800aef62996b0dee593 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:15.346762 902 tablet_service.cc:1515] Processing DeleteTablet for tablet a7ff996482164776b468cf305989866b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:13 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:15.349460 1151 ts_tablet_manager.cc:1428] T 9496ec6021784800aef62996b0dee593 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.019s user 0.004s sys 0.000s
I20250901 01:52:15.352571 1151 tablet_bootstrap.cc:492] T 84fe314502444bd88b038b87eead9cb5 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:15.353093 1103 heartbeater.cc:344] Connected to a master server at 127.0.104.60:44231
W20250901 01:52:15.366096 441 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 7cc1efcb6bff4aba8015d2d3b7f4c923 (table default.Table_5610feff8a344dc9be4bc1f0ad8ddd51 [id=78b00856a9b747c19f390b4fdae8ddcf], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet 7cc1efcb6bff4aba8015d2d3b7f4c923 consensus state has no leader
I20250901 01:52:15.368112 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_f4026aef620b4786b9012444560a1a76"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:15.369158 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f4026aef620b4786b9012444560a1a76 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:15.376507 1332 ts_tablet_manager.cc:1918] T d945b310fe6344af9cc9dd68f84c4bdf P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:15.379846 1306 ts_tablet_manager.cc:1918] T 920925ee9b014e14a79ceb7796ff7828 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:15.390009 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_0ff9dd032f74401b967bb5e4ed86d3ec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:15.390734 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0ff9dd032f74401b967bb5e4ed86d3ec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:15.386842 1151 tablet_bootstrap.cc:654] T 84fe314502444bd88b038b87eead9cb5 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:15.394025 1332 log.cc:1199] T d945b310fe6344af9cc9dd68f84c4bdf P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/d945b310fe6344af9cc9dd68f84c4bdf
I20250901 01:52:15.394167 1552 raft_consensus.cc:491] T d6a4040747fa4d588756506ee122d2dc P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:15.394519 1552 raft_consensus.cc:513] T d6a4040747fa4d588756506ee122d2dc P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:15.395355 1552 leader_election.cc:290] T d6a4040747fa4d588756506ee122d2dc P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:15.396471 992 leader_election.cc:343] T d6a4040747fa4d588756506ee122d2dc P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: d6a4040747fa4d588756506ee122d2dc
I20250901 01:52:15.395958 1332 ts_tablet_manager.cc:1939] T d945b310fe6344af9cc9dd68f84c4bdf P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:15.397356 1332 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44246 (request call id 171) took 1997 ms. Trace:
I20250901 01:52:15.408479 446 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet b73e362321284df1b18ef755dfe930a5
I20250901 01:52:15.409368 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_db4aaba6e67e436dab76b895e3443bdd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:15.409989 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_db4aaba6e67e436dab76b895e3443bdd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:15.412755 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet b73e362321284df1b18ef755dfe930a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:15 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:15.424716 1151 tablet_bootstrap.cc:492] T 84fe314502444bd88b038b87eead9cb5 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:15.426671 1151 ts_tablet_manager.cc:1397] T 84fe314502444bd88b038b87eead9cb5 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.075s user 0.006s sys 0.006s
I20250901 01:52:15.428236 1151 raft_consensus.cc:357] T 84fe314502444bd88b038b87eead9cb5 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:15.428694 1151 raft_consensus.cc:383] T 84fe314502444bd88b038b87eead9cb5 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:15.428902 1151 raft_consensus.cc:738] T 84fe314502444bd88b038b87eead9cb5 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:15.426918 1332 rpcz_store.cc:276] 0901 01:52:13.400129 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:13.400167 (+ 38us) service_pool.cc:225] Handling call
0901 01:52:15.126258 (+1726091us) ts_tablet_manager.cc:1145] Deleting tablet d945b310fe6344af9cc9dd68f84c4bdf
0901 01:52:15.126270 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:15.220545 (+ 94275us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:15.376329 (+155784us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:15.397308 (+ 20979us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1725909}
I20250901 01:52:15.430460 441 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet d945b310fe6344af9cc9dd68f84c4bdf (table default.Table_e2ea7aa90231448888c6cfead7e3d9c1 [id=c264ed0bf1d84bc9bd5c2bb0d1e8a45b]) successfully deleted
I20250901 01:52:15.429554 1151 consensus_queue.cc:260] T 84fe314502444bd88b038b87eead9cb5 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:15.435452 1151 ts_tablet_manager.cc:1428] T 84fe314502444bd88b038b87eead9cb5 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20250901 01:52:15.439239 1151 tablet_bootstrap.cc:492] T 7cc1efcb6bff4aba8015d2d3b7f4c923 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:15.381160 921 raft_consensus.cc:1273] T 85322a864d154f299f2d670fdd2a7bfb P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Refusing update from remote peer fd27ab6665304aaea53adbd102dd4f58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250901 01:52:15.383183 1150 tablet_bootstrap.cc:492] T 154574e17f214cc0ad56e6e6be3af899 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:15.444339 1150 ts_tablet_manager.cc:1397] T 154574e17f214cc0ad56e6e6be3af899 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.130s user 0.003s sys 0.007s
I20250901 01:52:15.443521 1332 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44246 (request call id 173) took 2042 ms. Trace:
I20250901 01:52:15.444618 1306 log.cc:1199] T 920925ee9b014e14a79ceb7796ff7828 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/920925ee9b014e14a79ceb7796ff7828
I20250901 01:52:15.445914 1552 consensus_queue.cc:1035] T 85322a864d154f299f2d670fdd2a7bfb P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:15.445250 1150 raft_consensus.cc:357] T 154574e17f214cc0ad56e6e6be3af899 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:15.446882 1150 raft_consensus.cc:383] T 154574e17f214cc0ad56e6e6be3af899 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:15.447000 1150 raft_consensus.cc:738] T 154574e17f214cc0ad56e6e6be3af899 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:15.447827 1150 consensus_queue.cc:260] T 154574e17f214cc0ad56e6e6be3af899 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:15.448463 1150 ts_tablet_manager.cc:1428] T 154574e17f214cc0ad56e6e6be3af899 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20250901 01:52:15.412010 902 tablet_service.cc:1515] Processing DeleteTablet for tablet b73e362321284df1b18ef755dfe930a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:15 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:15.452706 1150 tablet_bootstrap.cc:492] T 6c05eff658924ca4a9a1e7a37f917d67 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:15.454890 1150 tablet_bootstrap.cc:654] T 6c05eff658924ca4a9a1e7a37f917d67 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:15.459493 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_adb18e9d4ac64721b777cd20a2cd0800"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
#0 0x8920f0 in pthread_create /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/asan/asan_interceptors.cpp:214
W20250901 01:52:15.461258 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_adb18e9d4ac64721b777cd20a2cd0800 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:15.445672 1332 rpcz_store.cc:276] 0901 01:52:13.401083 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:13.401131 (+ 48us) service_pool.cc:225] Handling call
0901 01:52:15.431358 (+2030227us) ts_tablet_manager.cc:1145] Deleting tablet 60285f2ba037433999f5455fb9cc7acd
0901 01:52:15.431374 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:15.443121 (+ 11747us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2029667}
W20250901 01:52:15.463690 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 60285f2ba037433999f5455fb9cc7acd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 60285f2ba037433999f5455fb9cc7acd already in progress: creating tablet
I20250901 01:52:15.468766 1332 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44246 (request call id 183) took 1844 ms. Trace:
#1 0x7fe78bc459c0 in kudu::Thread::StartThread(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::function<void ()>, unsigned long, scoped_refptr<kudu::Thread>*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:643:15
#2 0xa63f96 in kudu::Thread::Create(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::function<void ()>, scoped_refptr<kudu::Thread>*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.h:147:12
I20250901 01:52:15.472463 1306 ts_tablet_manager.cc:1939] T 920925ee9b014e14a79ceb7796ff7828 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
#3 0x7fe7929d12e2 in kudu::rpc::ServicePool::Init(int) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/service_pool.cc:92:5
I20250901 01:52:15.475488 1150 tablet_bootstrap.cc:492] T 6c05eff658924ca4a9a1e7a37f917d67 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:15.475814 1150 ts_tablet_manager.cc:1397] T 6c05eff658924ca4a9a1e7a37f917d67 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.027s user 0.005s sys 0.004s
I20250901 01:52:15.479449 1332 rpcz_store.cc:276] 0901 01:52:13.624722 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:13.624948 (+ 226us) service_pool.cc:225] Handling call
0901 01:52:15.465140 (+1840192us) ts_tablet_manager.cc:1145] Deleting tablet 31390f38258c465cb1798615a1fcc790
0901 01:52:15.465154 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:15.468697 (+ 3543us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1839512}
W20250901 01:52:15.481798 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 31390f38258c465cb1798615a1fcc790 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 31390f38258c465cb1798615a1fcc790 already in progress: creating tablet
I20250901 01:52:15.482725 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 920925ee9b014e14a79ceb7796ff7828 (table default.Table_537be4a82c0348f6952b911b167d02ad [id=8199118617cb4707bc3ae9b1363750f7]) successfully deleted
I20250901 01:52:15.483943 1306 tablet_replica.cc:331] T a7ff996482164776b468cf305989866b P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:15.484200 1306 raft_consensus.cc:2241] T a7ff996482164776b468cf305989866b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:15.484390 1306 raft_consensus.cc:2270] T a7ff996482164776b468cf305989866b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:15.484632 1332 tablet_replica.cc:331] T 4d333b670df84f308f0cbf6da636b768 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:15.490738 1150 raft_consensus.cc:357] T 6c05eff658924ca4a9a1e7a37f917d67 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:15.491722 1150 raft_consensus.cc:383] T 6c05eff658924ca4a9a1e7a37f917d67 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:15.474222 1151 tablet_bootstrap.cc:654] T 7cc1efcb6bff4aba8015d2d3b7f4c923 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:15.491883 1150 raft_consensus.cc:738] T 6c05eff658924ca4a9a1e7a37f917d67 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:15.492458 1150 consensus_queue.cc:260] T 6c05eff658924ca4a9a1e7a37f917d67 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:15.493471 1150 ts_tablet_manager.cc:1428] T 6c05eff658924ca4a9a1e7a37f917d67 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.018s user 0.002s sys 0.001s
I20250901 01:52:15.493991 1150 tablet_bootstrap.cc:492] T 60285f2ba037433999f5455fb9cc7acd P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:15.496795 1589 tablet.cc:1722] T 85322a864d154f299f2d670fdd2a7bfb P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250901 01:52:15.501907 1150 tablet_bootstrap.cc:654] T 60285f2ba037433999f5455fb9cc7acd P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:15.513576 1293 raft_consensus.cc:491] T 154574e17f214cc0ad56e6e6be3af899 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:15.513782 1293 raft_consensus.cc:513] T 154574e17f214cc0ad56e6e6be3af899 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:15.514477 1293 leader_election.cc:290] T 154574e17f214cc0ad56e6e6be3af899 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:15.516449 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "154574e17f214cc0ad56e6e6be3af899" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:15.517246 859 leader_election.cc:343] T 154574e17f214cc0ad56e6e6be3af899 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:15.518324 1590 tablet.cc:1722] T 85322a864d154f299f2d670fdd2a7bfb P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250901 01:52:15.526543 1151 tablet_bootstrap.cc:492] T 7cc1efcb6bff4aba8015d2d3b7f4c923 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:15.526736 1151 ts_tablet_manager.cc:1397] T 7cc1efcb6bff4aba8015d2d3b7f4c923 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.090s user 0.004s sys 0.000s
I20250901 01:52:15.527606 1151 raft_consensus.cc:357] T 7cc1efcb6bff4aba8015d2d3b7f4c923 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:15.527807 1151 raft_consensus.cc:383] T 7cc1efcb6bff4aba8015d2d3b7f4c923 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:15.527904 1151 raft_consensus.cc:738] T 7cc1efcb6bff4aba8015d2d3b7f4c923 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:15.528122 1151 consensus_queue.cc:260] T 7cc1efcb6bff4aba8015d2d3b7f4c923 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:15.530668 1332 raft_consensus.cc:2241] T 4d333b670df84f308f0cbf6da636b768 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:15.531899 1151 ts_tablet_manager.cc:1428] T 7cc1efcb6bff4aba8015d2d3b7f4c923 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250901 01:52:15.533179 1151 tablet_bootstrap.cc:492] T 6c15cd14831f422eb615038603c79ed0 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:15.535234 1332 raft_consensus.cc:2270] T 4d333b670df84f308f0cbf6da636b768 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:15.535953 1151 tablet_bootstrap.cc:654] T 6c15cd14831f422eb615038603c79ed0 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:15.541204 1150 tablet_bootstrap.cc:492] T 60285f2ba037433999f5455fb9cc7acd P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
#4 0x7fe794e79830 in kudu::RpcServer::RegisterService(std::unique_ptr<kudu::rpc::ServiceIf, std::default_delete<kudu::rpc::ServiceIf> >) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/server/rpc_server.cc:238:3
I20250901 01:52:15.545179 1332 ts_tablet_manager.cc:1905] T 4d333b670df84f308f0cbf6da636b768 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:15.552649 1590 tablet.cc:1717] T 85322a864d154f299f2d670fdd2a7bfb P fd27ab6665304aaea53adbd102dd4f58: Skipping requested alter to schema version 4, tablet already version 4
I20250901 01:52:15.552894 1293 raft_consensus.cc:491] T fcb7b5b7d63d4f0b906fea97cb548de2 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:15.557770 1293 raft_consensus.cc:513] T fcb7b5b7d63d4f0b906fea97cb548de2 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:15.557770 1150 ts_tablet_manager.cc:1397] T 60285f2ba037433999f5455fb9cc7acd P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.064s user 0.000s sys 0.007s
I20250901 01:52:15.558593 1293 leader_election.cc:290] T fcb7b5b7d63d4f0b906fea97cb548de2 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:15.553285 1589 tablet.cc:1717] T 85322a864d154f299f2d670fdd2a7bfb P 00c282f779c1477d920f24dc0dc16628: Skipping requested alter to schema version 4, tablet already version 4
I20250901 01:52:15.558851 1150 raft_consensus.cc:357] T 60285f2ba037433999f5455fb9cc7acd P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:15.559013 1150 raft_consensus.cc:383] T 60285f2ba037433999f5455fb9cc7acd P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:15.559111 1150 raft_consensus.cc:738] T 60285f2ba037433999f5455fb9cc7acd P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:15.560444 1306 ts_tablet_manager.cc:1905] T a7ff996482164776b468cf305989866b P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:15.560506 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcb7b5b7d63d4f0b906fea97cb548de2" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:15.561197 859 leader_election.cc:343] T fcb7b5b7d63d4f0b906fea97cb548de2 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:15.562060 1150 consensus_queue.cc:260] T 60285f2ba037433999f5455fb9cc7acd P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:15.562711 1150 ts_tablet_manager.cc:1428] T 60285f2ba037433999f5455fb9cc7acd P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20250901 01:52:15.564796 450 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 85322a864d154f299f2d670fdd2a7bfb (table default.Table_68f31d33b83544cea7b959bc7c41f94b [id=1d8f34cc14cb4e90a562ff4424b8577a]). Expected version 4 got 0
I20250901 01:52:15.564924 1150 tablet_bootstrap.cc:492] T 4d27fb61943c45d4aca67689cc3a9ff3 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:15.566177 902 tablet_service.cc:1515] Processing DeleteTablet for tablet 154574e17f214cc0ad56e6e6be3af899 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:13 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:15.567199 450 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_30943b75394441a5a55b195055d1e0cc" } modify_external_catalogs: true
I20250901 01:52:15.567476 450 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_30943b75394441a5a55b195055d1e0cc" } modify_external_catalogs: true
I20250901 01:52:15.573143 1150 tablet_bootstrap.cc:654] T 4d27fb61943c45d4aca67689cc3a9ff3 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:15.582821 1332 ts_tablet_manager.cc:1918] T 4d333b670df84f308f0cbf6da636b768 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:15.585510 1332 log.cc:1199] T 4d333b670df84f308f0cbf6da636b768 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/4d333b670df84f308f0cbf6da636b768
I20250901 01:52:15.586277 1332 ts_tablet_manager.cc:1939] T 4d333b670df84f308f0cbf6da636b768 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:15.587253 1332 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44246 (request call id 184) took 1962 ms. Trace:
I20250901 01:52:15.587949 1332 rpcz_store.cc:276] 0901 01:52:13.624890 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:13.646114 (+ 21224us) service_pool.cc:225] Handling call
0901 01:52:15.483416 (+1837302us) ts_tablet_manager.cc:1145] Deleting tablet 4d333b670df84f308f0cbf6da636b768
0901 01:52:15.483594 (+ 178us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:15.569289 (+ 85695us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:15.582685 (+ 13396us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:15.587199 (+ 4514us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1836776}
I20250901 01:52:15.588889 441 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 4d333b670df84f308f0cbf6da636b768 (table default.Table_84b7ac25a6b94ac0bbf0865c39abac94 [id=c4ac57f676cd4799a98ad922c26ef14b]) successfully deleted
I20250901 01:52:15.589538 1332 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44246 (request call id 185) took 1827 ms. Trace:
I20250901 01:52:15.590058 1332 rpcz_store.cc:276] 0901 01:52:13.761930 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:13.761994 (+ 64us) service_pool.cc:225] Handling call
0901 01:52:15.589326 (+1827332us) ts_tablet_manager.cc:1145] Deleting tablet 533d14f5903e49599a34ea353ffd8c00
0901 01:52:15.589339 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:15.589487 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1827089}
W20250901 01:52:15.590775 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 533d14f5903e49599a34ea353ffd8c00 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 533d14f5903e49599a34ea353ffd8c00 already in progress: creating tablet
I20250901 01:52:15.591233 1332 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44246 (request call id 186) took 1723 ms. Trace:
I20250901 01:52:15.591625 1332 rpcz_store.cc:276] 0901 01:52:13.868010 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:13.868089 (+ 79us) service_pool.cc:225] Handling call
0901 01:52:15.591100 (+1723011us) ts_tablet_manager.cc:1145] Deleting tablet a7ff996482164776b468cf305989866b
0901 01:52:15.591109 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:15.591190 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1722709}
W20250901 01:52:15.592213 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet a7ff996482164776b468cf305989866b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a7ff996482164776b468cf305989866b already in progress: creating tablet
I20250901 01:52:15.592680 1332 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44246 (request call id 187) took 1713 ms. Trace:
I20250901 01:52:15.593145 1332 rpcz_store.cc:276] 0901 01:52:13.879009 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:13.879088 (+ 79us) service_pool.cc:225] Handling call
0901 01:52:15.592502 (+1713414us) ts_tablet_manager.cc:1145] Deleting tablet c9a4c80bec1d41458974a0c449aecaf7
0901 01:52:15.592514 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:15.592618 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1713015}
I20250901 01:52:15.600023 1293 raft_consensus.cc:491] T a59ac57aebe24860aa02f53ce0485910 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:15.600193 1293 raft_consensus.cc:513] T a59ac57aebe24860aa02f53ce0485910 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:15.600906 1293 leader_election.cc:290] T a59ac57aebe24860aa02f53ce0485910 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:15.601521 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a59ac57aebe24860aa02f53ce0485910" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:15.602102 859 leader_election.cc:343] T a59ac57aebe24860aa02f53ce0485910 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:15.602579 1332 tablet_replica.cc:331] T 4cdd369214ef4f5ebabddbf28b2ae3f7 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:15.603449 1332 raft_consensus.cc:2241] T 4cdd369214ef4f5ebabddbf28b2ae3f7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:15.604300 1332 raft_consensus.cc:2270] T 4cdd369214ef4f5ebabddbf28b2ae3f7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:15.605567 1150 tablet_bootstrap.cc:492] T 4d27fb61943c45d4aca67689cc3a9ff3 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:15.605808 1150 ts_tablet_manager.cc:1397] T 4d27fb61943c45d4aca67689cc3a9ff3 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.041s user 0.001s sys 0.006s
I20250901 01:52:15.606940 1150 raft_consensus.cc:357] T 4d27fb61943c45d4aca67689cc3a9ff3 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:15.607482 448 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_6cf5e8ab687249788f151b78ba59351a" } new_table_name: "default.Table_c857a6bd1be2434d861d57fea5bf4c1a" modify_external_catalogs: true
#5 0x7fe794e9ac47 in kudu::server::ServerBase::RegisterService(std::unique_ptr<kudu::rpc::ServiceIf, std::default_delete<kudu::rpc::ServiceIf> >) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/server/server_base.cc:1184:23
I20250901 01:52:15.612339 1150 raft_consensus.cc:383] T 4d27fb61943c45d4aca67689cc3a9ff3 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:15.612483 1150 raft_consensus.cc:738] T 4d27fb61943c45d4aca67689cc3a9ff3 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:15.612780 1150 consensus_queue.cc:260] T 4d27fb61943c45d4aca67689cc3a9ff3 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:15.613278 1150 ts_tablet_manager.cc:1428] T 4d27fb61943c45d4aca67689cc3a9ff3 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20250901 01:52:15.613632 1150 tablet_bootstrap.cc:492] T b633b461dbf34e13bc6e1cc08e6d20c1 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:15.615476 1150 tablet_bootstrap.cc:654] T b633b461dbf34e13bc6e1cc08e6d20c1 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:15.616700 1332 ts_tablet_manager.cc:1905] T 4cdd369214ef4f5ebabddbf28b2ae3f7 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:15.618834 969 heartbeater.cc:344] Connected to a master server at 127.0.104.60:44231
I20250901 01:52:15.621328 1151 tablet_bootstrap.cc:492] T 6c15cd14831f422eb615038603c79ed0 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:15.621866 1151 ts_tablet_manager.cc:1397] T 6c15cd14831f422eb615038603c79ed0 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.089s user 0.002s sys 0.003s
I20250901 01:52:15.623515 1151 raft_consensus.cc:357] T 6c15cd14831f422eb615038603c79ed0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:15.624257 1150 tablet_bootstrap.cc:492] T b633b461dbf34e13bc6e1cc08e6d20c1 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:15.624544 446 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_300cde3d9237488ba65cab60dd8f8721" } new_table_name: "default.Table_a5d705c4e57a44b8aca8eb68c601e4ae" modify_external_catalogs: true
I20250901 01:52:15.627856 1332 ts_tablet_manager.cc:1918] T 4cdd369214ef4f5ebabddbf28b2ae3f7 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:15.631322 1150 ts_tablet_manager.cc:1397] T b633b461dbf34e13bc6e1cc08e6d20c1 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.002s
I20250901 01:52:15.632359 1150 raft_consensus.cc:357] T b633b461dbf34e13bc6e1cc08e6d20c1 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:15.633666 1150 raft_consensus.cc:383] T b633b461dbf34e13bc6e1cc08e6d20c1 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:15.633939 1150 raft_consensus.cc:738] T b633b461dbf34e13bc6e1cc08e6d20c1 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:15.634498 1150 consensus_queue.cc:260] T b633b461dbf34e13bc6e1cc08e6d20c1 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:15.634644 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_4f2be95a1b824f06ba16e56abcdebc8a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:15.635267 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4f2be95a1b824f06ba16e56abcdebc8a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:15.635403 1150 ts_tablet_manager.cc:1428] T b633b461dbf34e13bc6e1cc08e6d20c1 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250901 01:52:15.636233 1150 tablet_bootstrap.cc:492] T b541a6e14f114ab29a864cca11806316 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:15.642760 1150 tablet_bootstrap.cc:654] T b541a6e14f114ab29a864cca11806316 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:15.644198 1332 log.cc:1199] T 4cdd369214ef4f5ebabddbf28b2ae3f7 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/4cdd369214ef4f5ebabddbf28b2ae3f7
I20250901 01:52:15.644905 1332 ts_tablet_manager.cc:1939] T 4cdd369214ef4f5ebabddbf28b2ae3f7 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:15.645959 1332 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44246 (request call id 194) took 1650 ms. Trace:
I20250901 01:52:15.646802 1332 rpcz_store.cc:276] 0901 01:52:13.995392 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:13.998155 (+ 2763us) service_pool.cc:225] Handling call
0901 01:52:15.602120 (+1603965us) ts_tablet_manager.cc:1145] Deleting tablet 4cdd369214ef4f5ebabddbf28b2ae3f7
0901 01:52:15.602130 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:15.624758 (+ 22628us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:15.627730 (+ 2972us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:15.645921 (+ 18191us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1603698}
I20250901 01:52:15.647735 441 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 4cdd369214ef4f5ebabddbf28b2ae3f7 (table default.Table_cf1c9f5fc5104126b163225918707284 [id=00ae7e46c65d4a56b1ffa193b853421a]) successfully deleted
I20250901 01:52:15.623854 1151 raft_consensus.cc:383] T 6c15cd14831f422eb615038603c79ed0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:15.648080 1151 raft_consensus.cc:738] T 6c15cd14831f422eb615038603c79ed0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:15.648455 1151 consensus_queue.cc:260] T 6c15cd14831f422eb615038603c79ed0 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:15.649154 1151 ts_tablet_manager.cc:1428] T 6c15cd14831f422eb615038603c79ed0 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.027s user 0.000s sys 0.002s
I20250901 01:52:15.649622 1151 tablet_bootstrap.cc:492] T 31390f38258c465cb1798615a1fcc790 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:15.652122 1151 tablet_bootstrap.cc:654] T 31390f38258c465cb1798615a1fcc790 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:15.652360 1332 tablet_replica.cc:331] T f0affc7a78634607852a48458412761d P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:15.660856 1332 raft_consensus.cc:2241] T f0affc7a78634607852a48458412761d P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:15.661114 1332 pending_rounds.cc:70] T f0affc7a78634607852a48458412761d P fd27ab6665304aaea53adbd102dd4f58: Trying to abort 1 pending ops.
I20250901 01:52:15.661264 1332 pending_rounds.cc:77] T f0affc7a78634607852a48458412761d P fd27ab6665304aaea53adbd102dd4f58: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7195408521867636736 op_type: NO_OP noop_request { }
I20250901 01:52:15.661448 1332 raft_consensus.cc:2887] T f0affc7a78634607852a48458412761d P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250901 01:52:15.661660 1332 raft_consensus.cc:2270] T f0affc7a78634607852a48458412761d P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250901 01:52:15.664575 994 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250901 01:52:15.675974 1332 ts_tablet_manager.cc:1905] T f0affc7a78634607852a48458412761d P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:15.680097 1150 tablet_bootstrap.cc:492] T b541a6e14f114ab29a864cca11806316 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:15.680305 1150 ts_tablet_manager.cc:1397] T b541a6e14f114ab29a864cca11806316 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.044s user 0.002s sys 0.002s
I20250901 01:52:15.681269 1150 raft_consensus.cc:357] T b541a6e14f114ab29a864cca11806316 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:15.681521 1150 raft_consensus.cc:383] T b541a6e14f114ab29a864cca11806316 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:15.681612 1150 raft_consensus.cc:738] T b541a6e14f114ab29a864cca11806316 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:15.681905 1150 consensus_queue.cc:260] T b541a6e14f114ab29a864cca11806316 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:15.682425 1150 ts_tablet_manager.cc:1428] T b541a6e14f114ab29a864cca11806316 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250901 01:52:15.682745 1150 tablet_bootstrap.cc:492] T 12c04e0c1f804179ae23509cdb24d395 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:15.684530 1150 tablet_bootstrap.cc:654] T 12c04e0c1f804179ae23509cdb24d395 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:15.686916 1552 raft_consensus.cc:491] T 9496ec6021784800aef62996b0dee593 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:15.687148 1552 raft_consensus.cc:513] T 9496ec6021784800aef62996b0dee593 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:15.688118 1552 leader_election.cc:290] T 9496ec6021784800aef62996b0dee593 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:15.690042 1150 tablet_bootstrap.cc:492] T 12c04e0c1f804179ae23509cdb24d395 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:15.690199 1150 ts_tablet_manager.cc:1397] T 12c04e0c1f804179ae23509cdb24d395 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.003s
I20250901 01:52:15.691098 1150 raft_consensus.cc:357] T 12c04e0c1f804179ae23509cdb24d395 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:15.691380 1150 raft_consensus.cc:383] T 12c04e0c1f804179ae23509cdb24d395 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:15.691478 1150 raft_consensus.cc:738] T 12c04e0c1f804179ae23509cdb24d395 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:15.694855 452 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 85322a864d154f299f2d670fdd2a7bfb (table default.Table_68f31d33b83544cea7b959bc7c41f94b [id=1d8f34cc14cb4e90a562ff4424b8577a]). Expected version 4 got 0
I20250901 01:52:15.695185 452 catalog_manager.cc:5582] T 4fd0e4f9d86c4362af96ce9964b72328 P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } }
I20250901 01:52:15.695544 452 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 4fd0e4f9d86c4362af96ce9964b72328 (table default.Table_2d3d2227e6a24bd2a86f457e553e698b [id=f48fd53522234290b6c0373fd7b02570]). Expected version 2 got 0
W20250901 01:52:15.696489 451 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a774cc6ae3924d299127fd9ffb01ca57 consensus state has no leader
I20250901 01:52:15.698452 1150 consensus_queue.cc:260] T 12c04e0c1f804179ae23509cdb24d395 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:15.699036 1150 ts_tablet_manager.cc:1428] T 12c04e0c1f804179ae23509cdb24d395 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20250901 01:52:15.699369 1150 tablet_bootstrap.cc:492] T 824229183e024feaa975c7f0387c9c1a P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
W20250901 01:52:15.700678 991 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250901 01:52:15.702271 1026 tablet_service.cc:1468] Processing CreateTablet for tablet af2c4bcbb00741f789edee109252d8cb (DEFAULT_TABLE table=default.Table_6f470109ee524f10a48d1080ec15a199 [id=dc57478deb384d52ae97315fcdce074a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:15.702786 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af2c4bcbb00741f789edee109252d8cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:15.703194 1552 raft_consensus.cc:491] T 7cc1efcb6bff4aba8015d2d3b7f4c923 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:15.703337 1552 raft_consensus.cc:513] T 7cc1efcb6bff4aba8015d2d3b7f4c923 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:15.704102 1552 leader_election.cc:290] T 7cc1efcb6bff4aba8015d2d3b7f4c923 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:15.704770 902 tablet_service.cc:1468] Processing CreateTablet for tablet af2c4bcbb00741f789edee109252d8cb (DEFAULT_TABLE table=default.Table_6f470109ee524f10a48d1080ec15a199 [id=dc57478deb384d52ae97315fcdce074a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:15.705224 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af2c4bcbb00741f789edee109252d8cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:15.710055 904 tablet_service.cc:1468] Processing CreateTablet for tablet c49d1143d54b4c319739488906b0e271 (DEFAULT_TABLE table=default.Table_583e53c2e86544e9be68eb27ab6068a3 [id=f376eaf65bb549429aa71b03fca81647]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:15.710458 904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c49d1143d54b4c319739488906b0e271. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:15.712764 899 tablet_service.cc:1468] Processing CreateTablet for tablet e1d18e999b5e42b1873631e0f1445081 (DEFAULT_TABLE table=default.Table_fe014107d0ee4fb8bdfca5356093854b [id=033c76e0598342c495ce2be42a0df5dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:15.713115 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1d18e999b5e42b1873631e0f1445081. 1 dirs total, 0 dirs full, 0 dirs failed
#6 0x7fe79b3cb5b3 in kudu::tserver::TabletServer::Start() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tserver/tablet_server.cc:157:3
I20250901 01:52:15.716790 894 tablet_service.cc:1468] Processing CreateTablet for tablet b7b802f976f3471faaaecc964ec87e48 (DEFAULT_TABLE table=default.Table_3552b4561f10466b9b77f797ca003e53 [id=ec380a4600f34080bc1e343130537377]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:15.716869 451 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_3b78b64e932d45458e89edf72fc45a6d" } new_table_name: "default.Table_bcd502e1360b48b1953fa97eadce7006" modify_external_catalogs: true
I20250901 01:52:15.717182 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7b802f976f3471faaaecc964ec87e48. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:15.720665 896 tablet_service.cc:1468] Processing CreateTablet for tablet 579b9cdf819f4609a532e96bec093fd5 (DEFAULT_TABLE table=default.Table_b0e916c76a7a4766aa0887cc8691488f [id=cf61137a852c48748a29f833f1860da7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:15.721021 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 579b9cdf819f4609a532e96bec093fd5. 1 dirs total, 0 dirs full, 0 dirs failed
W20250901 01:52:15.721220 859 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250901 01:52:15.723631 448 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b2e8b2cd10154de08ddd30c214592a4a consensus state has no leader
I20250901 01:52:15.723927 448 master_service.cc:948] ReplaceTablet: received request to replace tablet c763d252598745a088344aa0eb5fa3d3 from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:15.724318 897 tablet_service.cc:1468] Processing CreateTablet for tablet 7b16ca1daf394a6da65a4af8da82ec33 (DEFAULT_TABLE table=default.Table_9d8c1fb24bc24631a2ba2b34a395f24a [id=2d12d6d6275e499ba3928e76bfd3cfc7]), partition=RANGE (key) PARTITION UNBOUNDED
#7 0x7fe79b3daa20 in kudu::tserver::RunTabletServer() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tserver/tablet_server_runner.cc:86:3
I20250901 01:52:15.724720 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b16ca1daf394a6da65a4af8da82ec33. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:15.728138 898 tablet_service.cc:1468] Processing CreateTablet for tablet 0cb3262973934c8689b851adad60afe0 (DEFAULT_TABLE table=default.Table_0fc5eac5c02943af9b6801ac9cee55a3 [id=28235bf690f747b398aea70f53caa578]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:15.728515 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0cb3262973934c8689b851adad60afe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:15.732604 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_a210833682434f928e7da36d2fb4e09c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:15.733233 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a210833682434f928e7da36d2fb4e09c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:15.734638 1038 tablet_service.cc:1468] Processing CreateTablet for tablet 7b16ca1daf394a6da65a4af8da82ec33 (DEFAULT_TABLE table=default.Table_9d8c1fb24bc24631a2ba2b34a395f24a [id=2d12d6d6275e499ba3928e76bfd3cfc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:15.734845 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 49aaaec611784e6bac131efd58dbca51 (DEFAULT_TABLE table=default.Table_e098ab0a5d4045cfb18b7182ffda1b99 [id=e24b91b4ac904ef0a8c4725f70bced2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:15.735065 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b16ca1daf394a6da65a4af8da82ec33. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:15.735267 1035 tablet_service.cc:1468] Processing CreateTablet for tablet e1d18e999b5e42b1873631e0f1445081 (DEFAULT_TABLE table=default.Table_fe014107d0ee4fb8bdfca5356093854b [id=033c76e0598342c495ce2be42a0df5dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:15.735416 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 8dd587304e6342edbe96c9897b40e5c9 (DEFAULT_TABLE table=default.Table_ee242c34410e4c6887405d04a4b26124 [id=c9a2f96383664a4e8524c286cd9e4149]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:15.735620 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1d18e999b5e42b1873631e0f1445081. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:15.735818 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8dd587304e6342edbe96c9897b40e5c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:15.736114 1033 tablet_service.cc:1468] Processing CreateTablet for tablet b7b802f976f3471faaaecc964ec87e48 (DEFAULT_TABLE table=default.Table_3552b4561f10466b9b77f797ca003e53 [id=ec380a4600f34080bc1e343130537377]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:15.736507 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7b802f976f3471faaaecc964ec87e48. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:15.736723 1030 tablet_service.cc:1468] Processing CreateTablet for tablet c49d1143d54b4c319739488906b0e271 (DEFAULT_TABLE table=default.Table_583e53c2e86544e9be68eb27ab6068a3 [id=f376eaf65bb549429aa71b03fca81647]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:15.737068 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c49d1143d54b4c319739488906b0e271. 1 dirs total, 0 dirs full, 0 dirs failed
#8 0xb804d6 in kudu::tools::(anonymous namespace)::TServerRun(kudu::tools::RunnerContext const&) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/tool_action_tserver.cc:114:10
I20250901 01:52:15.740578 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49aaaec611784e6bac131efd58dbca51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:15.740707 1032 tablet_service.cc:1468] Processing CreateTablet for tablet 579b9cdf819f4609a532e96bec093fd5 (DEFAULT_TABLE table=default.Table_b0e916c76a7a4766aa0887cc8691488f [id=cf61137a852c48748a29f833f1860da7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:15.741070 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 579b9cdf819f4609a532e96bec093fd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:15.742568 1034 tablet_service.cc:1468] Processing CreateTablet for tablet 0cb3262973934c8689b851adad60afe0 (DEFAULT_TABLE table=default.Table_0fc5eac5c02943af9b6801ac9cee55a3 [id=28235bf690f747b398aea70f53caa578]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:15.742869 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0cb3262973934c8689b851adad60afe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:15.749811 446 master_service.cc:948] ReplaceTablet: received request to replace tablet c95798f0daf4437db18a291c4464522e from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:15.750016 1293 raft_consensus.cc:491] T 4d27fb61943c45d4aca67689cc3a9ff3 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:15.750196 901 tablet_service.cc:1468] Processing CreateTablet for tablet 8dd587304e6342edbe96c9897b40e5c9 (DEFAULT_TABLE table=default.Table_ee242c34410e4c6887405d04a4b26124 [id=c9a2f96383664a4e8524c286cd9e4149]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:15.750523 903 tablet_service.cc:1468] Processing CreateTablet for tablet 49aaaec611784e6bac131efd58dbca51 (DEFAULT_TABLE table=default.Table_e098ab0a5d4045cfb18b7182ffda1b99 [id=e24b91b4ac904ef0a8c4725f70bced2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:15.750675 901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8dd587304e6342edbe96c9897b40e5c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:15.750902 903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49aaaec611784e6bac131efd58dbca51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:15.751467 1306 ts_tablet_manager.cc:1918] T a7ff996482164776b468cf305989866b P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:15.752696 892 tablet_service.cc:1468] Processing CreateTablet for tablet 3c12d6a70ac24dce82e851782e6fe5aa (DEFAULT_TABLE table=default.Table_c3570db2dfd14451aa98edf48f7b2a5e [id=26bba0e60b6b43eb87d084be403c556b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:15.753170 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c12d6a70ac24dce82e851782e6fe5aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:15.753443 1306 log.cc:1199] T a7ff996482164776b468cf305989866b P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/a7ff996482164776b468cf305989866b
I20250901 01:52:15.750234 1293 raft_consensus.cc:513] T 4d27fb61943c45d4aca67689cc3a9ff3 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:15.755180 1293 leader_election.cc:290] T 4d27fb61943c45d4aca67689cc3a9ff3 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:15.763744 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_e113aaaa38db48fdb16d32569b81b930"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:15.764335 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e113aaaa38db48fdb16d32569b81b930 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:15.766461 1306 ts_tablet_manager.cc:1939] T a7ff996482164776b468cf305989866b P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:15.768126 895 tablet_service.cc:1468] Processing CreateTablet for tablet 50e3da53433f4c14b57db8a733fa53bc (DEFAULT_TABLE table=default.Table_4ce20bf6ad224378861e083f6a480edf [id=cd8eab7eafac490c896a2a9be18e426f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:15.768097 893 tablet_service.cc:1468] Processing CreateTablet for tablet eac50cbae691416e903cb2cbc907f96d (DEFAULT_TABLE table=default.Table_34a777af60f64f5ba5a17c0875d03f38 [id=e4a6d6fe44f947e59089e1d7855b785c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:15.768311 900 tablet_service.cc:1468] Processing CreateTablet for tablet 616ff34e8d314e0aa0e11ce60be11789 (DEFAULT_TABLE table=default.Table_6eca408a8cf342e1bafcbfbf927e214b [id=5ee13d46e06a4fa59b1070547573e523]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:15.768585 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50e3da53433f4c14b57db8a733fa53bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:15.768831 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 616ff34e8d314e0aa0e11ce60be11789. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:15.768780 891 tablet_service.cc:1468] Processing CreateTablet for tablet 7f64ebd080944190be2e64d02d70a8a7 (DEFAULT_TABLE table=default.Table_f94fe76831a54c6e8c2b13c81c251b14 [id=8b4cf5984d174d71a736959f5db32144]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:15.769541 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eac50cbae691416e903cb2cbc907f96d. 1 dirs total, 0 dirs full, 0 dirs failed
#9 0x9ab140 in std::_Function_handler<kudu::Status (kudu::tools::RunnerContext const&), kudu::Status (*)(kudu::tools::RunnerContext const&)>::_M_invoke(std::_Any_data const&, kudu::tools::RunnerContext const&) ../../../include/c++/7.5.0/bits/std_function.h:301:9
I20250901 01:52:15.769824 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f64ebd080944190be2e64d02d70a8a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:15.772800 890 tablet_service.cc:1468] Processing CreateTablet for tablet 20f7bfab18134e079e91c52c89468cc3 (DEFAULT_TABLE table=default.Table_18b93109e1224ea59b5c93825b3b27d6 [id=d2ef66bea68843bcb33819f88668437f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:15.773368 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20f7bfab18134e079e91c52c89468cc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:15.776880 889 tablet_service.cc:1468] Processing CreateTablet for tablet 15e4d972e8254cf899c3e0b71b6e2bd9 (DEFAULT_TABLE table=default.Table_ce16a1b94f714c2699ee6cd916bb0215 [id=551e2f9616d84026b96b6c22507f29de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:15.777380 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15e4d972e8254cf899c3e0b71b6e2bd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:15.788709 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet a7ff996482164776b468cf305989866b (table default.Table_1f8f7c00026749f593d89bc9e92f71cc [id=37883dea90384a2a8aa832201d66b7b6]) successfully deleted
W20250901 01:52:15.789040 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet b73e362321284df1b18ef755dfe930a5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b73e362321284df1b18ef755dfe930a5 already in progress: creating tablet
I20250901 01:52:15.767882 1306 tablet_replica.cc:331] T 154574e17f214cc0ad56e6e6be3af899 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:15.790328 898 tablet_service.cc:1468] Processing CreateTablet for tablet 2240302d9178445d8700e35f5ea274f7 (DEFAULT_TABLE table=default.Table_36f5f8854ed14d1ea48eb18865d00659 [id=bdf8a102ae7d43228173d7b82507d80d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:15.790885 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2240302d9178445d8700e35f5ea274f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:15.803104 1150 tablet_bootstrap.cc:654] T 824229183e024feaa975c7f0387c9c1a P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:15.803411 453 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 3fda93767e494a4684978c8522b829dd
I20250901 01:52:15.804198 453 master_service.cc:948] ReplaceTablet: received request to replace tablet b541a6e14f114ab29a864cca11806316 from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:15.804922 449 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 23af154a6f1c4e2e9352fb17c9024f10
I20250901 01:52:15.805568 449 master_service.cc:948] ReplaceTablet: received request to replace tablet 40d74d7b5a3b4d1dae09eebd9e07a0c4 from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:15.805363 1306 raft_consensus.cc:2241] T 154574e17f214cc0ad56e6e6be3af899 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:15.806604 1306 raft_consensus.cc:2270] T 154574e17f214cc0ad56e6e6be3af899 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:15.811780 1150 tablet_bootstrap.cc:492] T 824229183e024feaa975c7f0387c9c1a P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:15.811967 1306 ts_tablet_manager.cc:1905] T 154574e17f214cc0ad56e6e6be3af899 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:15.812773 898 tablet_service.cc:1515] Processing DeleteTablet for tablet 3fda93767e494a4684978c8522b829dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:15 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:15.811965 1150 ts_tablet_manager.cc:1397] T 824229183e024feaa975c7f0387c9c1a P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.113s user 0.000s sys 0.006s
I20250901 01:52:15.814045 1150 raft_consensus.cc:357] T 824229183e024feaa975c7f0387c9c1a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:15.815255 1150 raft_consensus.cc:383] T 824229183e024feaa975c7f0387c9c1a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:15.815547 1150 raft_consensus.cc:738] T 824229183e024feaa975c7f0387c9c1a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:15.816155 1150 consensus_queue.cc:260] T 824229183e024feaa975c7f0387c9c1a P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:15.819340 921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7cc1efcb6bff4aba8015d2d3b7f4c923" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:15.822091 1150 ts_tablet_manager.cc:1428] T 824229183e024feaa975c7f0387c9c1a P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20250901 01:52:15.827876 921 raft_consensus.cc:2466] T 7cc1efcb6bff4aba8015d2d3b7f4c923 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:15.827939 898 tablet_service.cc:1468] Processing CreateTablet for tablet 64fb46b942ff4e4c8749986ecc3dfe65 (DEFAULT_TABLE table=default.Table_9bf84b7a07be4f57a58101480fe6d2c0 [id=92cdf22693354ddf93b83c67cf553a0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:15.828107 1150 tablet_bootstrap.cc:492] T a774cc6ae3924d299127fd9ffb01ca57 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:15.828433 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64fb46b942ff4e4c8749986ecc3dfe65. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:15.830001 1150 tablet_bootstrap.cc:654] T a774cc6ae3924d299127fd9ffb01ca57 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:15.845705 889 tablet_service.cc:1468] Processing CreateTablet for tablet f4fd117c395e41c2ab679c684908c5b7 (DEFAULT_TABLE table=default.Table_5e32028e03b04431b57a5d3a653afda4 [id=305490ecfddb4d46927f7bb128c7f878]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:15.846552 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4fd117c395e41c2ab679c684908c5b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:15.848776 900 tablet_service.cc:1468] Processing CreateTablet for tablet aa424ac6ef1441e6b1cc2867b1c48186 (DEFAULT_TABLE table=default.Table_aba07c28a3e248e4b102730106d6a2ba [id=76928d8caa2f4134a014656cf2cc871a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:15.849156 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa424ac6ef1441e6b1cc2867b1c48186. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:15.849759 1293 raft_consensus.cc:491] T 12c04e0c1f804179ae23509cdb24d395 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:15.849947 1293 raft_consensus.cc:513] T 12c04e0c1f804179ae23509cdb24d395 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:15.851065 1293 leader_election.cc:290] T 12c04e0c1f804179ae23509cdb24d395 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:15.852311 1150 tablet_bootstrap.cc:492] T a774cc6ae3924d299127fd9ffb01ca57 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:15.852283 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 23af154a6f1c4e2e9352fb17c9024f10 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:15 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:15.852540 1150 ts_tablet_manager.cc:1397] T a774cc6ae3924d299127fd9ffb01ca57 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.030s user 0.006s sys 0.006s
I20250901 01:52:15.853600 1150 raft_consensus.cc:357] T a774cc6ae3924d299127fd9ffb01ca57 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:15.853842 1150 raft_consensus.cc:383] T a774cc6ae3924d299127fd9ffb01ca57 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:15.854012 1150 raft_consensus.cc:738] T a774cc6ae3924d299127fd9ffb01ca57 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:15.854285 1150 consensus_queue.cc:260] T a774cc6ae3924d299127fd9ffb01ca57 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:15.854969 898 tablet_service.cc:1468] Processing CreateTablet for tablet e9b179dba2cc49428ba8090c962d6136 (DEFAULT_TABLE table=default.Table_271fd01373bf49f2ab8b4faf82db7393 [id=d409bc5c1aa548c69feacab89bd3d3ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:15.855129 1150 ts_tablet_manager.cc:1428] T a774cc6ae3924d299127fd9ffb01ca57 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250901 01:52:15.855342 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9b179dba2cc49428ba8090c962d6136. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:15.855495 1150 tablet_bootstrap.cc:492] T 49d30da95b14400b845270122487a2a0 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:15.863740 900 tablet_service.cc:1468] Processing CreateTablet for tablet d0c3c363dda444519a17dca63ad9caa7 (DEFAULT_TABLE table=default.Table_6400e401888045348632f924f636d011 [id=8c1b263824ec4c15a5ec424364c78895]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:15.864332 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0c3c363dda444519a17dca63ad9caa7. 1 dirs total, 0 dirs full, 0 dirs failed
W20250901 01:52:15.868744 702 kernel_stack_watchdog.cc:198] Thread 757 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 4214ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fe78f073980 (unknown) at ??:0
@ 0x92ce64 __sanitizer::BlockingMutex::Lock() at ??:?
@ 0x932e1b __sanitizer::ThreadRegistry::CreateThread() at ??:?
@ 0x91983b __asan::AsanThread::Create() at ??:?
@ 0x8921ac __interceptor_pthread_create at ??:?
@ 0x7fe78bc459c1 kudu::Thread::StartThread() at ??:0
@ 0xa63f97 kudu::Thread::Create() at ??:?
@ 0x7fe78bc6580b kudu::ThreadPool::CreateThread() at ??:0
@ 0x7fe78bc6122d kudu::ThreadPool::DoSubmit() at ??:0
@ 0x7fe78bc6655f kudu::ThreadPool::Submit() at ??:0
@ 0x7fe79b49d226 kudu::tserver::TSTabletManager::DeleteTabletAsync() at ??:0
@ 0x7fe79b3e7ed0 kudu::tserver::TabletServiceAdminImpl::DeleteTablet() at ??:0
@ 0x7fe7955e6507 kudu::tserver::TabletServerAdminServiceIf::TabletServerAdminServiceIf()::$_3::operator()() at ??:0
@ 0x7fe7955e6236 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fe7929cf2cf std::function<>::operator()() at ??:0
@ 0x7fe7929ce1af kudu::rpc::GeneratedServiceIf::Handle() at ??:0
I20250901 01:52:15.875684 1293 raft_consensus.cc:491] T 6c05eff658924ca4a9a1e7a37f917d67 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:15.875914 1293 raft_consensus.cc:513] T 6c05eff658924ca4a9a1e7a37f917d67 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:15.876710 1293 leader_election.cc:290] T 6c05eff658924ca4a9a1e7a37f917d67 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:15.881762 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 7f64ebd080944190be2e64d02d70a8a7
W20250901 01:52:15.881965 991 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250901 01:52:15.882249 455 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:57810 (request call id 320) took 1002 ms. Trace:
I20250901 01:52:15.882354 455 rpcz_store.cc:276] 0901 01:52:14.880127 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:15.142461 (+262334us) service_pool.cc:225] Handling call
0901 01:52:15.142962 (+ 501us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0901 01:52:15.143096 (+ 134us) catalog_manager.cc:2790] Modifying in-memory table state
0901 01:52:15.143165 (+ 69us) catalog_manager.cc:2799] Locking tablets
0901 01:52:15.698178 (+555013us) catalog_manager.cc:2813] Removing table and tablets from system table
0901 01:52:15.881591 (+183413us) catalog_manager.cc:2831] Removing table from by-name map
0901 01:52:15.881666 (+ 75us) catalog_manager.cc:2847] Committing in-memory state
0901 01:52:15.881697 (+ 31us) catalog_manager.cc:2851] Aborting table tasks
0901 01:52:15.882217 (+ 520us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0901 01:52:15.758461 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:15.758538 (+ 77us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:15.758543 (+ 5us) write_op.cc:435] Acquired schema lock
0901 01:52:15.758546 (+ 3us) tablet.cc:660] Decoding operations
0901 01:52:15.758634 (+ 88us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:15.758641 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:15.758645 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0901 01:52:15.758735 (+ 90us) tablet.cc:699] Row locks acquired
0901 01:52:15.758740 (+ 5us) write_op.cc:260] PREPARE: finished
0901 01:52:15.758769 (+ 29us) write_op.cc:270] Start()
0901 01:52:15.758794 (+ 25us) write_op.cc:276] Timestamp: P: 1756691535758745 usec, L: 0
0901 01:52:15.758797 (+ 3us) op_driver.cc:348] REPLICATION: starting
0901 01:52:15.758928 (+ 131us) log.cc:844] Serialized 725 byte log entry
0901 01:52:15.880785 (+121857us) op_driver.cc:464] REPLICATION: finished
0901 01:52:15.881318 (+ 533us) write_op.cc:301] APPLY: starting
0901 01:52:15.881326 (+ 8us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:15.881352 (+ 26us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:15.881371 (+ 19us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:15.881415 (+ 44us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:15.881436 (+ 21us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0901 01:52:15.881442 (+ 6us) write_op.cc:312] APPLY: finished
0901 01:52:15.881481 (+ 39us) log.cc:844] Serialized 29 byte log entry
0901 01:52:15.881510 (+ 29us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:15.881522 (+ 12us) write_op.cc:454] Released schema lock
0901 01:52:15.881526 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":509,"num_ops":2,"prepare.queue_time_us":59715,"prepare.run_cpu_time_us":517,"prepare.run_wall_time_us":516,"replication_time_us":121975}]]}
I20250901 01:52:15.883018 900 tablet_service.cc:1468] Processing CreateTablet for tablet 42c1156e8dd84c6fa1396b46de13eb6f (DEFAULT_TABLE table=default.Table_d6119d1604674319914fa8cb698f16c5 [id=40d05d90b0354334840ad17764f5b6bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:15.883437 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42c1156e8dd84c6fa1396b46de13eb6f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250901 01:52:15.884774 452 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.3:47125 (request call id 29) took 822 ms (client timeout 1000 ms). Trace:
I20250901 01:52:15.884840 1150 tablet_bootstrap.cc:654] T 49d30da95b14400b845270122487a2a0 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:15.884902 452 rpcz_store.cc:269] 0901 01:52:15.062550 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:15.083839 (+ 21289us) service_pool.cc:225] Handling call
0901 01:52:15.086008 (+ 2169us) spinlock_profiling.cc:243] Waited 1.65 ms on lock 0x616000040f54. stack: 00007fc53edd57d6 00007fc53edd5590 00007fc53ddc83ad 0000000000972d94 0000000000972d5d 00007fc54dba014d 00007fc54dba0243 00007fc54a85f941 00007fc5523eaa5d 00007fc55256e6d8 00007fc5493a8c86 00007fc5493a89b5 00007fc545b832ce 00007fc545b821ae 00007fc545b883e5 00007fc545b88d0f
0901 01:52:15.884740 (+798732us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0901 01:52:15.757991 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:15.758070 (+ 79us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:15.758074 (+ 4us) write_op.cc:435] Acquired schema lock
0901 01:52:15.758076 (+ 2us) tablet.cc:660] Decoding operations
0901 01:52:15.758150 (+ 74us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:15.758158 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:15.758162 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0901 01:52:15.758200 (+ 38us) tablet.cc:699] Row locks acquired
0901 01:52:15.758202 (+ 2us) write_op.cc:260] PREPARE: finished
0901 01:52:15.758223 (+ 21us) write_op.cc:270] Start()
0901 01:52:15.758244 (+ 21us) write_op.cc:276] Timestamp: P: 1756691535758202 usec, L: 0
0901 01:52:15.758247 (+ 3us) op_driver.cc:348] REPLICATION: starting
0901 01:52:15.758362 (+ 115us) log.cc:844] Serialized 527 byte log entry
0901 01:52:15.880719 (+122357us) op_driver.cc:464] REPLICATION: finished
0901 01:52:15.880845 (+ 126us) write_op.cc:301] APPLY: starting
0901 01:52:15.880858 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:15.880908 (+ 50us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:15.880954 (+ 46us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:15.881002 (+ 48us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:15.881030 (+ 28us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0901 01:52:15.881038 (+ 8us) write_op.cc:312] APPLY: finished
0901 01:52:15.881111 (+ 73us) log.cc:844] Serialized 23 byte log entry
0901 01:52:15.881193 (+ 82us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:15.881207 (+ 14us) write_op.cc:454] Released schema lock
0901 01:52:15.881212 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0901 01:52:15.881265 (+ 53us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:15.881268 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":4,"spinlock_wait_cycles":3800832,"child_traces":[["op",{"apply.queue_time_us":80,"apply.run_cpu_time_us":415,"apply.run_wall_time_us":416,"num_ops":1,"prepare.queue_time_us":61753,"prepare.run_cpu_time_us":430,"prepare.run_wall_time_us":430,"replication_time_us":122445}]]}
I20250901 01:52:15.885574 452 master_service.cc:948] ReplaceTablet: received request to replace tablet 4a87e68850c4459eb2c8eafac89c6aa9 from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:15.890570 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet abe27e749fa0485195e0064a49d5431e
I20250901 01:52:15.892882 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 7f64ebd080944190be2e64d02d70a8a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:15 UTC) from {username='slave'} at 127.0.0.1:52846
W20250901 01:52:15.894109 451 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3c1fd0820d8a4995abe027f685da7f7b consensus state has no leader
I20250901 01:52:15.895517 891 tablet_service.cc:1515] Processing DeleteTablet for tablet abe27e749fa0485195e0064a49d5431e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:15 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:15.897661 455 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_874154cd2b0c4afcb0fa684f312984db" } new_table_name: "default.Table_4e1d0964cecf42e99b61dc169027cee1" modify_external_catalogs: true
I20250901 01:52:15.899614 451 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_f4026aef620b4786b9012444560a1a76" } modify_external_catalogs: true
I20250901 01:52:15.899854 451 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_f4026aef620b4786b9012444560a1a76" } modify_external_catalogs: true
I20250901 01:52:15.902644 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_fc0336a00d084f6e98e1d8d32142e66f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:15.903385 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fc0336a00d084f6e98e1d8d32142e66f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:15.912184 1306 ts_tablet_manager.cc:1918] T 154574e17f214cc0ad56e6e6be3af899 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:15.912381 1306 log.cc:1199] T 154574e17f214cc0ad56e6e6be3af899 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/154574e17f214cc0ad56e6e6be3af899
I20250901 01:52:15.912451 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 3c12d6a70ac24dce82e851782e6fe5aa (DEFAULT_TABLE table=default.Table_c3570db2dfd14451aa98edf48f7b2a5e [id=26bba0e60b6b43eb87d084be403c556b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:15.912828 1306 ts_tablet_manager.cc:1939] T 154574e17f214cc0ad56e6e6be3af899 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:15.913586 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 154574e17f214cc0ad56e6e6be3af899 (table default.Table_cb6a9252a90d4184b917db4c7edc06b1 [id=b550b0e6553e41a2a20ea3bf23e1c7a0]) successfully deleted
I20250901 01:52:15.914389 1150 tablet_bootstrap.cc:492] T 49d30da95b14400b845270122487a2a0 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:15.914378 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d27fb61943c45d4aca67689cc3a9ff3" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:15.914597 1150 ts_tablet_manager.cc:1397] T 49d30da95b14400b845270122487a2a0 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.059s user 0.005s sys 0.000s
I20250901 01:52:15.915577 1150 raft_consensus.cc:357] T 49d30da95b14400b845270122487a2a0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:15.915789 1150 raft_consensus.cc:383] T 49d30da95b14400b845270122487a2a0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:15.915877 1150 raft_consensus.cc:738] T 49d30da95b14400b845270122487a2a0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:15.916133 1150 consensus_queue.cc:260] T 49d30da95b14400b845270122487a2a0 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:15.917259 900 tablet_service.cc:1468] Processing CreateTablet for tablet 25ce0fe8c9544f828694184d19d13df9 (DEFAULT_TABLE table=default.Table_dcacb44eed4241d3acbae153ebaad663 [id=0241aa3ab1984557a151461005c6b97c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:15.917630 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25ce0fe8c9544f828694184d19d13df9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:15.919111 1031 tablet_service.cc:1468] Processing CreateTablet for tablet 2240302d9178445d8700e35f5ea274f7 (DEFAULT_TABLE table=default.Table_36f5f8854ed14d1ea48eb18865d00659 [id=bdf8a102ae7d43228173d7b82507d80d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:15.919860 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_048a628c8a35498f94012e430b3c29bc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:15.919766 1151 tablet_bootstrap.cc:492] T 31390f38258c465cb1798615a1fcc790 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
W20250901 01:52:15.920439 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_048a628c8a35498f94012e430b3c29bc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:15.920769 1036 tablet_service.cc:1468] Processing CreateTablet for tablet eac50cbae691416e903cb2cbc907f96d (DEFAULT_TABLE table=default.Table_34a777af60f64f5ba5a17c0875d03f38 [id=e4a6d6fe44f947e59089e1d7855b785c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:15.920986 1034 tablet_service.cc:1468] Processing CreateTablet for tablet 50e3da53433f4c14b57db8a733fa53bc (DEFAULT_TABLE table=default.Table_4ce20bf6ad224378861e083f6a480edf [id=cd8eab7eafac490c896a2a9be18e426f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:15.921692 446 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet c95798f0daf4437db18a291c4464522e
I20250901 01:52:15.922118 446 catalog_manager.cc:6652] ReplaceTablet: tablet c95798f0daf4437db18a291c4464522e deleted and replaced by tablet 7a2f6e2c7ee2402ea310c058e046416c
I20250901 01:52:15.922442 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_5e84ae9a9c2a4084a1f9c5ffb4594de2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:15.922940 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5e84ae9a9c2a4084a1f9c5ffb4594de2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:15.923391 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c12d6a70ac24dce82e851782e6fe5aa. 1 dirs total, 0 dirs full, 0 dirs failed
W20250901 01:52:15.924783 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 3fda93767e494a4684978c8522b829dd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3fda93767e494a4684978c8522b829dd already in progress: creating tablet
I20250901 01:52:15.926801 891 tablet_service.cc:1515] Processing DeleteTablet for tablet c95798f0daf4437db18a291c4464522e with delete_type TABLET_DATA_DELETED (replaced by tablet 7a2f6e2c7ee2402ea310c058e046416c) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:15.930135 1151 ts_tablet_manager.cc:1397] T 31390f38258c465cb1798615a1fcc790 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.271s user 0.003s sys 0.009s
I20250901 01:52:15.931452 1037 tablet_service.cc:1468] Processing CreateTablet for tablet 616ff34e8d314e0aa0e11ce60be11789 (DEFAULT_TABLE table=default.Table_6eca408a8cf342e1bafcbfbf927e214b [id=5ee13d46e06a4fa59b1070547573e523]), partition=RANGE (key) PARTITION UNBOUNDED
W20250901 01:52:15.933053 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 23af154a6f1c4e2e9352fb17c9024f10 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23af154a6f1c4e2e9352fb17c9024f10 already in progress: creating tablet
I20250901 01:52:15.936532 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50e3da53433f4c14b57db8a733fa53bc. 1 dirs total, 0 dirs full, 0 dirs failed
W20250901 01:52:15.938659 858 leader_election.cc:336] T 8f2e702121c845e0a09a254bbb81c142 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Timed out: RequestConsensusVote RPC to 127.0.104.1:40731 timed out after 3.069s (SENT)
I20250901 01:52:15.939011 454 catalog_manager.cc:6127] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: default.Table_68f31d33b83544cea7b959bc7c41f94b [id=1d8f34cc14cb4e90a562ff4424b8577a] alter complete (version 4)
I20250901 01:52:15.940323 453 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet b541a6e14f114ab29a864cca11806316
I20250901 01:52:15.940973 453 catalog_manager.cc:6652] ReplaceTablet: tablet b541a6e14f114ab29a864cca11806316 deleted and replaced by tablet 2599f98abfdc4871b573edfc2ad97d22
I20250901 01:52:15.941900 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2240302d9178445d8700e35f5ea274f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:15.948534 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eac50cbae691416e903cb2cbc907f96d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:15.948941 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 60285f2ba037433999f5455fb9cc7acd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:13 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:15.949326 890 tablet_service.cc:1515] Processing DeleteTablet for tablet b541a6e14f114ab29a864cca11806316 with delete_type TABLET_DATA_DELETED (replaced by tablet 2599f98abfdc4871b573edfc2ad97d22) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:15.950632 1151 raft_consensus.cc:357] T 31390f38258c465cb1798615a1fcc790 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:15.960182 889 tablet_service.cc:1468] Processing CreateTablet for tablet 4855448edb4b40c184838aa7f81c112b (DEFAULT_TABLE table=default.Table_52733c9a363b4ed9b6ff11ad8571e421 [id=453153553bea48319e132857f628d48a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:15.960901 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 7f64ebd080944190be2e64d02d70a8a7 (DEFAULT_TABLE table=default.Table_f94fe76831a54c6e8c2b13c81c251b14 [id=8b4cf5984d174d71a736959f5db32144]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:15.961843 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 616ff34e8d314e0aa0e11ce60be11789. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:15.962440 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4855448edb4b40c184838aa7f81c112b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250901 01:52:15.969605 455 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet eaa521c52a2f4408b59770bd2a05907c has no consensus state
I20250901 01:52:15.969794 1151 raft_consensus.cc:383] T 31390f38258c465cb1798615a1fcc790 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:15.969919 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 20f7bfab18134e079e91c52c89468cc3 (DEFAULT_TABLE table=default.Table_18b93109e1224ea59b5c93825b3b27d6 [id=d2ef66bea68843bcb33819f88668437f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:15.972389 1038 tablet_service.cc:1468] Processing CreateTablet for tablet 15e4d972e8254cf899c3e0b71b6e2bd9 (DEFAULT_TABLE table=default.Table_ce16a1b94f714c2699ee6cd916bb0215 [id=551e2f9616d84026b96b6c22507f29de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:15.972600 451 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet ddd501ec362d4728884c74d4a1835f89
I20250901 01:52:15.972927 449 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 40d74d7b5a3b4d1dae09eebd9e07a0c4
I20250901 01:52:15.973431 1151 raft_consensus.cc:738] T 31390f38258c465cb1798615a1fcc790 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:15.973524 449 catalog_manager.cc:6652] ReplaceTablet: tablet 40d74d7b5a3b4d1dae09eebd9e07a0c4 deleted and replaced by tablet 6cf09e8f299548dba2cce16d7b565bad
I20250901 01:52:15.973726 1151 consensus_queue.cc:260] T 31390f38258c465cb1798615a1fcc790 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:15.974294 1151 ts_tablet_manager.cc:1428] T 31390f38258c465cb1798615a1fcc790 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.044s user 0.002s sys 0.000s
I20250901 01:52:15.975033 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f64ebd080944190be2e64d02d70a8a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:15.977491 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20f7bfab18134e079e91c52c89468cc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:15.977700 898 tablet_service.cc:1515] Processing DeleteTablet for tablet 40d74d7b5a3b4d1dae09eebd9e07a0c4 with delete_type TABLET_DATA_DELETED (replaced by tablet 6cf09e8f299548dba2cce16d7b565bad) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:15.978821 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_853b0fae8015415b91325d10a633ccf6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:15.979074 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_2872910642aa48eca0350b2792008200"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:15.979632 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2872910642aa48eca0350b2792008200 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250901 01:52:15.979661 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_853b0fae8015415b91325d10a633ccf6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:15.980499 1033 tablet_service.cc:1515] Processing DeleteTablet for tablet 3fda93767e494a4684978c8522b829dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:15 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:15.981266 1033 tablet_service.cc:1468] Processing CreateTablet for tablet aa424ac6ef1441e6b1cc2867b1c48186 (DEFAULT_TABLE table=default.Table_aba07c28a3e248e4b102730106d6a2ba [id=76928d8caa2f4134a014656cf2cc871a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:15.981299 450 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_7bb211d400ca44d48416f325df223fa6" } new_table_name: "default.Table_203547c9458e43b3b48ea051d56892c6" modify_external_catalogs: true
I20250901 01:52:15.981350 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_8d0db1143f1547c2a7d41775b4d9ef09"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:15.981746 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa424ac6ef1441e6b1cc2867b1c48186. 1 dirs total, 0 dirs full, 0 dirs failed
W20250901 01:52:15.981853 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8d0db1143f1547c2a7d41775b4d9ef09 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:15.982939 449 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_8c1924d21a91425cbbbd8429d56cf0ab" } new_table_name: "default.Table_2eba037c989c4095bc355667be2fb109" modify_external_catalogs: true
I20250901 01:52:15.982965 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_bda1777b1e9c409ea39c00de92706b50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:15.984190 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bda1777b1e9c409ea39c00de92706b50 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:15.994210 1151 tablet_bootstrap.cc:492] T 22691943a134400c8df4617240e5d3ef P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:16.000771 455 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet a774cc6ae3924d299127fd9ffb01ca57 (table default.Table_1159dbb70cc940df8b509db6e70c350a [id=1d0bc46740444f1fbf2129270e806259]). Expected version 4 got 0
I20250901 01:52:16.001188 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 23af154a6f1c4e2e9352fb17c9024f10 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:15 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:16.001595 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet abe27e749fa0485195e0064a49d5431e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:15 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:16.002007 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 42c1156e8dd84c6fa1396b46de13eb6f (DEFAULT_TABLE table=default.Table_d6119d1604674319914fa8cb698f16c5 [id=40d05d90b0354334840ad17764f5b6bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:16.002388 455 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 824229183e024feaa975c7f0387c9c1a (table default.Table_5e65c5f068b84995a2d7a76fa65dda0f [id=b9d81ddd38fe47838877b010be2b40d0]). Expected version 1 got 0
I20250901 01:52:16.005963 455 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_bbdd9e12d55540eb885b874ef8358435" } new_table_name: "default.Table_62ececf8293e4e22bad9e9568f496bbe" modify_external_catalogs: true
I20250901 01:52:16.006165 1024 tablet_service.cc:1468] Processing CreateTablet for tablet f4fd117c395e41c2ab679c684908c5b7 (DEFAULT_TABLE table=default.Table_5e32028e03b04431b57a5d3a653afda4 [id=305490ecfddb4d46927f7bb128c7f878]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:16.007047 1022 tablet_service.cc:1468] Processing CreateTablet for tablet e9b179dba2cc49428ba8090c962d6136 (DEFAULT_TABLE table=default.Table_271fd01373bf49f2ab8b4faf82db7393 [id=d409bc5c1aa548c69feacab89bd3d3ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:16.007283 1021 tablet_service.cc:1468] Processing CreateTablet for tablet d0c3c363dda444519a17dca63ad9caa7 (DEFAULT_TABLE table=default.Table_6400e401888045348632f924f636d011 [id=8c1b263824ec4c15a5ec424364c78895]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:16.007771 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet 7f64ebd080944190be2e64d02d70a8a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:15 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:16.008019 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet c95798f0daf4437db18a291c4464522e with delete_type TABLET_DATA_DELETED (replaced by tablet 7a2f6e2c7ee2402ea310c058e046416c) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:16.008216 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet b541a6e14f114ab29a864cca11806316 with delete_type TABLET_DATA_DELETED (replaced by tablet 2599f98abfdc4871b573edfc2ad97d22) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:16.008664 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15e4d972e8254cf899c3e0b71b6e2bd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:16.008957 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 88a6ab29d00542aba54f0e4185fbf615 with delete_type TABLET_DATA_DELETED (replaced by tablet a774cc6ae3924d299127fd9ffb01ca57) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:16.009159 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 64fb46b942ff4e4c8749986ecc3dfe65 (DEFAULT_TABLE table=default.Table_9bf84b7a07be4f57a58101480fe6d2c0 [id=92cdf22693354ddf93b83c67cf553a0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:16.009496 452 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 4a87e68850c4459eb2c8eafac89c6aa9
W20250901 01:52:16.009819 992 leader_election.cc:343] T 9496ec6021784800aef62996b0dee593 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 9496ec6021784800aef62996b0dee593
I20250901 01:52:16.009860 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 25ce0fe8c9544f828694184d19d13df9 (DEFAULT_TABLE table=default.Table_dcacb44eed4241d3acbae153ebaad663 [id=0241aa3ab1984557a151461005c6b97c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:16.009984 452 catalog_manager.cc:6652] ReplaceTablet: tablet 4a87e68850c4459eb2c8eafac89c6aa9 deleted and replaced by tablet 615b2306fca84003b264da9d5f95ddab
I20250901 01:52:16.010063 992 leader_election.cc:304] T 7cc1efcb6bff4aba8015d2d3b7f4c923 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:16.010382 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_657603f2bb3c43bc85028218860da978"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:16.010576 1552 raft_consensus.cc:2802] T 7cc1efcb6bff4aba8015d2d3b7f4c923 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:16.010717 1552 raft_consensus.cc:491] T 7cc1efcb6bff4aba8015d2d3b7f4c923 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:16.010797 1552 raft_consensus.cc:3058] T 7cc1efcb6bff4aba8015d2d3b7f4c923 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
W20250901 01:52:16.011006 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_657603f2bb3c43bc85028218860da978 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:16.010581 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 4855448edb4b40c184838aa7f81c112b (DEFAULT_TABLE table=default.Table_52733c9a363b4ed9b6ff11ad8571e421 [id=453153553bea48319e132857f628d48a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:16.008960 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9b179dba2cc49428ba8090c962d6136. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:16.013178 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64fb46b942ff4e4c8749986ecc3dfe65. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:16.013782 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4fd117c395e41c2ab679c684908c5b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:16.014598 448 master_service.cc:948] ReplaceTablet: received request to replace tablet f06a77ecc841462396a8d5026c7327f4 from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:16.014845 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4855448edb4b40c184838aa7f81c112b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:16.015282 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0c3c363dda444519a17dca63ad9caa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:16.015983 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42c1156e8dd84c6fa1396b46de13eb6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:16.016510 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25ce0fe8c9544f828694184d19d13df9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:16.021240 1151 tablet_bootstrap.cc:654] T 22691943a134400c8df4617240e5d3ef P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:16.025333 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 40d74d7b5a3b4d1dae09eebd9e07a0c4 with delete_type TABLET_DATA_DELETED (replaced by tablet 6cf09e8f299548dba2cce16d7b565bad) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:16.031865 451 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_0fc5eac5c02943af9b6801ac9cee55a3" } modify_external_catalogs: true
I20250901 01:52:16.032128 451 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_0fc5eac5c02943af9b6801ac9cee55a3" } modify_external_catalogs: true
I20250901 01:52:16.032857 446 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_c9401a0bf1b84645bdfd72bc9bcbe5b6" } modify_external_catalogs: true
I20250901 01:52:16.033002 446 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_c9401a0bf1b84645bdfd72bc9bcbe5b6" } modify_external_catalogs: true
I20250901 01:52:16.033421 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "12c04e0c1f804179ae23509cdb24d395" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:16.038587 450 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b009cc95c819431ea9827a3e453b8da3 consensus state has no leader
I20250901 01:52:16.042490 1293 raft_consensus.cc:491] T a774cc6ae3924d299127fd9ffb01ca57 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:16.042768 1293 raft_consensus.cc:513] T a774cc6ae3924d299127fd9ffb01ca57 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:16.042856 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c05eff658924ca4a9a1e7a37f917d67" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:16.043692 1293 leader_election.cc:290] T a774cc6ae3924d299127fd9ffb01ca57 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:16.043879 1293 raft_consensus.cc:491] T 60285f2ba037433999f5455fb9cc7acd P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:16.044050 1293 raft_consensus.cc:513] T 60285f2ba037433999f5455fb9cc7acd P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:16.044797 1293 leader_election.cc:290] T 60285f2ba037433999f5455fb9cc7acd P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:16.044986 859 leader_election.cc:343] T 4d27fb61943c45d4aca67689cc3a9ff3 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
W20250901 01:52:16.045559 859 leader_election.cc:343] T 12c04e0c1f804179ae23509cdb24d395 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:16.050453 969 heartbeater.cc:344] Connected to a master server at 127.0.104.60:44231
I20250901 01:52:16.054255 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 4a87e68850c4459eb2c8eafac89c6aa9 with delete_type TABLET_DATA_DELETED (replaced by tablet 615b2306fca84003b264da9d5f95ddab) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:16.055058 898 tablet_service.cc:1515] Processing DeleteTablet for tablet 4d27fb61943c45d4aca67689cc3a9ff3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:13 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:16.057870 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet f06a77ecc841462396a8d5026c7327f4
I20250901 01:52:16.058543 448 catalog_manager.cc:6652] ReplaceTablet: tablet f06a77ecc841462396a8d5026c7327f4 deleted and replaced by tablet f3962b471bdb49aea6e9fa95f1f9dfe2
W20250901 01:52:16.060563 455 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 438cd35ff3f74c49b908d072560077db consensus state has no leader
W20250901 01:52:16.073459 992 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250901 01:52:16.050014 858 leader_election.cc:336] T fab3c1a8928a4066a15868ba708267f1 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Timed out: RequestConsensusVote RPC to 127.0.104.1:40731 timed out after 3.219s (SENT)
I20250901 01:52:16.082126 1332 ts_tablet_manager.cc:1918] T f0affc7a78634607852a48458412761d P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:16.090795 455 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_113664a018e84769a7ff61aa35a38618" } modify_external_catalogs: true
I20250901 01:52:16.091663 455 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_113664a018e84769a7ff61aa35a38618" } modify_external_catalogs: true
I20250901 01:52:16.093611 448 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_dcacb44eed4241d3acbae153ebaad663" } modify_external_catalogs: true
I20250901 01:52:16.094062 448 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_dcacb44eed4241d3acbae153ebaad663" } modify_external_catalogs: true
I20250901 01:52:16.096242 452 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_34a777af60f64f5ba5a17c0875d03f38" } new_table_name: "default.Table_8393a17fa28a481888acfc550afdb6a1" modify_external_catalogs: true
I20250901 01:52:16.101089 1150 ts_tablet_manager.cc:1428] T 49d30da95b14400b845270122487a2a0 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.101s user 0.003s sys 0.000s
W20250901 01:52:16.101032 859 leader_election.cc:343] T 6c05eff658924ca4a9a1e7a37f917d67 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:16.103211 1150 tablet_bootstrap.cc:492] T 4a87e68850c4459eb2c8eafac89c6aa9 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
W20250901 01:52:16.105911 452 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet eac50cbae691416e903cb2cbc907f96d consensus state has no leader
I20250901 01:52:16.110440 889 tablet_service.cc:1515] Processing DeleteTablet for tablet b633b461dbf34e13bc6e1cc08e6d20c1 with delete_type TABLET_DATA_DELETED (replaced by tablet 4855448edb4b40c184838aa7f81c112b) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:16.111018 1306 tablet_replica.cc:331] T 60285f2ba037433999f5455fb9cc7acd P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:16.111554 1306 raft_consensus.cc:2241] T 60285f2ba037433999f5455fb9cc7acd P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:16.113137 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 12c04e0c1f804179ae23509cdb24d395 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:13 UTC) from {username='slave'} at 127.0.0.1:52846
W20250901 01:52:16.117872 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 7f64ebd080944190be2e64d02d70a8a7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7f64ebd080944190be2e64d02d70a8a7 already in progress: creating tablet
I20250901 01:52:16.118973 898 tablet_service.cc:1515] Processing DeleteTablet for tablet f06a77ecc841462396a8d5026c7327f4 with delete_type TABLET_DATA_DELETED (replaced by tablet f3962b471bdb49aea6e9fa95f1f9dfe2) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:16.083876 1552 raft_consensus.cc:513] T 7cc1efcb6bff4aba8015d2d3b7f4c923 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:16.119172 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet abe27e749fa0485195e0064a49d5431e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet abe27e749fa0485195e0064a49d5431e already in progress: creating tablet
W20250901 01:52:16.120097 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet c95798f0daf4437db18a291c4464522e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c95798f0daf4437db18a291c4464522e already in progress: creating tablet
I20250901 01:52:16.122440 1150 tablet_bootstrap.cc:654] T 4a87e68850c4459eb2c8eafac89c6aa9 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:16.124882 900 tablet_service.cc:1515] Processing DeleteTablet for tablet 4a87e68850c4459eb2c8eafac89c6aa9 with delete_type TABLET_DATA_DELETED (replaced by tablet 615b2306fca84003b264da9d5f95ddab) from {username='slave'} at 127.0.0.1:52846
W20250901 01:52:16.127051 994 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250901 01:52:16.127192 1306 raft_consensus.cc:2270] T 60285f2ba037433999f5455fb9cc7acd P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:16.127314 1552 leader_election.cc:290] T 7cc1efcb6bff4aba8015d2d3b7f4c923 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:16.128337 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_0f91070385df441b96af4d5406a94724"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:16.129165 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0f91070385df441b96af4d5406a94724 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:16.143646 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a774cc6ae3924d299127fd9ffb01ca57" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:16.149503 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60285f2ba037433999f5455fb9cc7acd" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:16.150357 858 leader_election.cc:336] T c297e8bf11ff467a854eab41117ac376 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Timed out: RequestConsensusVote RPC to 127.0.104.1:40731 timed out after 3.026s (SENT)
I20250901 01:52:16.150537 858 leader_election.cc:304] T c297e8bf11ff467a854eab41117ac376 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:16.150816 858 raft_consensus.cc:2684] T c297e8bf11ff467a854eab41117ac376 P 00c282f779c1477d920f24dc0dc16628: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250901 01:52:16.151662 992 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250901 01:52:16.151651 859 leader_election.cc:343] T a774cc6ae3924d299127fd9ffb01ca57 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
W20250901 01:52:16.151898 859 leader_election.cc:343] T 60285f2ba037433999f5455fb9cc7acd P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:16.151968 1552 consensus_queue.cc:1035] T 4fd0e4f9d86c4362af96ce9964b72328 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:16.153204 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet f06a77ecc841462396a8d5026c7327f4 with delete_type TABLET_DATA_DELETED (replaced by tablet f3962b471bdb49aea6e9fa95f1f9dfe2) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:16.154537 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_8ac222c93e36436c9a3ecb2c489f6d74"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:16.155292 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8ac222c93e36436c9a3ecb2c489f6d74 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:16.156750 449 master_service.cc:948] ReplaceTablet: received request to replace tablet 8dd587304e6342edbe96c9897b40e5c9 from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:16.161469 1552 raft_consensus.cc:491] T 756cfe84a517459ca015aecf5cd81571 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:16.161631 1552 raft_consensus.cc:513] T 756cfe84a517459ca015aecf5cd81571 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:16.162186 1552 leader_election.cc:290] T 756cfe84a517459ca015aecf5cd81571 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:16.169849 1306 ts_tablet_manager.cc:1905] T 60285f2ba037433999f5455fb9cc7acd P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:16.172435 454 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_8ea6f8f2bd8848109b6a9f9e7f5b6e54" } new_table_name: "default.Table_604f2c5c58d74154bacf2a065a467f32" modify_external_catalogs: true
I20250901 01:52:16.173600 921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7cc1efcb6bff4aba8015d2d3b7f4c923" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:16.173913 921 raft_consensus.cc:3058] T 7cc1efcb6bff4aba8015d2d3b7f4c923 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:16.173892 923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "756cfe84a517459ca015aecf5cd81571" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:16.174095 923 raft_consensus.cc:2466] T 756cfe84a517459ca015aecf5cd81571 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:16.177245 921 raft_consensus.cc:2466] T 7cc1efcb6bff4aba8015d2d3b7f4c923 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:16.177804 450 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_3552b4561f10466b9b77f797ca003e53" } modify_external_catalogs: true
I20250901 01:52:16.177984 450 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_3552b4561f10466b9b77f797ca003e53" } modify_external_catalogs: true
I20250901 01:52:16.179076 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_5facf725e0fc44cda7d77f8511030fa5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:16.179756 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5facf725e0fc44cda7d77f8511030fa5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:16.183131 992 leader_election.cc:304] T 756cfe84a517459ca015aecf5cd81571 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:16.183432 1552 raft_consensus.cc:2802] T 756cfe84a517459ca015aecf5cd81571 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:16.183523 1552 raft_consensus.cc:491] T 756cfe84a517459ca015aecf5cd81571 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:16.183643 1552 raft_consensus.cc:3058] T 756cfe84a517459ca015aecf5cd81571 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:16.184492 992 leader_election.cc:304] T 7cc1efcb6bff4aba8015d2d3b7f4c923 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:16.184743 1374 raft_consensus.cc:2802] T 7cc1efcb6bff4aba8015d2d3b7f4c923 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:16.192261 1374 raft_consensus.cc:695] T 7cc1efcb6bff4aba8015d2d3b7f4c923 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:16.192624 1374 consensus_queue.cc:237] T 7cc1efcb6bff4aba8015d2d3b7f4c923 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:16.194913 454 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 72ba689f9230411995dc96e2b45601df consensus state has no leader
I20250901 01:52:16.196676 1552 raft_consensus.cc:513] T 756cfe84a517459ca015aecf5cd81571 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:16.197438 1552 leader_election.cc:290] T 756cfe84a517459ca015aecf5cd81571 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:16.199188 991 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250901 01:52:16.199337 1332 log.cc:1199] T f0affc7a78634607852a48458412761d P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/f0affc7a78634607852a48458412761d
I20250901 01:52:16.200018 1103 heartbeater.cc:344] Connected to a master server at 127.0.104.60:44231
I20250901 01:52:16.200304 1332 ts_tablet_manager.cc:1939] T f0affc7a78634607852a48458412761d P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:16.200647 921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "756cfe84a517459ca015aecf5cd81571" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:16.200893 921 raft_consensus.cc:3058] T 756cfe84a517459ca015aecf5cd81571 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:16.204213 921 raft_consensus.cc:2466] T 756cfe84a517459ca015aecf5cd81571 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:16.204790 1332 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44246 (request call id 193) took 2209 ms. Trace:
I20250901 01:52:16.204983 1332 rpcz_store.cc:276] 0901 01:52:13.995202 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:13.999392 (+ 4190us) service_pool.cc:225] Handling call
0901 01:52:15.652301 (+1652909us) ts_tablet_manager.cc:1145] Deleting tablet f0affc7a78634607852a48458412761d
0901 01:52:15.652311 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:15.879048 (+226737us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:16.081974 (+202926us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:16.204721 (+122747us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1651584}
I20250901 01:52:16.206391 992 leader_election.cc:304] T 756cfe84a517459ca015aecf5cd81571 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:16.206691 1552 raft_consensus.cc:2802] T 756cfe84a517459ca015aecf5cd81571 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:16.206926 441 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet f0affc7a78634607852a48458412761d (table default.Table_b7bffcbb78614d3cad2b1d375d7a97cc [id=535cf5f83c56443d93935ce80ff87efb]) successfully deleted
I20250901 01:52:16.207029 1552 raft_consensus.cc:695] T 756cfe84a517459ca015aecf5cd81571 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:16.207377 1552 consensus_queue.cc:237] T 756cfe84a517459ca015aecf5cd81571 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:16.208516 1150 tablet_bootstrap.cc:492] T 4a87e68850c4459eb2c8eafac89c6aa9 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:16.208695 1150 ts_tablet_manager.cc:1397] T 4a87e68850c4459eb2c8eafac89c6aa9 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.107s user 0.004s sys 0.001s
I20250901 01:52:16.208963 1332 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44246 (request call id 206) took 2103 ms. Trace:
I20250901 01:52:16.209676 1150 raft_consensus.cc:357] T 4a87e68850c4459eb2c8eafac89c6aa9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:16.209895 1150 raft_consensus.cc:383] T 4a87e68850c4459eb2c8eafac89c6aa9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:16.210001 1150 raft_consensus.cc:738] T 4a87e68850c4459eb2c8eafac89c6aa9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:16.210340 1150 consensus_queue.cc:260] T 4a87e68850c4459eb2c8eafac89c6aa9 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:16.211339 1151 tablet_bootstrap.cc:492] T 22691943a134400c8df4617240e5d3ef P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:16.217051 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_97f8d5c204174c918a795498e47d404d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:16.218254 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_97f8d5c204174c918a795498e47d404d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:16.223960 1332 rpcz_store.cc:276] 0901 01:52:14.102733 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:14.103002 (+ 269us) service_pool.cc:225] Handling call
0901 01:52:16.205860 (+2102858us) ts_tablet_manager.cc:1145] Deleting tablet f50f33de1b654663b9ef8a91e3436f11
0901 01:52:16.205874 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:16.206481 (+ 607us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2102573}
I20250901 01:52:16.224644 1332 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44246 (request call id 208) took 2121 ms. Trace:
I20250901 01:52:16.226346 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_53e30205c5fd488eb3b48c685a159d04"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:16.226904 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_53e30205c5fd488eb3b48c685a159d04 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:16.227666 1293 raft_consensus.cc:491] T 49d30da95b14400b845270122487a2a0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:16.227886 1293 raft_consensus.cc:513] T 49d30da95b14400b845270122487a2a0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:16.228777 1293 leader_election.cc:290] T 49d30da95b14400b845270122487a2a0 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:16.229331 1332 rpcz_store.cc:276] 0901 01:52:14.102984 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:14.104472 (+ 1488us) service_pool.cc:225] Handling call
0901 01:52:16.224459 (+2119987us) ts_tablet_manager.cc:1145] Deleting tablet d60d571b2c2840a2b20629bb832d1309
0901 01:52:16.224475 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:16.224607 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2119790}
I20250901 01:52:16.229763 1332 tablet_replica.cc:331] T b29e846dfae44f0e9ce3e5fea3763a4f P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
W20250901 01:52:16.230607 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet f50f33de1b654663b9ef8a91e3436f11 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f50f33de1b654663b9ef8a91e3436f11 already in progress: creating tablet
W20250901 01:52:16.230931 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet d60d571b2c2840a2b20629bb832d1309 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d60d571b2c2840a2b20629bb832d1309 already in progress: creating tablet
I20250901 01:52:16.235482 1332 raft_consensus.cc:2241] T b29e846dfae44f0e9ce3e5fea3763a4f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:16.235742 1058 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "49d30da95b14400b845270122487a2a0" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:16.235903 1151 ts_tablet_manager.cc:1397] T 22691943a134400c8df4617240e5d3ef P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.217s user 0.009s sys 0.002s
I20250901 01:52:16.236795 1332 raft_consensus.cc:2270] T b29e846dfae44f0e9ce3e5fea3763a4f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:16.238478 1332 ts_tablet_manager.cc:1905] T b29e846dfae44f0e9ce3e5fea3763a4f P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:16.239331 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_d5a6ac03c9ff4969ada556d1318b64f0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:16.240072 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d5a6ac03c9ff4969ada556d1318b64f0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:16.244927 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_983ec443070b4ffc8508ea825e30471a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:16.246088 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_983ec443070b4ffc8508ea825e30471a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250901 01:52:16.248220 859 leader_election.cc:343] T 49d30da95b14400b845270122487a2a0 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:16.250406 453 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_c3570db2dfd14451aa98edf48f7b2a5e" } new_table_name: "default.Table_d2049d0c20bb4949ae6c249b690e7f86" modify_external_catalogs: true
I20250901 01:52:16.257668 1151 raft_consensus.cc:357] T 22691943a134400c8df4617240e5d3ef P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:16.257941 1151 raft_consensus.cc:383] T 22691943a134400c8df4617240e5d3ef P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:16.258019 1151 raft_consensus.cc:738] T 22691943a134400c8df4617240e5d3ef P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:16.258271 1151 consensus_queue.cc:260] T 22691943a134400c8df4617240e5d3ef P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:16.258770 1151 ts_tablet_manager.cc:1428] T 22691943a134400c8df4617240e5d3ef P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.022s user 0.001s sys 0.001s
I20250901 01:52:16.259116 1151 tablet_bootstrap.cc:492] T 1f18420261ae41459e1f25b927bd84a9 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:16.266321 1151 tablet_bootstrap.cc:654] T 1f18420261ae41459e1f25b927bd84a9 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:16.267537 1374 raft_consensus.cc:491] T 84fe314502444bd88b038b87eead9cb5 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:16.267711 1374 raft_consensus.cc:513] T 84fe314502444bd88b038b87eead9cb5 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:16.268362 1374 leader_election.cc:290] T 84fe314502444bd88b038b87eead9cb5 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:16.268870 994 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250901 01:52:16.272038 453 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3c12d6a70ac24dce82e851782e6fe5aa consensus state has no leader
I20250901 01:52:16.273141 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_f027ccc695a74d1fb4595d9614f30381"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:16.274843 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f027ccc695a74d1fb4595d9614f30381 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:16.278306 1626 tablet.cc:1722] T 4fd0e4f9d86c4362af96ce9964b72328 P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250901 01:52:16.280300 1622 tablet.cc:1722] T 4fd0e4f9d86c4362af96ce9964b72328 P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250901 01:52:16.284333 1611 raft_consensus.cc:491] T b633b461dbf34e13bc6e1cc08e6d20c1 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:16.284554 1611 raft_consensus.cc:513] T b633b461dbf34e13bc6e1cc08e6d20c1 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:16.285933 1611 leader_election.cc:290] T b633b461dbf34e13bc6e1cc08e6d20c1 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:16.286521 1058 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b633b461dbf34e13bc6e1cc08e6d20c1" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:16.286629 992 leader_election.cc:343] T 84fe314502444bd88b038b87eead9cb5 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 84fe314502444bd88b038b87eead9cb5
W20250901 01:52:16.287016 859 leader_election.cc:343] T b633b461dbf34e13bc6e1cc08e6d20c1 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:16.288127 1629 tablet.cc:1722] T 24ddc13d1bb04db4a7850acbff091577 P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250901 01:52:16.291301 1332 ts_tablet_manager.cc:1918] T b29e846dfae44f0e9ce3e5fea3763a4f P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:16.291472 1332 log.cc:1199] T b29e846dfae44f0e9ce3e5fea3763a4f P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/b29e846dfae44f0e9ce3e5fea3763a4f
I20250901 01:52:16.291906 1332 ts_tablet_manager.cc:1939] T b29e846dfae44f0e9ce3e5fea3763a4f P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:16.292054 1626 tablet.cc:1722] T 24ddc13d1bb04db4a7850acbff091577 P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250901 01:52:16.292258 1332 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44246 (request call id 207) took 2189 ms. Trace:
I20250901 01:52:16.297309 1628 tablet.cc:1722] T eeae5294bd474f368f5766c260ccf3fe P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250901 01:52:16.297704 1611 raft_consensus.cc:491] T 824229183e024feaa975c7f0387c9c1a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:16.297919 1611 raft_consensus.cc:513] T 824229183e024feaa975c7f0387c9c1a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:16.299052 1611 leader_election.cc:290] T 824229183e024feaa975c7f0387c9c1a P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:16.307914 1624 tablet.cc:1722] T eeae5294bd474f368f5766c260ccf3fe P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250901 01:52:16.312482 1058 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "824229183e024feaa975c7f0387c9c1a" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:16.292375 1332 rpcz_store.cc:276] 0901 01:52:14.102874 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:14.104240 (+ 1366us) service_pool.cc:225] Handling call
0901 01:52:16.229682 (+2125442us) ts_tablet_manager.cc:1145] Deleting tablet b29e846dfae44f0e9ce3e5fea3763a4f
0901 01:52:16.229697 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:16.243127 (+ 13430us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:16.291165 (+ 48038us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:16.292209 (+ 1044us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2124915}
#10 0x7fe79fd404b8 in std::function<kudu::Status (kudu::tools::RunnerContext const&)>::operator()(kudu::tools::RunnerContext const&) const ../../../include/c++/7.5.0/bits/std_function.h:706:14
#11 0x7fe79fd3bfd4 in kudu::tools::Action::Run(std::vector<kudu::tools::Mode*, std::allocator<kudu::tools::Mode*> > const&, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) const /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/tool_action.cc:245:10
W20250901 01:52:16.319859 859 leader_election.cc:343] T 824229183e024feaa975c7f0387c9c1a P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:16.320282 441 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet b29e846dfae44f0e9ce3e5fea3763a4f (table default.Table_1e00e9f110c7483bb1a077829c64f85b [id=4ef5558babf64de7930fadf065ef6866]) successfully deleted
I20250901 01:52:16.324043 1151 tablet_bootstrap.cc:492] T 1f18420261ae41459e1f25b927bd84a9 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:16.324190 1151 ts_tablet_manager.cc:1397] T 1f18420261ae41459e1f25b927bd84a9 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.065s user 0.003s sys 0.003s
I20250901 01:52:16.325042 1332 tablet_replica.cc:331] T d6a4040747fa4d588756506ee122d2dc P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:16.325259 1332 raft_consensus.cc:2241] T d6a4040747fa4d588756506ee122d2dc P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:16.330386 1151 raft_consensus.cc:357] T 1f18420261ae41459e1f25b927bd84a9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:16.330687 1151 raft_consensus.cc:383] T 1f18420261ae41459e1f25b927bd84a9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:16.331008 1151 raft_consensus.cc:738] T 1f18420261ae41459e1f25b927bd84a9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:16.331285 1151 consensus_queue.cc:260] T 1f18420261ae41459e1f25b927bd84a9 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:16.332267 1332 raft_consensus.cc:2270] T d6a4040747fa4d588756506ee122d2dc P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:16.336607 1151 ts_tablet_manager.cc:1428] T 1f18420261ae41459e1f25b927bd84a9 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.012s user 0.000s sys 0.002s
I20250901 01:52:16.337003 1151 tablet_bootstrap.cc:492] T 920925ee9b014e14a79ceb7796ff7828 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:16.340221 1151 tablet_bootstrap.cc:654] T 920925ee9b014e14a79ceb7796ff7828 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:16.341475 1332 ts_tablet_manager.cc:1905] T d6a4040747fa4d588756506ee122d2dc P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:16.360944 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_c298ecfa989347b0b7a4d7227611539f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:16.361469 1332 ts_tablet_manager.cc:1918] T d6a4040747fa4d588756506ee122d2dc P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250901 01:52:16.361845 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c298ecfa989347b0b7a4d7227611539f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:16.362644 1151 tablet_bootstrap.cc:492] T 920925ee9b014e14a79ceb7796ff7828 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:16.363334 453 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_612c3ccdf69d4dbd95aa76bdf404fc54" } new_table_name: "default.Table_83f2c0acedbc4c6982caf5313646e90b" modify_external_catalogs: true
I20250901 01:52:16.363703 1151 ts_tablet_manager.cc:1397] T 920925ee9b014e14a79ceb7796ff7828 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.027s user 0.000s sys 0.006s
I20250901 01:52:16.364606 1151 raft_consensus.cc:357] T 920925ee9b014e14a79ceb7796ff7828 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:16.365451 1151 raft_consensus.cc:383] T 920925ee9b014e14a79ceb7796ff7828 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:16.365602 1151 raft_consensus.cc:738] T 920925ee9b014e14a79ceb7796ff7828 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:16.365948 1151 consensus_queue.cc:260] T 920925ee9b014e14a79ceb7796ff7828 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:16.369014 1151 ts_tablet_manager.cc:1428] T 920925ee9b014e14a79ceb7796ff7828 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250901 01:52:16.369436 1151 tablet_bootstrap.cc:492] T c27bb23c150f4565a1d0ba8f231cc386 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:16.370222 1332 log.cc:1199] T d6a4040747fa4d588756506ee122d2dc P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/d6a4040747fa4d588756506ee122d2dc
I20250901 01:52:16.371315 1332 ts_tablet_manager.cc:1939] T d6a4040747fa4d588756506ee122d2dc P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:16.373313 1332 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44246 (request call id 214) took 2160 ms. Trace:
I20250901 01:52:16.373987 1332 rpcz_store.cc:276] 0901 01:52:14.212880 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:14.213264 (+ 384us) service_pool.cc:225] Handling call
0901 01:52:16.323332 (+2110068us) ts_tablet_manager.cc:1145] Deleting tablet d6a4040747fa4d588756506ee122d2dc
0901 01:52:16.323346 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:16.358013 (+ 34667us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:16.361313 (+ 3300us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:16.373266 (+ 11953us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2107063}
I20250901 01:52:16.374682 1332 tablet_replica.cc:331] T 2a0d125423a043dcb11e67345599e468 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:16.374871 441 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet d6a4040747fa4d588756506ee122d2dc (table default.Table_6400e401888045348632f924f636d011 [id=8c1b263824ec4c15a5ec424364c78895]) successfully deleted
I20250901 01:52:16.375173 1332 raft_consensus.cc:2241] T 2a0d125423a043dcb11e67345599e468 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:16.375348 1332 raft_consensus.cc:2270] T 2a0d125423a043dcb11e67345599e468 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:16.375963 1332 ts_tablet_manager.cc:1905] T 2a0d125423a043dcb11e67345599e468 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:16.372560 1151 tablet_bootstrap.cc:654] T c27bb23c150f4565a1d0ba8f231cc386 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:16.384745 1150 ts_tablet_manager.cc:1428] T 4a87e68850c4459eb2c8eafac89c6aa9 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.176s user 0.002s sys 0.000s
I20250901 01:52:16.385228 1150 tablet_bootstrap.cc:492] T 533d14f5903e49599a34ea353ffd8c00 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
W20250901 01:52:16.389510 441 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 72ba689f9230411995dc96e2b45601df (table default.Table_604f2c5c58d74154bacf2a065a467f32 [id=351a05134f4948039c920cafbe050914], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 72ba689f9230411995dc96e2b45601df consensus state has no leader
I20250901 01:52:16.389941 1151 tablet_bootstrap.cc:492] T c27bb23c150f4565a1d0ba8f231cc386 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:16.390127 1151 ts_tablet_manager.cc:1397] T c27bb23c150f4565a1d0ba8f231cc386 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.005s
I20250901 01:52:16.391026 1151 raft_consensus.cc:357] T c27bb23c150f4565a1d0ba8f231cc386 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:16.391214 1151 raft_consensus.cc:383] T c27bb23c150f4565a1d0ba8f231cc386 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:16.391294 1151 raft_consensus.cc:738] T c27bb23c150f4565a1d0ba8f231cc386 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:16.391534 1151 consensus_queue.cc:260] T c27bb23c150f4565a1d0ba8f231cc386 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:16.395747 1151 ts_tablet_manager.cc:1428] T c27bb23c150f4565a1d0ba8f231cc386 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20250901 01:52:16.396080 1151 tablet_bootstrap.cc:492] T f65ec84958eb488d8b0f812b59b5b3d5 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:16.397848 1151 tablet_bootstrap.cc:654] T f65ec84958eb488d8b0f812b59b5b3d5 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:16.399853 1617 raft_consensus.cc:491] T 6c15cd14831f422eb615038603c79ed0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:16.400003 1617 raft_consensus.cc:513] T 6c15cd14831f422eb615038603c79ed0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:16.400727 1617 leader_election.cc:290] T 6c15cd14831f422eb615038603c79ed0 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:16.401746 992 leader_election.cc:343] T 6c15cd14831f422eb615038603c79ed0 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 6c15cd14831f422eb615038603c79ed0
W20250901 01:52:16.404193 453 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3618a3eb4f174c43805ce7dc0e5315cb has no consensus state
I20250901 01:52:16.406081 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_c2ed44ae561c4b88b22e8975a61b77aa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:16.407866 1332 ts_tablet_manager.cc:1918] T 2a0d125423a043dcb11e67345599e468 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:16.408752 1332 log.cc:1199] T 2a0d125423a043dcb11e67345599e468 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/2a0d125423a043dcb11e67345599e468
I20250901 01:52:16.410351 1332 ts_tablet_manager.cc:1939] T 2a0d125423a043dcb11e67345599e468 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:16.412626 1332 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44246 (request call id 215) took 2172 ms. Trace:
W20250901 01:52:16.413614 1572 log.cc:927] Time spent T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Append to log took a long time: real 0.105s user 0.003s sys 0.000s
I20250901 01:52:16.413084 1332 rpcz_store.cc:276] 0901 01:52:14.240294 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:14.240352 (+ 58us) service_pool.cc:225] Handling call
0901 01:52:16.374613 (+2134261us) ts_tablet_manager.cc:1145] Deleting tablet 2a0d125423a043dcb11e67345599e468
0901 01:52:16.374622 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:16.379389 (+ 4767us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:16.407338 (+ 27949us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:16.412360 (+ 5022us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2133964}
I20250901 01:52:16.414691 441 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 2a0d125423a043dcb11e67345599e468 (table default.Table_b27678f366d74143b716da42e7ca32c3 [id=500406ff1eb342f6af09fbb6f818e160]) successfully deleted
I20250901 01:52:16.415594 1332 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44246 (request call id 213) took 2202 ms. Trace:
I20250901 01:52:16.416145 1332 rpcz_store.cc:276] 0901 01:52:14.212748 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:14.284574 (+ 71826us) service_pool.cc:225] Handling call
0901 01:52:16.415071 (+2130497us) ts_tablet_manager.cc:1145] Deleting tablet ef58f477ebeb49639ed7d071e1a62c10
0901 01:52:16.415082 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:16.415455 (+ 373us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2130072}
W20250901 01:52:16.417464 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet ef58f477ebeb49639ed7d071e1a62c10 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ef58f477ebeb49639ed7d071e1a62c10 already in progress: creating tablet
I20250901 01:52:16.418133 1332 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44246 (request call id 216) took 2024 ms. Trace:
I20250901 01:52:16.418833 454 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_c857a6bd1be2434d861d57fea5bf4c1a" } new_table_name: "default.Table_4a412a59b6db421eb0187ffd02ecf541" modify_external_catalogs: true
I20250901 01:52:16.418735 1332 rpcz_store.cc:276] 0901 01:52:14.393640 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:14.393704 (+ 64us) service_pool.cc:225] Handling call
0901 01:52:16.417647 (+2023943us) ts_tablet_manager.cc:1145] Deleting tablet 121803e79ab64ec0b948b448646b9d0a
0901 01:52:16.417657 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:16.417917 (+ 260us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1885048}
I20250901 01:52:16.420439 1617 consensus_queue.cc:1035] T 467c2bf4a8cb4937b6f627bea28d8ddb P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:16.420526 1332 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44246 (request call id 217) took 1934 ms. Trace:
I20250901 01:52:16.420794 1332 rpcz_store.cc:276] 0901 01:52:14.486208 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:14.486317 (+ 109us) service_pool.cc:225] Handling call
0901 01:52:16.419996 (+1933679us) ts_tablet_manager.cc:1145] Deleting tablet 5a8c21aee9f84216b6f90538a5bf4c17
0901 01:52:16.420005 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:16.420270 (+ 265us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1885689}
I20250901 01:52:16.422597 1332 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44246 (request call id 218) took 1909 ms. Trace:
I20250901 01:52:16.422935 1332 rpcz_store.cc:276] 0901 01:52:14.513326 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:14.513407 (+ 81us) service_pool.cc:225] Handling call
0901 01:52:16.422063 (+1908656us) ts_tablet_manager.cc:1145] Deleting tablet 66a48a8ebe154b2cac89c234eaa378ff
0901 01:52:16.422075 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:16.422456 (+ 381us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1887345}
I20250901 01:52:16.424384 1332 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44246 (request call id 219) took 1901 ms. Trace:
I20250901 01:52:16.424881 1332 rpcz_store.cc:276] 0901 01:52:14.522878 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:14.522956 (+ 78us) service_pool.cc:225] Handling call
0901 01:52:16.423764 (+1900808us) ts_tablet_manager.cc:1145] Deleting tablet 82d163681fda47f38d79863bd28f576d
0901 01:52:16.423868 (+ 104us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:16.424152 (+ 284us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1888011}
I20250901 01:52:16.426332 1332 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44246 (request call id 220) took 1891 ms. Trace:
I20250901 01:52:16.426764 1332 rpcz_store.cc:276] 0901 01:52:14.534366 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:14.535799 (+ 1433us) service_pool.cc:225] Handling call
0901 01:52:16.425919 (+1890120us) ts_tablet_manager.cc:1145] Deleting tablet 121803e79ab64ec0b948b448646b9d0a
0901 01:52:16.425928 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:16.426185 (+ 257us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1889824}
I20250901 01:52:16.428519 1332 tablet_replica.cc:331] T 2685fb7258264463b5f1ec99cd7d67ad P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:16.429162 1332 raft_consensus.cc:2241] T 2685fb7258264463b5f1ec99cd7d67ad P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:16.429777 1332 raft_consensus.cc:2270] T 2685fb7258264463b5f1ec99cd7d67ad P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:16.432078 1332 ts_tablet_manager.cc:1905] T 2685fb7258264463b5f1ec99cd7d67ad P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:16.451992 1306 ts_tablet_manager.cc:1918] T 60285f2ba037433999f5455fb9cc7acd P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250901 01:52:16.454802 992 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250901 01:52:16.420553 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 121803e79ab64ec0b948b448646b9d0a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 121803e79ab64ec0b948b448646b9d0a already in progress: creating tablet
W20250901 01:52:16.456625 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 5a8c21aee9f84216b6f90538a5bf4c17 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5a8c21aee9f84216b6f90538a5bf4c17 already in progress: creating tablet
W20250901 01:52:16.456915 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 66a48a8ebe154b2cac89c234eaa378ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 66a48a8ebe154b2cac89c234eaa378ff already in progress: creating tablet
W20250901 01:52:16.457181 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 121803e79ab64ec0b948b448646b9d0a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 121803e79ab64ec0b948b448646b9d0a already in progress: creating tablet
I20250901 01:52:16.464198 1332 ts_tablet_manager.cc:1918] T 2685fb7258264463b5f1ec99cd7d67ad P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:16.464874 1332 log.cc:1199] T 2685fb7258264463b5f1ec99cd7d67ad P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/2685fb7258264463b5f1ec99cd7d67ad
W20250901 01:52:16.466295 454 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b2e8b2cd10154de08ddd30c214592a4a consensus state has no leader
W20250901 01:52:16.452221 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c2ed44ae561c4b88b22e8975a61b77aa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:16.468077 1332 ts_tablet_manager.cc:1939] T 2685fb7258264463b5f1ec99cd7d67ad P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:16.469398 1332 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44246 (request call id 224) took 1860 ms. Trace:
I20250901 01:52:16.469789 1332 rpcz_store.cc:276] 0901 01:52:14.609078 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:14.609134 (+ 56us) service_pool.cc:225] Handling call
0901 01:52:16.428308 (+1819174us) ts_tablet_manager.cc:1145] Deleting tablet 2685fb7258264463b5f1ec99cd7d67ad
0901 01:52:16.428315 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:16.446597 (+ 18282us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:16.463395 (+ 16798us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:16.469273 (+ 5878us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1818971}
I20250901 01:52:16.471850 1332 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44246 (request call id 227) took 1818 ms. Trace:
I20250901 01:52:16.472512 441 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 2685fb7258264463b5f1ec99cd7d67ad (table default.Table_dcacb44eed4241d3acbae153ebaad663 [id=0241aa3ab1984557a151461005c6b97c]) successfully deleted
I20250901 01:52:16.472177 1332 rpcz_store.cc:276] 0901 01:52:14.652840 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:14.652919 (+ 79us) service_pool.cc:225] Handling call
0901 01:52:16.471299 (+1818380us) ts_tablet_manager.cc:1145] Deleting tablet b633b461dbf34e13bc6e1cc08e6d20c1
0901 01:52:16.471309 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:16.471733 (+ 424us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1818106}
W20250901 01:52:16.474881 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet b633b461dbf34e13bc6e1cc08e6d20c1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b633b461dbf34e13bc6e1cc08e6d20c1 already in progress: creating tablet
I20250901 01:52:16.475062 1332 tablet_replica.cc:331] T 360755d30b5d47ba98c9c7e23ff5ab7e P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:16.476094 1332 raft_consensus.cc:2241] T 360755d30b5d47ba98c9c7e23ff5ab7e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:16.476922 1332 raft_consensus.cc:2270] T 360755d30b5d47ba98c9c7e23ff5ab7e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:16.478137 453 master_service.cc:948] ReplaceTablet: received request to replace tablet 4855448edb4b40c184838aa7f81c112b from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:16.480369 1332 ts_tablet_manager.cc:1905] T 360755d30b5d47ba98c9c7e23ff5ab7e P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
#12 0xbb3ed3 in kudu::tools::DispatchCommand(std::vector<kudu::tools::Mode*, std::allocator<kudu::tools::Mode*> > const&, kudu::tools::Action*, std::deque<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/tool_main.cc:142:15
I20250901 01:52:16.501797 1151 tablet_bootstrap.cc:492] T f65ec84958eb488d8b0f812b59b5b3d5 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:16.502048 1151 ts_tablet_manager.cc:1397] T f65ec84958eb488d8b0f812b59b5b3d5 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.106s user 0.015s sys 0.026s
I20250901 01:52:16.503141 1151 raft_consensus.cc:357] T f65ec84958eb488d8b0f812b59b5b3d5 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:16.503420 1151 raft_consensus.cc:383] T f65ec84958eb488d8b0f812b59b5b3d5 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:16.503543 1151 raft_consensus.cc:738] T f65ec84958eb488d8b0f812b59b5b3d5 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:16.501353 454 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_58fc08621bb54011abf9c7484939eb75" } new_table_name: "default.Table_0f959237963c4e33932e193285c0eacb" modify_external_catalogs: true
I20250901 01:52:16.503947 1151 consensus_queue.cc:260] T f65ec84958eb488d8b0f812b59b5b3d5 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:16.507592 1332 ts_tablet_manager.cc:1918] T 360755d30b5d47ba98c9c7e23ff5ab7e P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:16.508109 1332 log.cc:1199] T 360755d30b5d47ba98c9c7e23ff5ab7e P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/360755d30b5d47ba98c9c7e23ff5ab7e
I20250901 01:52:16.508886 1151 ts_tablet_manager.cc:1428] T f65ec84958eb488d8b0f812b59b5b3d5 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20250901 01:52:16.509405 1332 ts_tablet_manager.cc:1939] T 360755d30b5d47ba98c9c7e23ff5ab7e P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:16.510990 1332 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44246 (request call id 228) took 1846 ms. Trace:
I20250901 01:52:16.511385 1332 rpcz_store.cc:276] 0901 01:52:14.664878 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:14.667808 (+ 2930us) service_pool.cc:225] Handling call
0901 01:52:16.474808 (+1807000us) ts_tablet_manager.cc:1145] Deleting tablet 360755d30b5d47ba98c9c7e23ff5ab7e
0901 01:52:16.474821 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:16.493623 (+ 18802us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:16.507215 (+ 13592us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:16.510861 (+ 3646us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1806649}
I20250901 01:52:16.512866 1151 tablet_bootstrap.cc:492] T ceb724d859d24d569598e18389dae3b3 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:16.513227 441 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 360755d30b5d47ba98c9c7e23ff5ab7e (table default.Table_64b711e409d7434c8b176bf1a3ef8c86 [id=a762733ef34449a5a7a3898464cd8045]) successfully deleted
I20250901 01:52:16.513866 1332 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44246 (request call id 229) took 1834 ms. Trace:
I20250901 01:52:16.514245 1332 rpcz_store.cc:276] 0901 01:52:14.679776 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:14.680143 (+ 367us) service_pool.cc:225] Handling call
0901 01:52:16.513151 (+1833008us) ts_tablet_manager.cc:1145] Deleting tablet b6daeff0070a4db08e90020752d21c29
0901 01:52:16.513165 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:16.513712 (+ 547us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1832674}
I20250901 01:52:16.515995 1332 tablet_replica.cc:331] T 0e698923812841b7a5d37d57da103c5c P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
W20250901 01:52:16.516085 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet b6daeff0070a4db08e90020752d21c29 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b6daeff0070a4db08e90020752d21c29 already in progress: creating tablet
I20250901 01:52:16.517038 1332 raft_consensus.cc:2241] T 0e698923812841b7a5d37d57da103c5c P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:16.518244 1332 raft_consensus.cc:2270] T 0e698923812841b7a5d37d57da103c5c P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:16.521124 1332 ts_tablet_manager.cc:1905] T 0e698923812841b7a5d37d57da103c5c P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:16.525442 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 0cb3262973934c8689b851adad60afe0
I20250901 01:52:16.526316 451 master_service.cc:948] ReplaceTablet: received request to replace tablet b73e362321284df1b18ef755dfe930a5 from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:16.527103 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_add9fa2d6db444b6896dd970425259ae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:16.527915 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_add9fa2d6db444b6896dd970425259ae in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:16.533774 1038 tablet_service.cc:1515] Processing DeleteTablet for tablet 0cb3262973934c8689b851adad60afe0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:16 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:16.534202 1151 tablet_bootstrap.cc:654] T ceb724d859d24d569598e18389dae3b3 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:16.545971 451 master_service.cc:948] ReplaceTablet: received request to replace tablet 579b9cdf819f4609a532e96bec093fd5 from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:16.554284 446 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 1286c0ab85704837a99b6e5a62184d77
I20250901 01:52:16.555260 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_8770c960d4bd4c49b043ffe889b4b87b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:16.556062 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8770c960d4bd4c49b043ffe889b4b87b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:16.558171 1038 tablet_service.cc:1515] Processing DeleteTablet for tablet 1286c0ab85704837a99b6e5a62184d77 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:16 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:16.561090 1332 ts_tablet_manager.cc:1918] T 0e698923812841b7a5d37d57da103c5c P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:16.561960 1332 log.cc:1199] T 0e698923812841b7a5d37d57da103c5c P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/0e698923812841b7a5d37d57da103c5c
I20250901 01:52:16.566216 1332 ts_tablet_manager.cc:1939] T 0e698923812841b7a5d37d57da103c5c P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:16.571195 1332 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44246 (request call id 232) took 1829 ms. Trace:
I20250901 01:52:16.572391 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_cb2cf28c0e004f7daa4fd83bc027bfaa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:16.573470 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cb2cf28c0e004f7daa4fd83bc027bfaa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:16.572299 1332 rpcz_store.cc:276] 0901 01:52:14.741241 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:14.741287 (+ 46us) service_pool.cc:225] Handling call
0901 01:52:16.515562 (+1774275us) ts_tablet_manager.cc:1145] Deleting tablet 0e698923812841b7a5d37d57da103c5c
0901 01:52:16.515578 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:16.544293 (+ 28715us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:16.560073 (+ 15780us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:16.570799 (+ 10726us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1773956}
I20250901 01:52:16.575141 441 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 0e698923812841b7a5d37d57da103c5c (table default.Table_68682089b6364db98892ca5d82ed1322 [id=83600209eb4649ec9f7842507aa8a1ff]) successfully deleted
I20250901 01:52:16.577776 1332 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44246 (request call id 233) took 1818 ms. Trace:
I20250901 01:52:16.579818 1332 rpcz_store.cc:276] 0901 01:52:14.759051 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:14.784873 (+ 25822us) service_pool.cc:225] Handling call
0901 01:52:16.575925 (+1791052us) ts_tablet_manager.cc:1145] Deleting tablet ceb724d859d24d569598e18389dae3b3
0901 01:52:16.576079 (+ 154us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:16.577564 (+ 1485us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1790585}
W20250901 01:52:16.583005 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet ceb724d859d24d569598e18389dae3b3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ceb724d859d24d569598e18389dae3b3 already in progress: creating tablet
I20250901 01:52:16.584818 1332 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44246 (request call id 237) took 1674 ms. Trace:
I20250901 01:52:16.586763 1332 rpcz_store.cc:276] 0901 01:52:14.909791 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:14.910142 (+ 351us) service_pool.cc:225] Handling call
0901 01:52:16.583689 (+1673547us) ts_tablet_manager.cc:1145] Deleting tablet d77aadd7291647a0bae7c421f244c252
0901 01:52:16.584136 (+ 447us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:16.584604 (+ 468us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1673101}
I20250901 01:52:16.588984 446 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_6400e401888045348632f924f636d011" } new_table_name: "default.Table_531b3137a19f4061878ea96aae227517" modify_external_catalogs: true
W20250901 01:52:16.589378 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet d77aadd7291647a0bae7c421f244c252 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d77aadd7291647a0bae7c421f244c252 already in progress: creating tablet
I20250901 01:52:16.591401 1332 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44246 (request call id 236) took 1693 ms. Trace:
I20250901 01:52:16.592763 1332 rpcz_store.cc:276] 0901 01:52:14.897653 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:14.897730 (+ 77us) service_pool.cc:225] Handling call
0901 01:52:16.590038 (+1692308us) ts_tablet_manager.cc:1145] Deleting tablet f8a5886ddb644a86b76df44aab309057
0901 01:52:16.590220 (+ 182us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:16.591031 (+ 811us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1678583}
W20250901 01:52:16.595077 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet f8a5886ddb644a86b76df44aab309057 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f8a5886ddb644a86b76df44aab309057 already in progress: creating tablet
I20250901 01:52:16.597076 1332 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44246 (request call id 248) took 1371 ms. Trace:
I20250901 01:52:16.598069 1332 rpcz_store.cc:276] 0901 01:52:15.224728 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:15.224794 (+ 66us) service_pool.cc:225] Handling call
0901 01:52:16.595449 (+1370655us) ts_tablet_manager.cc:1145] Deleting tablet 9d886fa63f8f4996b087997b542e8bb1
0901 01:52:16.595637 (+ 188us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:16.596461 (+ 824us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1370206}
W20250901 01:52:16.600020 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 9d886fa63f8f4996b087997b542e8bb1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9d886fa63f8f4996b087997b542e8bb1 already in progress: creating tablet
W20250901 01:52:16.600325 446 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d0c3c363dda444519a17dca63ad9caa7 consensus state has no leader
I20250901 01:52:16.601480 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_ad2bde750a6f4c9aad14a28a9a0848c9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:16.601617 1332 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44246 (request call id 249) took 1330 ms. Trace:
I20250901 01:52:16.602689 1332 rpcz_store.cc:276] 0901 01:52:15.270783 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:15.270852 (+ 69us) service_pool.cc:225] Handling call
0901 01:52:16.600604 (+1329752us) ts_tablet_manager.cc:1145] Deleting tablet c763d252598745a088344aa0eb5fa3d3
0901 01:52:16.600784 (+ 180us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:16.601410 (+ 626us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1326783}
I20250901 01:52:16.606191 1332 tablet_replica.cc:331] T 920925ee9b014e14a79ceb7796ff7828 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:16.608381 1332 raft_consensus.cc:2241] T 920925ee9b014e14a79ceb7796ff7828 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:16.609900 1332 raft_consensus.cc:2270] T 920925ee9b014e14a79ceb7796ff7828 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:16.612473 1332 ts_tablet_manager.cc:1905] T 920925ee9b014e14a79ceb7796ff7828 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
#13 0xbb55b9 in kudu::tools::RunTool(int, char**, bool) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/tool_main.cc:214:16
I20250901 01:52:16.623026 1332 ts_tablet_manager.cc:1918] T 920925ee9b014e14a79ceb7796ff7828 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:16.623174 1332 log.cc:1199] T 920925ee9b014e14a79ceb7796ff7828 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/920925ee9b014e14a79ceb7796ff7828
I20250901 01:52:16.623359 1306 log.cc:1199] T 60285f2ba037433999f5455fb9cc7acd P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/60285f2ba037433999f5455fb9cc7acd
I20250901 01:52:16.623571 1332 ts_tablet_manager.cc:1939] T 920925ee9b014e14a79ceb7796ff7828 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:16.624451 1332 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44246 (request call id 250) took 1318 ms. Trace:
W20250901 01:52:16.605383 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet c763d252598745a088344aa0eb5fa3d3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c763d252598745a088344aa0eb5fa3d3 already in progress: creating tablet
I20250901 01:52:16.610538 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 25ce0fe8c9544f828694184d19d13df9
I20250901 01:52:16.625991 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_88d759cd5be3472088b92a5efe832a70"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:16.626713 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_88d759cd5be3472088b92a5efe832a70 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:16.624703 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_2b74c001cb4f4d82ab314557f4f09422"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:16.629452 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2b74c001cb4f4d82ab314557f4f09422 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:16.626384 1552 raft_consensus.cc:491] T 22691943a134400c8df4617240e5d3ef P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:16.630941 1552 raft_consensus.cc:513] T 22691943a134400c8df4617240e5d3ef P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:16.631685 1552 leader_election.cc:290] T 22691943a134400c8df4617240e5d3ef P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:16.631866 1332 rpcz_store.cc:276] 0901 01:52:15.305875 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:15.305946 (+ 71us) service_pool.cc:225] Handling call
0901 01:52:16.605591 (+1299645us) ts_tablet_manager.cc:1145] Deleting tablet 920925ee9b014e14a79ceb7796ff7828
0901 01:52:16.605780 (+ 189us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:16.619221 (+ 13441us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:16.622900 (+ 3679us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:16.624356 (+ 1456us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1299215}
I20250901 01:52:16.632025 1038 tablet_service.cc:1515] Processing DeleteTablet for tablet 25ce0fe8c9544f828694184d19d13df9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:16 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:16.632743 1332 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44246 (request call id 253) took 1220 ms. Trace:
I20250901 01:52:16.632841 1332 rpcz_store.cc:276] 0901 01:52:15.412574 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:15.412638 (+ 64us) service_pool.cc:225] Handling call
0901 01:52:16.632540 (+1219902us) ts_tablet_manager.cc:1145] Deleting tablet b73e362321284df1b18ef755dfe930a5
0901 01:52:16.632553 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:16.632680 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1206492}
I20250901 01:52:16.634106 1332 tablet_replica.cc:331] T 88a6ab29d00542aba54f0e4185fbf615 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:16.634094 441 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 920925ee9b014e14a79ceb7796ff7828 (table default.Table_537be4a82c0348f6952b911b167d02ad [id=8199118617cb4707bc3ae9b1363750f7]) successfully deleted
I20250901 01:52:16.634289 1332 raft_consensus.cc:2241] T 88a6ab29d00542aba54f0e4185fbf615 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250901 01:52:16.634344 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet b73e362321284df1b18ef755dfe930a5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b73e362321284df1b18ef755dfe930a5 already in progress: creating tablet
I20250901 01:52:16.634485 1332 raft_consensus.cc:2270] T 88a6ab29d00542aba54f0e4185fbf615 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250901 01:52:16.634608 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 3fda93767e494a4684978c8522b829dd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3fda93767e494a4684978c8522b829dd already in progress: creating tablet
W20250901 01:52:16.635044 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 23af154a6f1c4e2e9352fb17c9024f10 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23af154a6f1c4e2e9352fb17c9024f10 already in progress: creating tablet
I20250901 01:52:16.635075 1332 ts_tablet_manager.cc:1905] T 88a6ab29d00542aba54f0e4185fbf615 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:16.635288 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet abe27e749fa0485195e0064a49d5431e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet abe27e749fa0485195e0064a49d5431e already in progress: creating tablet
W20250901 01:52:16.635538 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 7f64ebd080944190be2e64d02d70a8a7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7f64ebd080944190be2e64d02d70a8a7 already in progress: creating tablet
W20250901 01:52:16.635658 992 leader_election.cc:343] T 22691943a134400c8df4617240e5d3ef P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 22691943a134400c8df4617240e5d3ef
W20250901 01:52:16.635774 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet c95798f0daf4437db18a291c4464522e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c95798f0daf4437db18a291c4464522e already in progress: creating tablet
W20250901 01:52:16.636015 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet b541a6e14f114ab29a864cca11806316 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b541a6e14f114ab29a864cca11806316 already in progress: creating tablet
W20250901 01:52:16.638981 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ad2bde750a6f4c9aad14a28a9a0848c9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
#14 0xbb6357 in main /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/tool_main.cc:309:10
#15 0x7fe787c9abf6 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21bf6)
Thread T138 (tablet-delete [) created by T55 (rpc worker-755) here:
I20250901 01:52:16.495065 1611 raft_consensus.cc:491] T 4a87e68850c4459eb2c8eafac89c6aa9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:16.644729 1611 raft_consensus.cc:513] T 4a87e68850c4459eb2c8eafac89c6aa9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:16.645661 1611 leader_election.cc:290] T 4a87e68850c4459eb2c8eafac89c6aa9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:16.512773 858 leader_election.cc:336] T 8f2e702121c845e0a09a254bbb81c142 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Timed out: RequestConsensusVote RPC to 127.0.104.1:40731 timed out after 3.356s (SENT)
#0 0x8920f0 in pthread_create /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/asan/asan_interceptors.cpp:214
I20250901 01:52:16.649734 1332 ts_tablet_manager.cc:1918] T 88a6ab29d00542aba54f0e4185fbf615 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:16.649894 1306 ts_tablet_manager.cc:1939] T 60285f2ba037433999f5455fb9cc7acd P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:16.650805 1058 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a87e68850c4459eb2c8eafac89c6aa9" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:16.651932 1332 log.cc:1199] T 88a6ab29d00542aba54f0e4185fbf615 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/88a6ab29d00542aba54f0e4185fbf615
I20250901 01:52:16.652312 1332 ts_tablet_manager.cc:1939] T 88a6ab29d00542aba54f0e4185fbf615 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
W20250901 01:52:16.656466 992 leader_election.cc:336] T c297e8bf11ff467a854eab41117ac376 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Timed out: RequestConsensusVote RPC to 127.0.104.1:40731 timed out after 3.122s (SENT)
W20250901 01:52:16.662600 991 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250901 01:52:16.677982 1552 raft_consensus.cc:491] T c27bb23c150f4565a1d0ba8f231cc386 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:16.678224 1552 raft_consensus.cc:513] T c27bb23c150f4565a1d0ba8f231cc386 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:16.678937 1552 leader_election.cc:290] T c27bb23c150f4565a1d0ba8f231cc386 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
#1 0x7fe78bc459c0 in kudu::Thread::StartThread(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::function<void ()>, unsigned long, scoped_refptr<kudu::Thread>*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:643:15
#2 0xa63f96 in kudu::Thread::Create(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::function<void ()>, scoped_refptr<kudu::Thread>*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.h:147:12
#3 0x7fe78bc6580a in kudu::ThreadPool::CreateThread() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/threadpool.cc:814:10
#4 0x7fe78bc6122c in kudu::ThreadPool::DoSubmit(std::function<void ()>, kudu::ThreadPoolToken*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/threadpool.cc:599:21
#5 0x7fe78bc6655e in kudu::ThreadPool::Submit(std::function<void ()>) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/threadpool.cc:491:10
#6 0x7fe79b49d225 in kudu::tserver::TSTabletManager::DeleteTabletAsync(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, kudu::tablet::TabletDataState, std::optional<long> const&, std::function<void (kudu::Status const&, kudu::tserver::TabletServerErrorPB_Code)> const&) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tserver/ts_tablet_manager.cc:1121:35
#7 0x7fe79b3e7ecf in kudu::tserver::TabletServiceAdminImpl::DeleteTablet(kudu::tserver::DeleteTabletRequestPB const*, kudu::tserver::DeleteTabletResponsePB*, kudu::rpc::RpcContext*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tserver/tablet_service.cc:1533:30
#8 0x7fe7955e6506 in kudu::tserver::TabletServerAdminServiceIf::TabletServerAdminServiceIf(scoped_refptr<kudu::MetricEntity> const&, scoped_refptr<kudu::rpc::ResultTracker> const&)::$_3::operator()(google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*) const /home/jenkins-slave/workspace/build_and_test_flaky@2/build/asan/src/kudu/tserver/tserver_admin.service.cc:215:13
#9 0x7fe7955e6235 in std::_Function_handler<void (google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*), kudu::tserver::TabletServerAdminServiceIf::TabletServerAdminServiceIf(scoped_refptr<kudu::MetricEntity> const&, scoped_refptr<kudu::rpc::ResultTracker> const&)::$_3>::_M_invoke(std::_Any_data const&, google::protobuf::Message const*&&, google::protobuf::Message*&&, kudu::rpc::RpcContext*&&) ../../../include/c++/7.5.0/bits/std_function.h:316:2
#10 0x7fe7929cf2ce in std::function<void (google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*)>::operator()(google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*) const ../../../include/c++/7.5.0/bits/std_function.h:706:14
#11 0x7fe7929ce1ae in kudu::rpc::GeneratedServiceIf::Handle(kudu::rpc::InboundCall*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/service_if.cc:137:3
#12 0x7fe7929d43e5 in kudu::rpc::ServicePool::RunThread() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/service_pool.cc:229:15
#13 0x7fe7929d4d0f in kudu::rpc::ServicePool::Init(int)::$_0::operator()() const /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/service_pool.cc:92:5
#14 0x7fe7929d4b38 in std::_Function_handler<void (), kudu::rpc::ServicePool::Init(int)::$_0>::_M_invoke(std::_Any_data const&) ../../../include/c++/7.5.0/bits/std_function.h:316:2
#15 0xa774dc in std::function<void ()>::operator()() const ../../../include/c++/7.5.0/bits/std_function.h:706:14
#16 0x7fe78bc46cc8 in kudu::Thread::SuperviseThread(void*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:694:3
#17 0x7fe78f0686da in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76da)
Thread T55 (rpc worker-755) created by T0 here:
#0 0x8920f0 in pthread_create /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/asan/asan_interceptors.cpp:214
#1 0x7fe78bc459c0 in kudu::Thread::StartThread(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::function<void ()>, unsigned long, scoped_refptr<kudu::Thread>*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:643:15
#2 0xa63f96 in kudu::Thread::Create(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::function<void ()>, scoped_refptr<kudu::Thread>*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.h:147:12
#3 0x7fe7929d12e2 in kudu::rpc::ServicePool::Init(int) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/service_pool.cc:92:5
#4 0x7fe794e79830 in kudu::RpcServer::RegisterService(std::unique_ptr<kudu::rpc::ServiceIf, std::default_delete<kudu::rpc::ServiceIf> >) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/server/rpc_server.cc:238:3
#5 0x7fe794e9ac47 in kudu::server::ServerBase::RegisterService(std::unique_ptr<kudu::rpc::ServiceIf, std::default_delete<kudu::rpc::ServiceIf> >) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/server/server_base.cc:1184:23
#6 0x7fe79b3cb5b3 in kudu::tserver::TabletServer::Start() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tserver/tablet_server.cc:157:3
#7 0x7fe79b3daa20 in kudu::tserver::RunTabletServer() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tserver/tablet_server_runner.cc:86:3
#8 0xb804d6 in kudu::tools::(anonymous namespace)::TServerRun(kudu::tools::RunnerContext const&) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/tool_action_tserver.cc:114:10
#9 0x9ab140 in std::_Function_handler<kudu::Status (kudu::tools::RunnerContext const&), kudu::Status (*)(kudu::tools::RunnerContext const&)>::_M_invoke(std::_Any_data const&, kudu::tools::RunnerContext const&) ../../../include/c++/7.5.0/bits/std_function.h:301:9
#10 0x7fe79fd404b8 in std::function<kudu::Status (kudu::tools::RunnerContext const&)>::operator()(kudu::tools::RunnerContext const&) const ../../../include/c++/7.5.0/bits/std_function.h:706:14
#11 0x7fe79fd3bfd4 in kudu::tools::Action::Run(std::vector<kudu::tools::Mode*, std::allocator<kudu::tools::Mode*> > const&, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) const /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/tool_action.cc:245:10
#12 0xbb3ed3 in kudu::tools::DispatchCommand(std::vector<kudu::tools::Mode*, std::allocator<kudu::tools::Mode*> > const&, kudu::tools::Action*, std::deque<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/tool_main.cc:142:15
#13 0xbb55b9 in kudu::tools::RunTool(int, char**, bool) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/tool_main.cc:214:16
#14 0xbb6357 in main /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/tool_main.cc:309:10
#15 0x7fe787c9abf6 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21bf6)
Thread T135 (tablet-open [wo) created by T62 (rpc worker-762) here:
#0 0x8920f0 in pthread_create /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/asan/asan_interceptors.cpp:214
#1 0x7fe78bc459c0 in kudu::Thread::StartThread(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::function<void ()>, unsigned long, scoped_refptr<kudu::Thread>*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:643:15
#2 0xa63f96 in kudu::Thread::Create(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::function<void ()>, scoped_refptr<kudu::Thread>*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.h:147:12
#3 0x7fe78bc6580a in kudu::ThreadPool::CreateThread() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/threadpool.cc:814:10
#4 0x7fe78bc6122c in kudu::ThreadPool::DoSubmit(std::function<void ()>, kudu::ThreadPoolToken*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/threadpool.cc:599:21
#5 0x7fe78bc6655e in kudu::ThreadPool::Submit(std::function<void ()>) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/threadpool.cc:491:10
#6 0x7fe79b491a6c in kudu::tserver::TSTabletManager::CreateNewTablet(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, kudu::Partition const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, kudu::Schema const&, kudu::PartitionSchema const&, kudu::consensus::RaftConfigPB, std::optional<kudu::TableExtraConfigPB>, std::optional<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::optional<kudu::TableTypePB>, scoped_refptr<kudu::tablet::TabletReplica>*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tserver/ts_tablet_manager.cc:703:3
#7 0x7fe79b3e5a62 in kudu::tserver::TabletServiceAdminImpl::CreateTablet(kudu::tserver::CreateTabletRequestPB const*, kudu::tserver::CreateTabletResponsePB*, kudu::rpc::RpcContext*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tserver/tablet_service.cc:1475:34
#8 0x7fe7955e56a6 in kudu::tserver::TabletServerAdminServiceIf::TabletServerAdminServiceIf(scoped_refptr<kudu::MetricEntity> const&, scoped_refptr<kudu::rpc::ResultTracker> const&)::$_1::operator()(google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*) const /home/jenkins-slave/workspace/build_and_test_flaky@2/build/asan/src/kudu/tserver/tserver_admin.service.cc:189:13
#9 0x7fe7955e53d5 in std::_Function_handler<void (google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*), kudu::tserver::TabletServerAdminServiceIf::TabletServerAdminServiceIf(scoped_refptr<kudu::MetricEntity> const&, scoped_refptr<kudu::rpc::ResultTracker> const&)::$_1>::_M_invoke(std::_Any_data const&, google::protobuf::Message const*&&, google::protobuf::Message*&&, kudu::rpc::RpcContext*&&) ../../../include/c++/7.5.0/bits/std_function.h:316:2
#10 0x7fe7929cf2ce in std::function<void (google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*)>::operator()(google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*) const ../../../include/c++/7.5.0/bits/std_function.h:706:14
#11 0x7fe7929ce1ae in kudu::rpc::GeneratedServiceIf::Handle(kudu::rpc::InboundCall*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/service_if.cc:137:3
#12 0x7fe7929d43e5 in kudu::rpc::ServicePool::RunThread() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/service_pool.cc:229:15
#13 0x7fe7929d4d0f in kudu::rpc::ServicePool::Init(int)::$_0::operator()() const /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/service_pool.cc:92:5
#14 0x7fe7929d4b38 in std::_Function_handler<void (), kudu::rpc::ServicePool::Init(int)::$_0>::_M_invoke(std::_Any_data const&) ../../../include/c++/7.5.0/bits/std_function.h:316:2
#15 0xa774dc in std::function<void ()>::operator()() const ../../../include/c++/7.5.0/bits/std_function.h:706:14
#16 0x7fe78bc46cc8 in kudu::Thread::SuperviseThread(void*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:694:3
#17 0x7fe78f0686da in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76da)
Thread T62 (rpc worker-762) created by T0 here:
#0 0x8920f0 in pthread_create /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/asan/asan_interceptors.cpp:214
#1 0x7fe78bc459c0 in kudu::Thread::StartThread(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::function<void ()>, unsigned long, scoped_refptr<kudu::Thread>*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:643:15
#2 0xa63f96 in kudu::Thread::Create(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::function<void ()>, scoped_refptr<kudu::Thread>*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.h:147:12
#3 0x7fe7929d12e2 in kudu::rpc::ServicePool::Init(int) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/service_pool.cc:92:5
I20250901 01:52:16.641170 453 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 4855448edb4b40c184838aa7f81c112b
I20250901 01:52:16.653726 447 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_edc9e269bf86407799be609875a22b0f" } new_table_name: "default.Table_e37cc89ad7ee4d96852b121d67b06248" modify_external_catalogs: true
I20250901 01:52:16.660554 441 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 88a6ab29d00542aba54f0e4185fbf615 (table default.Table_1159dbb70cc940df8b509db6e70c350a [id=1d0bc46740444f1fbf2129270e806259]) successfully deleted
I20250901 01:52:16.682052 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_3df3e67f179f46a8be496e3bfe3cb89f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:16.684832 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 579b9cdf819f4609a532e96bec093fd5
I20250901 01:52:16.686625 452 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 7cc1efcb6bff4aba8015d2d3b7f4c923 (table default.Table_5610feff8a344dc9be4bc1f0ad8ddd51 [id=78b00856a9b747c19f390b4fdae8ddcf]). Expected version 3 got 0
W20250901 01:52:16.707044 991 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250901 01:52:16.707311 1552 raft_consensus.cc:491] T 31390f38258c465cb1798615a1fcc790 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:16.707501 1552 raft_consensus.cc:513] T 31390f38258c465cb1798615a1fcc790 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:16.708349 1552 leader_election.cc:290] T 31390f38258c465cb1798615a1fcc790 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:16.709743 992 leader_election.cc:343] T 31390f38258c465cb1798615a1fcc790 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 31390f38258c465cb1798615a1fcc790
I20250901 01:52:16.557403 900 tablet_service.cc:1515] Processing DeleteTablet for tablet 1286c0ab85704837a99b6e5a62184d77 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:16 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:16.611660 1293 raft_consensus.cc:491] T b541a6e14f114ab29a864cca11806316 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:16.710484 1293 raft_consensus.cc:513] T b541a6e14f114ab29a864cca11806316 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:16.711331 1293 leader_election.cc:290] T b541a6e14f114ab29a864cca11806316 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:16.455947 1150 tablet_bootstrap.cc:654] T 533d14f5903e49599a34ea353ffd8c00 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:16.628822 898 tablet_service.cc:1515] Processing DeleteTablet for tablet 25ce0fe8c9544f828694184d19d13df9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:16 UTC) from {username='slave'} at 127.0.0.1:52846
W20250901 01:52:16.649923 858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:40731, user_credentials={real_user=slave}} blocked reactor thread for 137125us
I20250901 01:52:16.529259 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 0cb3262973934c8689b851adad60afe0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:16 UTC) from {username='slave'} at 127.0.0.1:52846
W20250901 01:52:16.657042 859 leader_election.cc:343] T 4a87e68850c4459eb2c8eafac89c6aa9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
W20250901 01:52:16.716320 859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:39447, user_credentials={real_user=slave}} blocked reactor thread for 59304.3us
I20250901 01:52:16.679757 921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c27bb23c150f4565a1d0ba8f231cc386" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:16.716894 921 raft_consensus.cc:2466] T c27bb23c150f4565a1d0ba8f231cc386 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:16.717419 1058 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b541a6e14f114ab29a864cca11806316" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:16.718124 992 leader_election.cc:304] T c27bb23c150f4565a1d0ba8f231cc386 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
W20250901 01:52:16.718158 859 leader_election.cc:343] T b541a6e14f114ab29a864cca11806316 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:16.693053 890 tablet_service.cc:1468] Processing CreateTablet for tablet 46bfbdff3f904715a8c24907b20fcab0 (DEFAULT_TABLE table=default.Table_a210833682434f928e7da36d2fb4e09c [id=e71440216cc74a5a87964a8e456f784c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:16.718596 1552 raft_consensus.cc:2802] T c27bb23c150f4565a1d0ba8f231cc386 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:16.718741 1552 raft_consensus.cc:491] T c27bb23c150f4565a1d0ba8f231cc386 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:16.718837 1552 raft_consensus.cc:3058] T c27bb23c150f4565a1d0ba8f231cc386 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:16.718905 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46bfbdff3f904715a8c24907b20fcab0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:16.722210 1552 raft_consensus.cc:513] T c27bb23c150f4565a1d0ba8f231cc386 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:16.723829 921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c27bb23c150f4565a1d0ba8f231cc386" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:16.724071 921 raft_consensus.cc:3058] T c27bb23c150f4565a1d0ba8f231cc386 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:16.728248 921 raft_consensus.cc:2466] T c27bb23c150f4565a1d0ba8f231cc386 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:16.728984 1552 leader_election.cc:290] T c27bb23c150f4565a1d0ba8f231cc386 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:16.729869 992 leader_election.cc:304] T c27bb23c150f4565a1d0ba8f231cc386 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:16.730393 1552 raft_consensus.cc:2802] T c27bb23c150f4565a1d0ba8f231cc386 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:16.733306 1552 raft_consensus.cc:695] T c27bb23c150f4565a1d0ba8f231cc386 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:16.734124 1552 consensus_queue.cc:237] T c27bb23c150f4565a1d0ba8f231cc386 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:16.702991 454 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 615b2306fca84003b264da9d5f95ddab has no consensus state
I20250901 01:52:16.757818 1306 tablet_replica.cc:331] T b541a6e14f114ab29a864cca11806316 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:16.758502 1306 raft_consensus.cc:2241] T b541a6e14f114ab29a864cca11806316 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:16.758860 1306 raft_consensus.cc:2270] T b541a6e14f114ab29a864cca11806316 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:16.759214 1151 tablet_bootstrap.cc:492] T ceb724d859d24d569598e18389dae3b3 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:16.759603 1151 ts_tablet_manager.cc:1397] T ceb724d859d24d569598e18389dae3b3 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.247s user 0.006s sys 0.034s
I20250901 01:52:16.759663 1306 ts_tablet_manager.cc:1905] T b541a6e14f114ab29a864cca11806316 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:16.761705 1151 raft_consensus.cc:357] T ceb724d859d24d569598e18389dae3b3 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:16.763207 1150 tablet_bootstrap.cc:492] T 533d14f5903e49599a34ea353ffd8c00 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:16.763490 1150 ts_tablet_manager.cc:1397] T 533d14f5903e49599a34ea353ffd8c00 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.378s user 0.002s sys 0.005s
I20250901 01:52:16.763494 1151 raft_consensus.cc:383] T ceb724d859d24d569598e18389dae3b3 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:16.764572 1150 raft_consensus.cc:357] T 533d14f5903e49599a34ea353ffd8c00 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:16.764881 1150 raft_consensus.cc:383] T 533d14f5903e49599a34ea353ffd8c00 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:16.764992 1150 raft_consensus.cc:738] T 533d14f5903e49599a34ea353ffd8c00 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:16.766132 1150 consensus_queue.cc:260] T 533d14f5903e49599a34ea353ffd8c00 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
#4 0x7fe794e79830 in kudu::RpcServer::RegisterService(std::unique_ptr<kudu::rpc::ServiceIf, std::default_delete<kudu::rpc::ServiceIf> >) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/server/rpc_server.cc:238:3
#5 0x7fe794e9ac47 in kudu::server::ServerBase::RegisterService(std::unique_ptr<kudu::rpc::ServiceIf, std::default_delete<kudu::rpc::ServiceIf> >) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/server/server_base.cc:1184:23
#6 0x7fe79b3cb5b3 in kudu::tserver::TabletServer::Start() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tserver/tablet_server.cc:157:3
#7 0x7fe79b3daa20 in kudu::tserver::RunTabletServer() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tserver/tablet_server_runner.cc:86:3
#8 0xb804d6 in kudu::tools::(anonymous namespace)::TServerRun(kudu::tools::RunnerContext const&) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/tool_action_tserver.cc:114:10
#9 0x9ab140 in std::_Function_handler<kudu::Status (kudu::tools::RunnerContext const&), kudu::Status (*)(kudu::tools::RunnerContext const&)>::_M_invoke(std::_Any_data const&, kudu::tools::RunnerContext const&) ../../../include/c++/7.5.0/bits/std_function.h:301:9
#10 0x7fe79fd404b8 in std::function<kudu::Status (kudu::tools::RunnerContext const&)>::operator()(kudu::tools::RunnerContext const&) const ../../../include/c++/7.5.0/bits/std_function.h:706:14
#11 0x7fe79fd3bfd4 in kudu::tools::Action::Run(std::vector<kudu::tools::Mode*, std::allocator<kudu::tools::Mode*> > const&, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) const /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/tool_action.cc:245:10
#12 0xbb3ed3 in kudu::tools::DispatchCommand(std::vector<kudu::tools::Mode*, std::allocator<kudu::tools::Mode*> > const&, kudu::tools::Action*, std::deque<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/tool_main.cc:142:15
#13 0xbb55b9 in kudu::tools::RunTool(int, char**, bool) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/tool_main.cc:214:16
#14 0xbb6357 in main /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/tool_main.cc:309:10
I20250901 01:52:16.767127 1151 raft_consensus.cc:738] T ceb724d859d24d569598e18389dae3b3 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
#15 0x7fe787c9abf6 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21bf6)
SUMMARY: AddressSanitizer: heap-use-after-free /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/gutil/atomicops-internals-x86.h:200:10 in base::subtle::NoBarrier_Load(int const volatile*)
Shadow bytes around the buggy address:
0x0c3280015fc0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c3280015fd0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c3280015fe0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c3280015ff0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c3280016000: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x0c3280016010: fd[fd]fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c3280016020: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c3280016030: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c3280016040: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c3280016050: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c3280016060: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
Shadow gap: cc
==693==ABORTING
W20250901 01:52:16.771507 441 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.3:39447, user_credentials={real_user=slave}} blocked reactor thread for 45712.3us
W20250901 01:52:16.771852 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 40d74d7b5a3b4d1dae09eebd9e07a0c4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 40d74d7b5a3b4d1dae09eebd9e07a0c4 already in progress: creating tablet
I20250901 01:52:16.770964 1151 consensus_queue.cc:260] T ceb724d859d24d569598e18389dae3b3 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:16.773343 452 catalog_manager.cc:5582] T 467c2bf4a8cb4937b6f627bea28d8ddb P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } }
I20250901 01:52:16.773828 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 60285f2ba037433999f5455fb9cc7acd (table default.Table_b582d9a2134b4184a195cfe0a8335a33 [id=cdf183c0a78b473b8d3428663415ab8d]) successfully deleted
W20250901 01:52:16.775280 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3df3e67f179f46a8be496e3bfe3cb89f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:16.777959 1151 ts_tablet_manager.cc:1428] T ceb724d859d24d569598e18389dae3b3 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.018s user 0.000s sys 0.003s
I20250901 01:52:16.780570 1151 tablet_bootstrap.cc:492] T 34042ea5c1b645969c1de41fe587c343 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
W20250901 01:52:16.781267 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 4a87e68850c4459eb2c8eafac89c6aa9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4a87e68850c4459eb2c8eafac89c6aa9 already in progress: creating tablet
W20250901 01:52:16.781556 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet f06a77ecc841462396a8d5026c7327f4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f06a77ecc841462396a8d5026c7327f4 already in progress: creating tablet
W20250901 01:52:16.781909 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 0cb3262973934c8689b851adad60afe0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0cb3262973934c8689b851adad60afe0 already in progress: creating tablet
W20250901 01:52:16.782115 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 1286c0ab85704837a99b6e5a62184d77 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1286c0ab85704837a99b6e5a62184d77 already in progress: creating tablet
W20250901 01:52:16.782392 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 25ce0fe8c9544f828694184d19d13df9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 25ce0fe8c9544f828694184d19d13df9 already in progress: creating tablet
I20250901 01:52:16.782970 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_1da54cdf1dfd4c0fb5f2538e26f17ad9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:16.783313 1021 tablet_service.cc:1468] Processing CreateTablet for tablet a289a66b2dfa44dc8c8547ff83c1716e (DEFAULT_TABLE table=default.Table_4f2be95a1b824f06ba16e56abcdebc8a [id=434cfb105b894499b15bf360a8cffda4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:16.783481 1038 tablet_service.cc:1468] Processing CreateTablet for tablet 46bfbdff3f904715a8c24907b20fcab0 (DEFAULT_TABLE table=default.Table_a210833682434f928e7da36d2fb4e09c [id=e71440216cc74a5a87964a8e456f784c]), partition=RANGE (key) PARTITION UNBOUNDED
W20250901 01:52:16.783581 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1da54cdf1dfd4c0fb5f2538e26f17ad9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:16.783735 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a289a66b2dfa44dc8c8547ff83c1716e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:16.783975 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46bfbdff3f904715a8c24907b20fcab0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:16.784168 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 579b9cdf819f4609a532e96bec093fd5 with delete_type TABLET_DATA_DELETED (replaced by tablet 56475ba491a24301a534304811f30140) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:16.784443 1033 tablet_service.cc:1515] Processing DeleteTablet for tablet 4855448edb4b40c184838aa7f81c112b with delete_type TABLET_DATA_DELETED (replaced by tablet 5e2472ee5d6e4d3bbf1f24027ba531ca) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:16.784490 452 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 467c2bf4a8cb4937b6f627bea28d8ddb (table default.Table_a0e83c6959d64160a5fdec67e0e49ba3 [id=f3735a7cf9634c11abc2c8121e2ed930]). Expected version 1 got 0
I20250901 01:52:16.785249 451 catalog_manager.cc:6652] ReplaceTablet: tablet 579b9cdf819f4609a532e96bec093fd5 deleted and replaced by tablet 56475ba491a24301a534304811f30140
I20250901 01:52:16.786505 453 catalog_manager.cc:6652] ReplaceTablet: tablet 4855448edb4b40c184838aa7f81c112b deleted and replaced by tablet 5e2472ee5d6e4d3bbf1f24027ba531ca
W20250901 01:52:16.792520 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 579b9cdf819f4609a532e96bec093fd5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 579b9cdf819f4609a532e96bec093fd5 already in progress: creating tablet
W20250901 01:52:16.792793 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 4855448edb4b40c184838aa7f81c112b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4855448edb4b40c184838aa7f81c112b already in progress: creating tablet
I20250901 01:52:16.794548 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 831e42de95854541a49ae6b3e039d3e9
I20250901 01:52:16.795332 1038 tablet_service.cc:1515] Processing DeleteTablet for tablet 831e42de95854541a49ae6b3e039d3e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:16 UTC) from {username='slave'} at 127.0.0.1:44246
W20250901 01:52:16.795959 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 831e42de95854541a49ae6b3e039d3e9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 831e42de95854541a49ae6b3e039d3e9 already in progress: creating tablet
W20250901 01:52:16.797854 994 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250901 01:52:16.798523 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_ae42ba019ee14c69b1a16d1177f3642c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:16.799147 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ae42ba019ee14c69b1a16d1177f3642c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:16.807953 453 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_647007ba008f42bcb27440e4d85a1d90" } new_table_name: "default.Table_a4ebd8ab87c842f082da5cd04d1bddac" modify_external_catalogs: true
I20250901 01:52:16.811849 1151 tablet_bootstrap.cc:654] T 34042ea5c1b645969c1de41fe587c343 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:16.812570 449 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 8dd587304e6342edbe96c9897b40e5c9
I20250901 01:52:16.813024 449 catalog_manager.cc:6652] ReplaceTablet: tablet 8dd587304e6342edbe96c9897b40e5c9 deleted and replaced by tablet 05630a72ddae4de0b890dece2feacc31
I20250901 01:52:16.813249 449 master_service.cc:948] ReplaceTablet: received request to replace tablet d58f69fad01f403eab74a47d3695ed83 from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:16.814780 1038 tablet_service.cc:1515] Processing DeleteTablet for tablet 8dd587304e6342edbe96c9897b40e5c9 with delete_type TABLET_DATA_DELETED (replaced by tablet 05630a72ddae4de0b890dece2feacc31) from {username='slave'} at 127.0.0.1:44246
W20250901 01:52:16.815431 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 8dd587304e6342edbe96c9897b40e5c9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8dd587304e6342edbe96c9897b40e5c9 already in progress: creating tablet
I20250901 01:52:16.816864 1038 tablet_service.cc:1468] Processing CreateTablet for tablet 6741c56499b44f648fb032c8cf50205b (DEFAULT_TABLE table=default.Table_db4aaba6e67e436dab76b895e3443bdd [id=c5667c5a65d045cea3cb52deb5801015]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:16.817237 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6741c56499b44f648fb032c8cf50205b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:16.819792 1021 tablet_service.cc:1468] Processing CreateTablet for tablet f80f13db501c42c7950499df8fcb6fc8 (DEFAULT_TABLE table=default.Table_0ff9dd032f74401b967bb5e4ed86d3ec [id=ff772d20b287476f94716379b86d2a71]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:16.820184 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f80f13db501c42c7950499df8fcb6fc8. 1 dirs total, 0 dirs full, 0 dirs failed
W20250901 01:52:16.824060 992 connection.cc:537] client connection to 127.0.104.1:40731 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 01:52:16.824501 992 leader_election.cc:336] T 8e6e96b4de214cadba1ccd19390cffb0 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250901 01:52:16.824685 992 leader_election.cc:304] T 8e6e96b4de214cadba1ccd19390cffb0 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
W20250901 01:52:16.824954 992 raft_consensus.cc:2684] T 8e6e96b4de214cadba1ccd19390cffb0 P fd27ab6665304aaea53adbd102dd4f58: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250901 01:52:16.825729 992 leader_election.cc:336] T eeae5294bd474f368f5766c260ccf3fe P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250901 01:52:16.825824 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet b7b802f976f3471faaaecc964ec87e48
W20250901 01:52:16.825956 992 leader_election.cc:336] T a4f5685260304d589dd467953d9bebfc P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250901 01:52:16.826076 992 leader_election.cc:304] T a4f5685260304d589dd467953d9bebfc P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
W20250901 01:52:16.826248 992 raft_consensus.cc:2684] T a4f5685260304d589dd467953d9bebfc P fd27ab6665304aaea53adbd102dd4f58: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250901 01:52:16.826514 450 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_1426e76865454f76a0529397d709ac7f" } new_table_name: "default.Table_1186b4165684445da50e713ed6a227f9" modify_external_catalogs: true
W20250901 01:52:16.826967 992 leader_election.cc:336] T 7a8a3aca289c4735abeb9c95c414ab76 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250901 01:52:16.827088 992 leader_election.cc:304] T 7a8a3aca289c4735abeb9c95c414ab76 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
W20250901 01:52:16.827275 992 raft_consensus.cc:2684] T 7a8a3aca289c4735abeb9c95c414ab76 P fd27ab6665304aaea53adbd102dd4f58: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250901 01:52:16.827591 992 leader_election.cc:336] T 4f8e3fcaa19740d397f4d118ac4cbdb0 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 01:52:16.827826 992 leader_election.cc:336] T 97e40c318f64455d91b0e4f8866990ce P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 01:52:16.828034 992 leader_election.cc:336] T 4f8e3fcaa19740d397f4d118ac4cbdb0 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 01:52:16.828912 992 leader_election.cc:336] T eeae5294bd474f368f5766c260ccf3fe P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 01:52:16.829320 992 leader_election.cc:336] T 97e40c318f64455d91b0e4f8866990ce P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 01:52:16.830428 992 leader_election.cc:336] T c445c300d71643609071f0f36528a9c4 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 01:52:16.830749 992 leader_election.cc:336] T 288e8029382f4154bc1e2103b3035a60 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 01:52:16.831064 992 leader_election.cc:336] T c31993d35207443288835f90091f73cf P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250901 01:52:16.831188 992 leader_election.cc:304] T c31993d35207443288835f90091f73cf P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
W20250901 01:52:16.831408 992 raft_consensus.cc:2684] T c31993d35207443288835f90091f73cf P fd27ab6665304aaea53adbd102dd4f58: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250901 01:52:16.831892 992 leader_election.cc:336] T 288e8029382f4154bc1e2103b3035a60 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 01:52:16.832417 992 leader_election.cc:336] T 3eaf8baac9f1485f944cd9e74b8b5569 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 01:52:16.832659 992 leader_election.cc:336] T c445c300d71643609071f0f36528a9c4 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 01:52:16.833205 440 connection.cc:537] client connection to 127.0.104.1:40731 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 01:52:16.833329 440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 7 similar messages]
W20250901 01:52:16.833595 992 leader_election.cc:336] T c297e8bf11ff467a854eab41117ac376 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 01:52:16.834096 992 leader_election.cc:336] T 3eaf8baac9f1485f944cd9e74b8b5569 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 01:52:16.834776 992 leader_election.cc:336] T 8d5b10ede9c84be098795c0a787e5a78 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 01:52:16.835536 992 leader_election.cc:336] T 061b760d8e6647c6ab0b5d538840a230 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250901 01:52:16.779111 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 579b9cdf819f4609a532e96bec093fd5 with delete_type TABLET_DATA_DELETED (replaced by tablet 56475ba491a24301a534304811f30140) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:16.779923 898 tablet_service.cc:1515] Processing DeleteTablet for tablet 4855448edb4b40c184838aa7f81c112b with delete_type TABLET_DATA_DELETED (replaced by tablet 5e2472ee5d6e4d3bbf1f24027ba531ca) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:16.780495 890 tablet_service.cc:1468] Processing CreateTablet for tablet a289a66b2dfa44dc8c8547ff83c1716e (DEFAULT_TABLE table=default.Table_4f2be95a1b824f06ba16e56abcdebc8a [id=434cfb105b894499b15bf360a8cffda4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:16.796180 900 tablet_service.cc:1515] Processing DeleteTablet for tablet 831e42de95854541a49ae6b3e039d3e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:16 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:16.815783 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 8dd587304e6342edbe96c9897b40e5c9 with delete_type TABLET_DATA_DELETED (replaced by tablet 05630a72ddae4de0b890dece2feacc31) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:16.819152 899 tablet_service.cc:1468] Processing CreateTablet for tablet 6741c56499b44f648fb032c8cf50205b (DEFAULT_TABLE table=default.Table_db4aaba6e67e436dab76b895e3443bdd [id=c5667c5a65d045cea3cb52deb5801015]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:16.841183 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a289a66b2dfa44dc8c8547ff83c1716e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:16.820780 902 tablet_service.cc:1468] Processing CreateTablet for tablet f80f13db501c42c7950499df8fcb6fc8 (DEFAULT_TABLE table=default.Table_0ff9dd032f74401b967bb5e4ed86d3ec [id=ff772d20b287476f94716379b86d2a71]), partition=RANGE (key) PARTITION UNBOUNDED
W20250901 01:52:16.823544 858 connection.cc:537] client connection to 127.0.104.1:40731 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 01:52:16.841636 858 leader_election.cc:336] T 360755d30b5d47ba98c9c7e23ff5ab7e P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250901 01:52:16.841756 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f80f13db501c42c7950499df8fcb6fc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:16.841778 858 leader_election.cc:304] T 360755d30b5d47ba98c9c7e23ff5ab7e P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:16.842084 858 raft_consensus.cc:2684] T 360755d30b5d47ba98c9c7e23ff5ab7e P 00c282f779c1477d920f24dc0dc16628: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250901 01:52:16.842746 858 leader_election.cc:336] T 9496ec6021784800aef62996b0dee593 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250901 01:52:16.842898 858 leader_election.cc:304] T 9496ec6021784800aef62996b0dee593 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:16.843022 1151 tablet_bootstrap.cc:492] T 34042ea5c1b645969c1de41fe587c343 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
W20250901 01:52:16.843108 858 raft_consensus.cc:2684] T 9496ec6021784800aef62996b0dee593 P 00c282f779c1477d920f24dc0dc16628: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250901 01:52:16.843222 1151 ts_tablet_manager.cc:1397] T 34042ea5c1b645969c1de41fe587c343 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.063s user 0.005s sys 0.001s
W20250901 01:52:16.843815 858 leader_election.cc:336] T 6c15cd14831f422eb615038603c79ed0 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250901 01:52:16.843946 858 leader_election.cc:304] T 6c15cd14831f422eb615038603c79ed0 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:16.844125 858 raft_consensus.cc:2684] T 6c15cd14831f422eb615038603c79ed0 P 00c282f779c1477d920f24dc0dc16628: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250901 01:52:16.828783 903 tablet_service.cc:1515] Processing DeleteTablet for tablet b7b802f976f3471faaaecc964ec87e48 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:16 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:16.844589 1151 raft_consensus.cc:357] T 34042ea5c1b645969c1de41fe587c343 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:16.844857 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6741c56499b44f648fb032c8cf50205b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:16.844959 1151 raft_consensus.cc:383] T 34042ea5c1b645969c1de41fe587c343 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250901 01:52:16.845057 858 leader_election.cc:336] T 447ffe73741640ceb73f35676232a094 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250901 01:52:16.845096 1151 raft_consensus.cc:738] T 34042ea5c1b645969c1de41fe587c343 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:16.845194 858 leader_election.cc:304] T 447ffe73741640ceb73f35676232a094 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:16.845412 858 raft_consensus.cc:2684] T 447ffe73741640ceb73f35676232a094 P 00c282f779c1477d920f24dc0dc16628: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250901 01:52:16.845574 1151 consensus_queue.cc:260] T 34042ea5c1b645969c1de41fe587c343 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:16.847376 1150 ts_tablet_manager.cc:1428] T 533d14f5903e49599a34ea353ffd8c00 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.084s user 0.003s sys 0.000s
I20250901 01:52:16.847890 1151 ts_tablet_manager.cc:1428] T 34042ea5c1b645969c1de41fe587c343 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20250901 01:52:16.849864 1150 tablet_bootstrap.cc:492] T 606fac5c6efe48d1877104b01c23ba5b P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
W20250901 01:52:16.851114 992 leader_election.cc:336] T 116f65a42ab34bf6b8f8113cb7afdc21 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250901 01:52:16.851063 1151 tablet_bootstrap.cc:492] T a5f4b0a2d72a4b8095ae4af1fe7c0c5d P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
W20250901 01:52:16.851512 992 leader_election.cc:336] T bf302e76746b4f7bbe4469914f83720f P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 01:52:16.851756 992 leader_election.cc:336] T 061b760d8e6647c6ab0b5d538840a230 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 01:52:16.852072 992 leader_election.cc:336] T 7a8a3aca289c4735abeb9c95c414ab76 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 01:52:16.856738 858 leader_election.cc:336] T 2685fb7258264463b5f1ec99cd7d67ad P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250901 01:52:16.856873 858 leader_election.cc:304] T 2685fb7258264463b5f1ec99cd7d67ad P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:16.857092 858 raft_consensus.cc:2684] T 2685fb7258264463b5f1ec99cd7d67ad P 00c282f779c1477d920f24dc0dc16628: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250901 01:52:16.857733 858 leader_election.cc:336] T f0affc7a78634607852a48458412761d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250901 01:52:16.857853 858 leader_election.cc:304] T f0affc7a78634607852a48458412761d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:16.858011 858 raft_consensus.cc:2684] T f0affc7a78634607852a48458412761d P 00c282f779c1477d920f24dc0dc16628: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250901 01:52:16.858482 858 leader_election.cc:336] T 4fd0e4f9d86c4362af96ce9964b72328 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250901 01:52:16.858585 858 leader_election.cc:304] T 4fd0e4f9d86c4362af96ce9964b72328 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:16.859313 1293 raft_consensus.cc:2747] T 4fd0e4f9d86c4362af96ce9964b72328 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:16.859939 858 leader_election.cc:336] T 467c2bf4a8cb4937b6f627bea28d8ddb P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250901 01:52:16.860081 858 leader_election.cc:304] T 467c2bf4a8cb4937b6f627bea28d8ddb P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:16.860348 1293 raft_consensus.cc:2747] T 467c2bf4a8cb4937b6f627bea28d8ddb P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:16.861441 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_e59cac5d3f0a4dd6b9846e7449b60005"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:16.861660 1151 tablet_bootstrap.cc:654] T a5f4b0a2d72a4b8095ae4af1fe7c0c5d P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:16.862157 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e59cac5d3f0a4dd6b9846e7449b60005 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250901 01:52:16.862936 858 leader_election.cc:336] T eeae5294bd474f368f5766c260ccf3fe P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250901 01:52:16.863116 858 leader_election.cc:304] T eeae5294bd474f368f5766c260ccf3fe P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:16.863373 1293 raft_consensus.cc:2747] T eeae5294bd474f368f5766c260ccf3fe P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:16.864456 858 leader_election.cc:336] T 4f8e3fcaa19740d397f4d118ac4cbdb0 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250901 01:52:16.864609 858 leader_election.cc:304] T 4f8e3fcaa19740d397f4d118ac4cbdb0 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:16.864827 858 raft_consensus.cc:2684] T 4f8e3fcaa19740d397f4d118ac4cbdb0 P 00c282f779c1477d920f24dc0dc16628: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250901 01:52:16.865419 858 leader_election.cc:336] T 2a0d125423a043dcb11e67345599e468 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250901 01:52:16.865651 858 leader_election.cc:304] T 2a0d125423a043dcb11e67345599e468 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:16.865895 858 raft_consensus.cc:2684] T 2a0d125423a043dcb11e67345599e468 P 00c282f779c1477d920f24dc0dc16628: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250901 01:52:16.867411 858 leader_election.cc:336] T fab3c1a8928a4066a15868ba708267f1 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 01:52:16.868227 858 leader_election.cc:336] T 116f65a42ab34bf6b8f8113cb7afdc21 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250901 01:52:16.869191 858 leader_election.cc:304] T 116f65a42ab34bf6b8f8113cb7afdc21 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:16.869827 992 leader_election.cc:336] T bf302e76746b4f7bbe4469914f83720f P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 01:52:16.870041 858 raft_consensus.cc:2684] T 116f65a42ab34bf6b8f8113cb7afdc21 P 00c282f779c1477d920f24dc0dc16628: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250901 01:52:16.870574 858 leader_election.cc:336] T 8e6e96b4de214cadba1ccd19390cffb0 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250901 01:52:16.871133 1552 raft_consensus.cc:491] T f65ec84958eb488d8b0f812b59b5b3d5 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:16.871393 1552 raft_consensus.cc:513] T f65ec84958eb488d8b0f812b59b5b3d5 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:16.872015 1552 leader_election.cc:290] T f65ec84958eb488d8b0f812b59b5b3d5 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:16.872915 858 leader_election.cc:304] T 8e6e96b4de214cadba1ccd19390cffb0 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:16.873134 858 raft_consensus.cc:2684] T 8e6e96b4de214cadba1ccd19390cffb0 P 00c282f779c1477d920f24dc0dc16628: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250901 01:52:16.873415 992 leader_election.cc:336] T 116f65a42ab34bf6b8f8113cb7afdc21 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 01:52:16.874437 992 leader_election.cc:336] T 8d5b10ede9c84be098795c0a787e5a78 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 01:52:16.874799 858 leader_election.cc:336] T 288e8029382f4154bc1e2103b3035a60 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250901 01:52:16.874994 858 leader_election.cc:304] T 288e8029382f4154bc1e2103b3035a60 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:16.875155 992 leader_election.cc:304] T 8d5b10ede9c84be098795c0a787e5a78 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
W20250901 01:52:16.875350 858 raft_consensus.cc:2684] T 288e8029382f4154bc1e2103b3035a60 P 00c282f779c1477d920f24dc0dc16628: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250901 01:52:16.875550 992 raft_consensus.cc:2684] T 8d5b10ede9c84be098795c0a787e5a78 P fd27ab6665304aaea53adbd102dd4f58: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250901 01:52:16.876106 858 leader_election.cc:336] T c445c300d71643609071f0f36528a9c4 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250901 01:52:16.873623 1103 heartbeater.cc:344] Connected to a master server at 127.0.104.60:44231
I20250901 01:52:16.884660 858 leader_election.cc:304] T c445c300d71643609071f0f36528a9c4 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:16.884954 858 raft_consensus.cc:2684] T c445c300d71643609071f0f36528a9c4 P 00c282f779c1477d920f24dc0dc16628: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250901 01:52:16.886307 858 leader_election.cc:336] T 061b760d8e6647c6ab0b5d538840a230 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 01:52:16.887075 440 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.1:40731, user_credentials={real_user=slave}} blocked reactor thread for 53717.5us
I20250901 01:52:16.888509 858 leader_election.cc:304] T 061b760d8e6647c6ab0b5d538840a230 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:16.888801 1293 raft_consensus.cc:2747] T 061b760d8e6647c6ab0b5d538840a230 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:16.889722 858 leader_election.cc:336] T 77405097e68344628f4eda667afaed21 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 01:52:16.890014 858 leader_election.cc:336] T 4cdd369214ef4f5ebabddbf28b2ae3f7 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250901 01:52:16.890151 858 leader_election.cc:304] T 4cdd369214ef4f5ebabddbf28b2ae3f7 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:16.890369 858 raft_consensus.cc:2684] T 4cdd369214ef4f5ebabddbf28b2ae3f7 P 00c282f779c1477d920f24dc0dc16628: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250901 01:52:16.890910 858 leader_election.cc:336] T 97e40c318f64455d91b0e4f8866990ce P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250901 01:52:16.891062 858 leader_election.cc:304] T 97e40c318f64455d91b0e4f8866990ce P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:16.896570 858 raft_consensus.cc:2684] T 97e40c318f64455d91b0e4f8866990ce P 00c282f779c1477d920f24dc0dc16628: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250901 01:52:16.899868 859 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250901 01:52:16.900004 921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f65ec84958eb488d8b0f812b59b5b3d5" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:16.900220 921 raft_consensus.cc:2466] T f65ec84958eb488d8b0f812b59b5b3d5 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:16.900394 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet b7b802f976f3471faaaecc964ec87e48 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:16 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:16.901366 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 3618a3eb4f174c43805ce7dc0e5315cb (DEFAULT_TABLE table=default.Table_83f2c0acedbc4c6982caf5313646e90b [id=31a2e50c32644321825da40176127015]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:16.901818 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3618a3eb4f174c43805ce7dc0e5315cb. 1 dirs total, 0 dirs full, 0 dirs failed
W20250901 01:52:16.903550 447 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 34042ea5c1b645969c1de41fe587c343 consensus state has no leader
W20250901 01:52:16.906167 452 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.3:47125 (request call id 30) took 753 ms (client timeout 1000 ms). Trace:
W20250901 01:52:16.906307 452 rpcz_store.cc:269] 0901 01:52:16.152550 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:16.152622 (+ 72us) service_pool.cc:225] Handling call
0901 01:52:16.906126 (+753504us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0901 01:52:16.874658 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:16.874759 (+ 101us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:16.874763 (+ 4us) write_op.cc:435] Acquired schema lock
0901 01:52:16.874766 (+ 3us) tablet.cc:660] Decoding operations
0901 01:52:16.874842 (+ 76us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:16.874850 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:16.874854 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0901 01:52:16.874899 (+ 45us) tablet.cc:699] Row locks acquired
0901 01:52:16.874902 (+ 3us) write_op.cc:260] PREPARE: finished
0901 01:52:16.874925 (+ 23us) write_op.cc:270] Start()
0901 01:52:16.874948 (+ 23us) write_op.cc:276] Timestamp: P: 1756691536874890 usec, L: 0
0901 01:52:16.874964 (+ 16us) op_driver.cc:348] REPLICATION: starting
0901 01:52:16.875096 (+ 132us) log.cc:844] Serialized 527 byte log entry
0901 01:52:16.901518 (+ 26422us) op_driver.cc:464] REPLICATION: finished
0901 01:52:16.902946 (+ 1428us) write_op.cc:301] APPLY: starting
0901 01:52:16.902957 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:16.902986 (+ 29us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:16.903006 (+ 20us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:16.903040 (+ 34us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:16.903063 (+ 23us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0901 01:52:16.903069 (+ 6us) write_op.cc:312] APPLY: finished
0901 01:52:16.903116 (+ 47us) log.cc:844] Serialized 23 byte log entry
0901 01:52:16.903188 (+ 72us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:16.903201 (+ 13us) write_op.cc:454] Released schema lock
0901 01:52:16.903206 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":8,"child_traces":[["op",{"apply.queue_time_us":1382,"apply.run_cpu_time_us":328,"apply.run_wall_time_us":328,"num_ops":1,"prepare.queue_time_us":89482,"prepare.run_cpu_time_us":491,"prepare.run_wall_time_us":490,"replication_time_us":26540,"threads_started":1}]]}
I20250901 01:52:16.907967 1572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Bootstrap replayed 1/1 log segments. Stats: ops{read=45 overwritten=0 applied=44 ignored=0} inserts{seen=43 ignored=0} mutations{seen=30 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250901 01:52:16.908747 1021 tablet_service.cc:1468] Processing CreateTablet for tablet ad45c90ba0da43c9860745e534664a98 (DEFAULT_TABLE table=default.Table_c5c427e0288c4f879bd65c0f006c0f5d [id=6fd739b0e725424ca844dbb652ea8028]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:16.908869 1572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Bootstrap complete.
I20250901 01:52:16.909003 451 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 467c2bf4a8cb4937b6f627bea28d8ddb (table default.Table_a0e83c6959d64160a5fdec67e0e49ba3 [id=f3735a7cf9634c11abc2c8121e2ed930]). Expected version 1 got 0
I20250901 01:52:16.909161 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad45c90ba0da43c9860745e534664a98. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:16.912711 1038 tablet_service.cc:1468] Processing CreateTablet for tablet 17e8340dbd1a406cb782dad3fe88cc7e (DEFAULT_TABLE table=default.Table_50ae3908a7944f70949c54be8fbf1564 [id=58ff38d356d74951b663d8ef7530f1c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:16.913057 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17e8340dbd1a406cb782dad3fe88cc7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:16.919677 1033 tablet_service.cc:1468] Processing CreateTablet for tablet 81c36693199945d98465579b509852e2 (DEFAULT_TABLE table=default.Table_edd1006860a84e209149771e23bb8834 [id=662be9c7921d422f93307d06187b1cae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:16.919739 1038 tablet_service.cc:1468] Processing CreateTablet for tablet c754129977904f978f87fd89d7092e36 (DEFAULT_TABLE table=default.Table_e113aaaa38db48fdb16d32569b81b930 [id=f09f20dc4db844a8abc825e9086093ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:16.920053 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 84fe314502444bd88b038b87eead9cb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:12 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:16.920125 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81c36693199945d98465579b509852e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:16.920492 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 31390f38258c465cb1798615a1fcc790 with delete_type TABLET_DATA_DELETED (replaced by tablet 4f4be080f20d41c798835b7111a5318d) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:16.920679 1025 tablet_service.cc:1468] Processing CreateTablet for tablet e249474dcbf642189560c91ec3a99fdf (DEFAULT_TABLE table=default.Table_adb18e9d4ac64721b777cd20a2cd0800 [id=ea928501fe954cd0b81d38add4ef7128]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:16.920907 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 9496ec6021784800aef62996b0dee593 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:13 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:16.921648 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 6c15cd14831f422eb615038603c79ed0 with delete_type TABLET_DATA_DELETED (replaced by tablet 824229183e024feaa975c7f0387c9c1a) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:16.922044 1151 tablet_bootstrap.cc:492] T a5f4b0a2d72a4b8095ae4af1fe7c0c5d P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:16.922204 1151 ts_tablet_manager.cc:1397] T a5f4b0a2d72a4b8095ae4af1fe7c0c5d P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.072s user 0.004s sys 0.006s
I20250901 01:52:16.922586 1023 tablet_service.cc:1468] Processing CreateTablet for tablet a8483e7212d94daba7e5401256c33aef (DEFAULT_TABLE table=default.Table_142ccc8cc7414621bc0c31326d5aa0e9 [id=6a01b6c6977546be9c6128ca5f200a92]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:16.923084 1151 raft_consensus.cc:357] T a5f4b0a2d72a4b8095ae4af1fe7c0c5d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:16.923200 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 7a2f6e2c7ee2402ea310c058e046416c (DEFAULT_TABLE table=default.Table_a89cc8e48563460baabc34c821ceae35 [id=9c3930e9fcfe4b49ba4bea67a2b0ad07]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:16.923292 1151 raft_consensus.cc:383] T a5f4b0a2d72a4b8095ae4af1fe7c0c5d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:16.923383 1151 raft_consensus.cc:738] T a5f4b0a2d72a4b8095ae4af1fe7c0c5d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:16.923648 1151 consensus_queue.cc:260] T a5f4b0a2d72a4b8095ae4af1fe7c0c5d P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:16.923802 858 leader_election.cc:336] T 4d333b670df84f308f0cbf6da636b768 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250901 01:52:16.923884 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 6cf09e8f299548dba2cce16d7b565bad (DEFAULT_TABLE table=default.Table_749fcd4bf97d439c9fd49b138a52a3fd [id=24fffc7ebf5a47bdb37211910301acf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:16.923933 858 leader_election.cc:304] T 4d333b670df84f308f0cbf6da636b768 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:16.924115 858 raft_consensus.cc:2684] T 4d333b670df84f308f0cbf6da636b768 P 00c282f779c1477d920f24dc0dc16628: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250901 01:52:16.924360 1151 ts_tablet_manager.cc:1428] T a5f4b0a2d72a4b8095ae4af1fe7c0c5d P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20250901 01:52:16.924729 858 leader_election.cc:336] T bf302e76746b4f7bbe4469914f83720f P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250901 01:52:16.924870 858 leader_election.cc:304] T bf302e76746b4f7bbe4469914f83720f P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:16.925038 858 raft_consensus.cc:2684] T bf302e76746b4f7bbe4469914f83720f P 00c282f779c1477d920f24dc0dc16628: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250901 01:52:16.886936 1150 tablet_bootstrap.cc:654] T 606fac5c6efe48d1877104b01c23ba5b P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:16.925855 992 leader_election.cc:336] T d6a4040747fa4d588756506ee122d2dc P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:16.926005 992 leader_election.cc:304] T d6a4040747fa4d588756506ee122d2dc P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:16.926195 1306 ts_tablet_manager.cc:1918] T b541a6e14f114ab29a864cca11806316 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250901 01:52:16.926198 992 raft_consensus.cc:2684] T d6a4040747fa4d588756506ee122d2dc P fd27ab6665304aaea53adbd102dd4f58: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250901 01:52:16.926729 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 8ca5cf4e99014c0d84a3acad4ef713af (DEFAULT_TABLE table=default.Table_746f677d34894fc0a027945dd2c04b53 [id=303e76c918bc4c39aa47a4d24849a358]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:16.926321 1306 log.cc:1199] T b541a6e14f114ab29a864cca11806316 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/b541a6e14f114ab29a864cca11806316
I20250901 01:52:16.928185 1306 ts_tablet_manager.cc:1939] T b541a6e14f114ab29a864cca11806316 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
W20250901 01:52:16.926771 992 consensus_peers.cc:489] T 85322a864d154f299f2d670fdd2a7bfb P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250901 01:52:16.929730 992 leader_election.cc:336] T 0e698923812841b7a5d37d57da103c5c P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:16.929862 992 leader_election.cc:304] T 0e698923812841b7a5d37d57da103c5c P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
W20250901 01:52:16.930049 992 raft_consensus.cc:2684] T 0e698923812841b7a5d37d57da103c5c P fd27ab6665304aaea53adbd102dd4f58: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250901 01:52:16.930563 992 leader_election.cc:336] T 467c2bf4a8cb4937b6f627bea28d8ddb P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:16.930776 992 leader_election.cc:336] T 467c2bf4a8cb4937b6f627bea28d8ddb P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:16.930976 992 leader_election.cc:336] T 2a0d125423a043dcb11e67345599e468 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:16.931073 992 leader_election.cc:304] T 2a0d125423a043dcb11e67345599e468 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
W20250901 01:52:16.931250 992 raft_consensus.cc:2684] T 2a0d125423a043dcb11e67345599e468 P fd27ab6665304aaea53adbd102dd4f58: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250901 01:52:16.931725 992 leader_election.cc:336] T 2685fb7258264463b5f1ec99cd7d67ad P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:16.932214 890 tablet_service.cc:1468] Processing CreateTablet for tablet 3618a3eb4f174c43805ce7dc0e5315cb (DEFAULT_TABLE table=default.Table_83f2c0acedbc4c6982caf5313646e90b [id=31a2e50c32644321825da40176127015]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:16.932718 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3618a3eb4f174c43805ce7dc0e5315cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:16.933889 1306 tablet_replica.cc:331] T 4d27fb61943c45d4aca67689cc3a9ff3 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:16.934139 1306 raft_consensus.cc:2241] T 4d27fb61943c45d4aca67689cc3a9ff3 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:16.934309 1306 raft_consensus.cc:2270] T 4d27fb61943c45d4aca67689cc3a9ff3 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:16.936823 1332 tablet_replica.cc:331] T 84fe314502444bd88b038b87eead9cb5 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:16.937018 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e249474dcbf642189560c91ec3a99fdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:16.937078 1332 raft_consensus.cc:2241] T 84fe314502444bd88b038b87eead9cb5 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:16.937244 1332 raft_consensus.cc:2270] T 84fe314502444bd88b038b87eead9cb5 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:16.937814 1332 ts_tablet_manager.cc:1905] T 84fe314502444bd88b038b87eead9cb5 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:16.939518 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8483e7212d94daba7e5401256c33aef. 1 dirs total, 0 dirs full, 0 dirs failed
W20250901 01:52:16.940200 453 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3a8231590ea643b0a1f83eec4d9a8191 consensus state has no leader
I20250901 01:52:16.940562 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a2f6e2c7ee2402ea310c058e046416c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:16.941825 447 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_cb2cf28c0e004f7daa4fd83bc027bfaa" } modify_external_catalogs: true
I20250901 01:52:16.942023 447 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_cb2cf28c0e004f7daa4fd83bc027bfaa" } modify_external_catalogs: true
I20250901 01:52:16.943573 452 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_1159dbb70cc940df8b509db6e70c350a" } modify_external_catalogs: true
I20250901 01:52:16.943786 452 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_1159dbb70cc940df8b509db6e70c350a" } modify_external_catalogs: true
W20250901 01:52:16.944141 858 leader_election.cc:336] T 24ddc13d1bb04db4a7850acbff091577 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 01:52:16.944476 858 leader_election.cc:336] T 7a8a3aca289c4735abeb9c95c414ab76 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250901 01:52:16.944600 858 leader_election.cc:304] T 7a8a3aca289c4735abeb9c95c414ab76 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:16.944779 858 raft_consensus.cc:2684] T 7a8a3aca289c4735abeb9c95c414ab76 P 00c282f779c1477d920f24dc0dc16628: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250901 01:52:16.945353 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_9163155cb611406a8ce0d5ae600a15c8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:16.945963 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9163155cb611406a8ce0d5ae600a15c8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250901 01:52:16.945991 858 leader_election.cc:336] T 24ddc13d1bb04db4a7850acbff091577 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 01:52:16.946676 858 leader_election.cc:336] T 8d5b10ede9c84be098795c0a787e5a78 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 01:52:16.947149 858 leader_election.cc:336] T 8d5b10ede9c84be098795c0a787e5a78 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250901 01:52:16.947240 858 leader_election.cc:304] T 8d5b10ede9c84be098795c0a787e5a78 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:16.947273 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_83cf2d86a1284544ba2e7aa9ebb66bb7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:16.947429 858 raft_consensus.cc:2684] T 8d5b10ede9c84be098795c0a787e5a78 P 00c282f779c1477d920f24dc0dc16628: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250901 01:52:16.947832 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_83cf2d86a1284544ba2e7aa9ebb66bb7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250901 01:52:16.948339 858 leader_election.cc:336] T 77405097e68344628f4eda667afaed21 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 01:52:16.948916 858 leader_election.cc:336] T 342b7fde3ede494b9eec6e361db255dc P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250901 01:52:16.949062 858 leader_election.cc:304] T 342b7fde3ede494b9eec6e361db255dc P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:16.949285 858 raft_consensus.cc:2684] T 342b7fde3ede494b9eec6e361db255dc P 00c282f779c1477d920f24dc0dc16628: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250901 01:52:16.949426 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6cf09e8f299548dba2cce16d7b565bad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:16.949733 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ca5cf4e99014c0d84a3acad4ef713af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:16.946377 1572 raft_consensus.cc:357] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f [term 2 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3add7edb97a40ef9f7c6bc011d13227" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 43871 } } peers { permanent_uuid: "d2ae2cb6af0f4b3a8f47ab1cdd6567d3" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42375 } } peers { permanent_uuid: "7ce2288f784044eea010664e3def133f" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 44231 } }
I20250901 01:52:16.950548 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c754129977904f978f87fd89d7092e36. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:16.950345 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_48252c70dcfd4a9e8dbd3b8efba88da6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:16.951357 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_48252c70dcfd4a9e8dbd3b8efba88da6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:16.952500 1151 tablet_bootstrap.cc:492] T 4410da9e95454ecf95faf80221b0a71c P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:16.955788 449 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet d58f69fad01f403eab74a47d3695ed83
I20250901 01:52:16.956216 449 catalog_manager.cc:6652] ReplaceTablet: tablet d58f69fad01f403eab74a47d3695ed83 deleted and replaced by tablet 74b709cb5ec64006b2f12bead8217d80
I20250901 01:52:16.956540 449 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_6eca408a8cf342e1bafcbfbf927e214b" } new_table_name: "default.Table_0451d0afa26f4d5f8d4214a946eea04f" modify_external_catalogs: true
I20250901 01:52:16.931864 992 leader_election.cc:304] T 2685fb7258264463b5f1ec99cd7d67ad P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
W20250901 01:52:16.957440 992 raft_consensus.cc:2684] T 2685fb7258264463b5f1ec99cd7d67ad P fd27ab6665304aaea53adbd102dd4f58: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250901 01:52:16.967796 454 master_service.cc:948] ReplaceTablet: received request to replace tablet 3b18a7084727490380d26b3485e4e799 from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:16.971920 1332 ts_tablet_manager.cc:1918] T 84fe314502444bd88b038b87eead9cb5 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:16.974159 1332 log.cc:1199] T 84fe314502444bd88b038b87eead9cb5 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/84fe314502444bd88b038b87eead9cb5
I20250901 01:52:16.975219 1332 ts_tablet_manager.cc:1939] T 84fe314502444bd88b038b87eead9cb5 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:16.975834 1151 tablet_bootstrap.cc:654] T 4410da9e95454ecf95faf80221b0a71c P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:16.976861 890 tablet_service.cc:1468] Processing CreateTablet for tablet ad45c90ba0da43c9860745e534664a98 (DEFAULT_TABLE table=default.Table_c5c427e0288c4f879bd65c0f006c0f5d [id=6fd739b0e725424ca844dbb652ea8028]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:16.977059 902 tablet_service.cc:1468] Processing CreateTablet for tablet 17e8340dbd1a406cb782dad3fe88cc7e (DEFAULT_TABLE table=default.Table_50ae3908a7944f70949c54be8fbf1564 [id=58ff38d356d74951b663d8ef7530f1c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:16.977358 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad45c90ba0da43c9860745e534664a98. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:16.977588 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17e8340dbd1a406cb782dad3fe88cc7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:16.979154 899 tablet_service.cc:1468] Processing CreateTablet for tablet c754129977904f978f87fd89d7092e36 (DEFAULT_TABLE table=default.Table_e113aaaa38db48fdb16d32569b81b930 [id=f09f20dc4db844a8abc825e9086093ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:16.979564 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c754129977904f978f87fd89d7092e36. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:16.979225 1306 ts_tablet_manager.cc:1905] T 4d27fb61943c45d4aca67689cc3a9ff3 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:16.982900 903 tablet_service.cc:1468] Processing CreateTablet for tablet 81c36693199945d98465579b509852e2 (DEFAULT_TABLE table=default.Table_edd1006860a84e209149771e23bb8834 [id=662be9c7921d422f93307d06187b1cae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:16.983283 903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81c36693199945d98465579b509852e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:16.984886 1332 tablet_replica.cc:331] T 31390f38258c465cb1798615a1fcc790 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:16.985109 1332 raft_consensus.cc:2241] T 31390f38258c465cb1798615a1fcc790 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:16.990360 1332 raft_consensus.cc:2270] T 31390f38258c465cb1798615a1fcc790 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:16.991271 1332 ts_tablet_manager.cc:1905] T 31390f38258c465cb1798615a1fcc790 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:16.993142 992 leader_election.cc:336] T 360755d30b5d47ba98c9c7e23ff5ab7e P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:16.995065 450 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2ff914352c7c43de800f3d54ecf63072 consensus state has no leader
I20250901 01:52:16.997795 992 leader_election.cc:304] T 360755d30b5d47ba98c9c7e23ff5ab7e P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
W20250901 01:52:17.001698 992 raft_consensus.cc:2684] T 360755d30b5d47ba98c9c7e23ff5ab7e P fd27ab6665304aaea53adbd102dd4f58: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250901 01:52:17.003866 451 catalog_manager.cc:6127] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: default.Table_2d3d2227e6a24bd2a86f457e553e698b [id=f48fd53522234290b6c0373fd7b02570] alter complete (version 2)
I20250901 01:52:17.004668 447 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet d1b5f156ebdb441da58bead8d745e34f
W20250901 01:52:17.007748 992 leader_election.cc:336] T 4fd0e4f9d86c4362af96ce9964b72328 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:17.008006 992 leader_election.cc:336] T d945b310fe6344af9cc9dd68f84c4bdf P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:17.008107 992 leader_election.cc:304] T d945b310fe6344af9cc9dd68f84c4bdf P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
W20250901 01:52:17.008296 992 raft_consensus.cc:2684] T d945b310fe6344af9cc9dd68f84c4bdf P fd27ab6665304aaea53adbd102dd4f58: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250901 01:52:17.008832 992 consensus_peers.cc:489] T eeae5294bd474f368f5766c260ccf3fe P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250901 01:52:17.008982 992 leader_election.cc:336] T 9496ec6021784800aef62996b0dee593 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:17.009081 992 leader_election.cc:304] T 9496ec6021784800aef62996b0dee593 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:17.010187 452 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet a774cc6ae3924d299127fd9ffb01ca57
I20250901 01:52:17.011926 1332 ts_tablet_manager.cc:1918] T 31390f38258c465cb1798615a1fcc790 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:17.012068 1332 log.cc:1199] T 31390f38258c465cb1798615a1fcc790 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/31390f38258c465cb1798615a1fcc790
I20250901 01:52:17.013549 1151 tablet_bootstrap.cc:492] T 4410da9e95454ecf95faf80221b0a71c P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:17.013720 1151 ts_tablet_manager.cc:1397] T 4410da9e95454ecf95faf80221b0a71c P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.072s user 0.008s sys 0.000s
I20250901 01:52:17.014343 1552 raft_consensus.cc:2747] T 9496ec6021784800aef62996b0dee593 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:17.014284 891 tablet_service.cc:1468] Processing CreateTablet for tablet e249474dcbf642189560c91ec3a99fdf (DEFAULT_TABLE table=default.Table_adb18e9d4ac64721b777cd20a2cd0800 [id=ea928501fe954cd0b81d38add4ef7128]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:17.015074 1150 tablet_bootstrap.cc:492] T 606fac5c6efe48d1877104b01c23ba5b P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:17.014967 1151 raft_consensus.cc:357] T 4410da9e95454ecf95faf80221b0a71c P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:17.015257 1150 ts_tablet_manager.cc:1397] T 606fac5c6efe48d1877104b01c23ba5b P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.165s user 0.002s sys 0.004s
I20250901 01:52:17.015261 1151 raft_consensus.cc:383] T 4410da9e95454ecf95faf80221b0a71c P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:17.015450 1151 raft_consensus.cc:738] T 4410da9e95454ecf95faf80221b0a71c P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:17.015781 1151 consensus_queue.cc:260] T 4410da9e95454ecf95faf80221b0a71c P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:17.019747 452 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_68f31d33b83544cea7b959bc7c41f94b" } new_table_name: "default.Table_5a72e7206f604795915f3a577d37f874" modify_external_catalogs: true
I20250901 01:52:17.021730 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_762edea68aae4d33ba89049358cfb236"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:17.022441 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_762edea68aae4d33ba89049358cfb236 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:17.022693 1151 ts_tablet_manager.cc:1428] T 4410da9e95454ecf95faf80221b0a71c P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20250901 01:52:17.024216 450 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_271fd01373bf49f2ab8b4faf82db7393" } modify_external_catalogs: true
I20250901 01:52:17.024561 450 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_271fd01373bf49f2ab8b4faf82db7393" } modify_external_catalogs: true
I20250901 01:52:17.025314 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_97c44ea002094df8a6af3dccd0099c39"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:17.026167 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_97c44ea002094df8a6af3dccd0099c39 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:17.027410 1151 tablet_bootstrap.cc:492] T 7442c6ff699141e1aff208ebdc9d192a P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:17.027647 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e249474dcbf642189560c91ec3a99fdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:17.027735 1150 raft_consensus.cc:357] T 606fac5c6efe48d1877104b01c23ba5b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:17.027966 1150 raft_consensus.cc:383] T 606fac5c6efe48d1877104b01c23ba5b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:17.029467 1151 tablet_bootstrap.cc:654] T 7442c6ff699141e1aff208ebdc9d192a P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:17.027704 992 leader_election.cc:336] T 7cc1efcb6bff4aba8015d2d3b7f4c923 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:17.030413 992 consensus_peers.cc:489] T 4fd0e4f9d86c4362af96ce9964b72328 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250901 01:52:17.030535 992 consensus_peers.cc:489] T 467c2bf4a8cb4937b6f627bea28d8ddb P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250901 01:52:17.030615 992 leader_election.cc:336] T 7cc1efcb6bff4aba8015d2d3b7f4c923 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:17.030807 992 leader_election.cc:336] T 756cfe84a517459ca015aecf5cd81571 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:17.030989 992 leader_election.cc:336] T 756cfe84a517459ca015aecf5cd81571 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:17.031183 992 leader_election.cc:336] T 31390f38258c465cb1798615a1fcc790 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:17.031276 992 leader_election.cc:304] T 31390f38258c465cb1798615a1fcc790 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
W20250901 01:52:17.031468 992 raft_consensus.cc:2684] T 31390f38258c465cb1798615a1fcc790 P fd27ab6665304aaea53adbd102dd4f58: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250901 01:52:17.031651 992 leader_election.cc:336] T c27bb23c150f4565a1d0ba8f231cc386 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:17.031847 992 leader_election.cc:336] T c27bb23c150f4565a1d0ba8f231cc386 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:17.031997 992 leader_election.cc:336] T 22691943a134400c8df4617240e5d3ef P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:17.032087 992 leader_election.cc:304] T 22691943a134400c8df4617240e5d3ef P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:17.032213 900 tablet_service.cc:1468] Processing CreateTablet for tablet a8483e7212d94daba7e5401256c33aef (DEFAULT_TABLE table=default.Table_142ccc8cc7414621bc0c31326d5aa0e9 [id=6a01b6c6977546be9c6128ca5f200a92]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:17.033180 1150 raft_consensus.cc:738] T 606fac5c6efe48d1877104b01c23ba5b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:17.033435 1150 consensus_queue.cc:260] T 606fac5c6efe48d1877104b01c23ba5b P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:17.034368 1150 ts_tablet_manager.cc:1428] T 606fac5c6efe48d1877104b01c23ba5b P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.019s user 0.000s sys 0.002s
I20250901 01:52:17.036500 1332 ts_tablet_manager.cc:1939] T 31390f38258c465cb1798615a1fcc790 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:17.037002 1552 raft_consensus.cc:2747] T 22691943a134400c8df4617240e5d3ef P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:17.037084 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8483e7212d94daba7e5401256c33aef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:17.037750 1332 tablet_replica.cc:331] T 9496ec6021784800aef62996b0dee593 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:17.037958 1332 raft_consensus.cc:2241] T 9496ec6021784800aef62996b0dee593 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:17.038116 1332 raft_consensus.cc:2270] T 9496ec6021784800aef62996b0dee593 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:17.038650 1332 ts_tablet_manager.cc:1905] T 9496ec6021784800aef62996b0dee593 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:17.039152 992 leader_election.cc:336] T 6c15cd14831f422eb615038603c79ed0 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:17.039300 992 leader_election.cc:304] T 6c15cd14831f422eb615038603c79ed0 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:17.039992 1552 raft_consensus.cc:2747] T 6c15cd14831f422eb615038603c79ed0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:17.040277 992 leader_election.cc:336] T 84fe314502444bd88b038b87eead9cb5 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:17.040431 992 leader_election.cc:304] T 84fe314502444bd88b038b87eead9cb5 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
W20250901 01:52:17.040606 992 raft_consensus.cc:2684] T 84fe314502444bd88b038b87eead9cb5 P fd27ab6665304aaea53adbd102dd4f58: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250901 01:52:17.041121 992 consensus_peers.cc:489] T 061b760d8e6647c6ab0b5d538840a230 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250901 01:52:17.041378 992 leader_election.cc:336] T 447ffe73741640ceb73f35676232a094 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:17.041616 992 leader_election.cc:336] T 447ffe73741640ceb73f35676232a094 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:17.042654 890 tablet_service.cc:1468] Processing CreateTablet for tablet 6cf09e8f299548dba2cce16d7b565bad (DEFAULT_TABLE table=default.Table_749fcd4bf97d439c9fd49b138a52a3fd [id=24fffc7ebf5a47bdb37211910301acf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:17.042532 899 tablet_service.cc:1468] Processing CreateTablet for tablet 7a2f6e2c7ee2402ea310c058e046416c (DEFAULT_TABLE table=default.Table_a89cc8e48563460baabc34c821ceae35 [id=9c3930e9fcfe4b49ba4bea67a2b0ad07]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:17.043874 1150 tablet_bootstrap.cc:492] T 82d163681fda47f38d79863bd28f576d P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:17.046031 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6cf09e8f299548dba2cce16d7b565bad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:17.046836 898 tablet_service.cc:1468] Processing CreateTablet for tablet 8ca5cf4e99014c0d84a3acad4ef713af (DEFAULT_TABLE table=default.Table_746f677d34894fc0a027945dd2c04b53 [id=303e76c918bc4c39aa47a4d24849a358]), partition=RANGE (key) PARTITION UNBOUNDED
W20250901 01:52:17.052194 858 leader_election.cc:336] T c27bb23c150f4565a1d0ba8f231cc386 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:17.052592 992 leader_election.cc:336] T f0affc7a78634607852a48458412761d P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:17.052909 992 leader_election.cc:336] T f0affc7a78634607852a48458412761d P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:17.053515 992 leader_election.cc:336] T 342b7fde3ede494b9eec6e361db255dc P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:17.045671 889 tablet_service.cc:1468] Processing CreateTablet for tablet eaa521c52a2f4408b59770bd2a05907c (DEFAULT_TABLE table=default.Table_4e1d0964cecf42e99b61dc169027cee1 [id=a33b9b981bd64d7981829d663b0b48b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:17.056114 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ca5cf4e99014c0d84a3acad4ef713af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:17.056133 992 leader_election.cc:304] T 342b7fde3ede494b9eec6e361db255dc P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
W20250901 01:52:17.056309 992 raft_consensus.cc:2684] T 342b7fde3ede494b9eec6e361db255dc P fd27ab6665304aaea53adbd102dd4f58: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250901 01:52:17.056942 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a2f6e2c7ee2402ea310c058e046416c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:17.056857 858 leader_election.cc:304] T c27bb23c150f4565a1d0ba8f231cc386 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:17.060153 992 leader_election.cc:336] T 4d333b670df84f308f0cbf6da636b768 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:17.060276 992 leader_election.cc:304] T 4d333b670df84f308f0cbf6da636b768 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
W20250901 01:52:17.060513 992 raft_consensus.cc:2684] T 4d333b670df84f308f0cbf6da636b768 P fd27ab6665304aaea53adbd102dd4f58: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250901 01:52:17.061120 992 leader_election.cc:336] T 4cdd369214ef4f5ebabddbf28b2ae3f7 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:17.061249 992 leader_election.cc:304] T 4cdd369214ef4f5ebabddbf28b2ae3f7 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
W20250901 01:52:17.061429 992 raft_consensus.cc:2684] T 4cdd369214ef4f5ebabddbf28b2ae3f7 P fd27ab6665304aaea53adbd102dd4f58: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250901 01:52:17.062016 992 leader_election.cc:336] T 85322a864d154f299f2d670fdd2a7bfb P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:17.062253 992 leader_election.cc:336] T 85322a864d154f299f2d670fdd2a7bfb P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:17.062445 992 leader_election.cc:336] T b29e846dfae44f0e9ce3e5fea3763a4f P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:17.062532 992 leader_election.cc:304] T b29e846dfae44f0e9ce3e5fea3763a4f P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
W20250901 01:52:17.062692 992 raft_consensus.cc:2684] T b29e846dfae44f0e9ce3e5fea3763a4f P fd27ab6665304aaea53adbd102dd4f58: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250901 01:52:17.063079 992 leader_election.cc:336] T 88a6ab29d00542aba54f0e4185fbf615 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:17.063192 992 leader_election.cc:304] T 88a6ab29d00542aba54f0e4185fbf615 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
W20250901 01:52:17.063318 992 raft_consensus.cc:2684] T 88a6ab29d00542aba54f0e4185fbf615 P fd27ab6665304aaea53adbd102dd4f58: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250901 01:52:17.063757 992 leader_election.cc:336] T 4fd0e4f9d86c4362af96ce9964b72328 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:17.065260 1151 tablet_bootstrap.cc:492] T 7442c6ff699141e1aff208ebdc9d192a P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:17.065485 1151 ts_tablet_manager.cc:1397] T 7442c6ff699141e1aff208ebdc9d192a P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.038s user 0.005s sys 0.000s
I20250901 01:52:17.066272 1151 raft_consensus.cc:357] T 7442c6ff699141e1aff208ebdc9d192a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:17.066566 1151 raft_consensus.cc:383] T 7442c6ff699141e1aff208ebdc9d192a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:17.066700 1151 raft_consensus.cc:738] T 7442c6ff699141e1aff208ebdc9d192a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:17.066963 1151 consensus_queue.cc:260] T 7442c6ff699141e1aff208ebdc9d192a P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:17.069090 448 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_e2ea7aa90231448888c6cfead7e3d9c1" } modify_external_catalogs: true
I20250901 01:52:17.069331 448 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_e2ea7aa90231448888c6cfead7e3d9c1" } modify_external_catalogs: true
I20250901 01:52:17.070184 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eaa521c52a2f4408b59770bd2a05907c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:17.071331 455 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_a0e83c6959d64160a5fdec67e0e49ba3" } modify_external_catalogs: true
I20250901 01:52:17.071492 455 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_a0e83c6959d64160a5fdec67e0e49ba3" } modify_external_catalogs: true
I20250901 01:52:17.073424 1293 raft_consensus.cc:2747] T c27bb23c150f4565a1d0ba8f231cc386 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:17.074424 1151 ts_tablet_manager.cc:1428] T 7442c6ff699141e1aff208ebdc9d192a P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20250901 01:52:17.075935 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_cbeebbc633854c008e07b7d39d2024ee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:17.076300 1150 tablet_bootstrap.cc:654] T 82d163681fda47f38d79863bd28f576d P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:17.077811 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cbeebbc633854c008e07b7d39d2024ee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:17.081601 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet b541a6e14f114ab29a864cca11806316 (table default.Table_0e63d21acfcb4c028c9ea6406b6a04e4 [id=81323e6fbe424655b8e185f52657fadc]) successfully deleted
W20250901 01:52:17.081934 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 40d74d7b5a3b4d1dae09eebd9e07a0c4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 40d74d7b5a3b4d1dae09eebd9e07a0c4 already in progress: creating tablet
I20250901 01:52:17.082345 1151 tablet_bootstrap.cc:492] T 2f28451a3ecc481a958588466f7fec77 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
W20250901 01:52:17.083595 858 leader_election.cc:336] T 5adf6b492640417ca071c94702b48cb9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:17.084151 1151 tablet_bootstrap.cc:654] T 2f28451a3ecc481a958588466f7fec77 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:17.087191 1332 ts_tablet_manager.cc:1918] T 9496ec6021784800aef62996b0dee593 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:17.087196 902 tablet_service.cc:1468] Processing CreateTablet for tablet 081e2630e284498aa819f85bf8326315 (DEFAULT_TABLE table=default.Table_537be4a82c0348f6952b911b167d02ad [id=8199118617cb4707bc3ae9b1363750f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:17.088302 904 tablet_service.cc:1468] Processing CreateTablet for tablet 641a83a3e50d451189c33c7b7b4ca070 (DEFAULT_TABLE table=default.Table_fc0336a00d084f6e98e1d8d32142e66f [id=0e69044796ea4292b1dc100db2f26cee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:17.088640 1332 log.cc:1199] T 9496ec6021784800aef62996b0dee593 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/9496ec6021784800aef62996b0dee593
I20250901 01:52:17.089208 890 tablet_service.cc:1468] Processing CreateTablet for tablet 8ed6616ce54548ba83ba299c4ecf3c38 (DEFAULT_TABLE table=default.Table_54591f7e2f0f4649a25e6beac813f588 [id=946ec3e53dcd4de4b8501b3e87e99072]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:17.089818 1332 ts_tablet_manager.cc:1939] T 9496ec6021784800aef62996b0dee593 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:17.090348 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 081e2630e284498aa819f85bf8326315. 1 dirs total, 0 dirs full, 0 dirs failed
W20250901 01:52:17.090783 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet b7b802f976f3471faaaecc964ec87e48 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b7b802f976f3471faaaecc964ec87e48 already in progress: creating tablet
I20250901 01:52:17.091034 897 tablet_service.cc:1468] Processing CreateTablet for tablet 2599f98abfdc4871b573edfc2ad97d22 (DEFAULT_TABLE table=default.Table_0e63d21acfcb4c028c9ea6406b6a04e4 [id=81323e6fbe424655b8e185f52657fadc]), partition=RANGE (key) PARTITION UNBOUNDED
W20250901 01:52:17.091536 449 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 616ff34e8d314e0aa0e11ce60be11789 consensus state has no leader
W20250901 01:52:17.086189 440 catalog_manager.cc:4671] TS 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): CreateTablet RPC failed for tablet 66a48a8ebe154b2cac89c234eaa378ff: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:17.093916 858 leader_election.cc:304] T 5adf6b492640417ca071c94702b48cb9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:17.094970 1293 raft_consensus.cc:2747] T 5adf6b492640417ca071c94702b48cb9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:17.095983 904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 641a83a3e50d451189c33c7b7b4ca070. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:17.097267 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2599f98abfdc4871b573edfc2ad97d22. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:17.097290 992 leader_election.cc:304] T f65ec84958eb488d8b0f812b59b5b3d5 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:17.097985 1332 tablet_replica.cc:331] T 6c15cd14831f422eb615038603c79ed0 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:17.098182 1552 raft_consensus.cc:2802] T f65ec84958eb488d8b0f812b59b5b3d5 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:17.098953 1552 raft_consensus.cc:491] T f65ec84958eb488d8b0f812b59b5b3d5 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:17.099052 1552 raft_consensus.cc:3058] T f65ec84958eb488d8b0f812b59b5b3d5 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:17.102145 1552 raft_consensus.cc:513] T f65ec84958eb488d8b0f812b59b5b3d5 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:17.102862 1552 leader_election.cc:290] T f65ec84958eb488d8b0f812b59b5b3d5 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:17.103197 1332 raft_consensus.cc:2241] T 6c15cd14831f422eb615038603c79ed0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:17.103447 1332 raft_consensus.cc:2270] T 6c15cd14831f422eb615038603c79ed0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:17.104158 451 catalog_manager.cc:6127] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: default.Table_2eba037c989c4095bc355667be2fb109 [id=18aecbf3fe4643c4b589ee47706161bd] alter complete (version 3)
I20250901 01:52:17.104278 1332 ts_tablet_manager.cc:1905] T 6c15cd14831f422eb615038603c79ed0 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:17.104945 441 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 84fe314502444bd88b038b87eead9cb5 (table default.Table_6a5316d194864096bdfdf11e382f69d2 [id=c57384464f9f44f797e98c309b6866f5]) successfully deleted
W20250901 01:52:17.105050 858 leader_election.cc:336] T 4410da9e95454ecf95faf80221b0a71c P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:17.108923 441 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 31390f38258c465cb1798615a1fcc790 (table default.Table_1a83c4b1e2734d038d0309401d0ab08e [id=c1902e6ac21b4a8a8af96d647f85e288]) successfully deleted
I20250901 01:52:17.109032 449 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_c5c427e0288c4f879bd65c0f006c0f5d" } new_table_name: "default.Table_28ee2277a7d442ec97b05865b2661076" modify_external_catalogs: true
I20250901 01:52:17.109458 441 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 9496ec6021784800aef62996b0dee593 (table default.Table_95c28a4758e148368604152622fe51be [id=bbcca84b801942d094d955c87cafacb4]) successfully deleted
I20250901 01:52:17.110944 1023 tablet_service.cc:1468] Processing CreateTablet for tablet eaa521c52a2f4408b59770bd2a05907c (DEFAULT_TABLE table=default.Table_4e1d0964cecf42e99b61dc169027cee1 [id=a33b9b981bd64d7981829d663b0b48b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:17.111368 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eaa521c52a2f4408b59770bd2a05907c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:17.111910 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ed6616ce54548ba83ba299c4ecf3c38. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:17.112025 858 leader_election.cc:304] T 4410da9e95454ecf95faf80221b0a71c P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:17.112432 1293 raft_consensus.cc:2747] T 4410da9e95454ecf95faf80221b0a71c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:17.112872 858 leader_election.cc:336] T 186ec810d02e4748aad15228b5866ac1 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:17.113094 858 leader_election.cc:304] T 186ec810d02e4748aad15228b5866ac1 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:17.113399 1293 raft_consensus.cc:2747] T 186ec810d02e4748aad15228b5866ac1 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:17.113824 858 leader_election.cc:336] T 7442c6ff699141e1aff208ebdc9d192a P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:17.114012 858 leader_election.cc:304] T 7442c6ff699141e1aff208ebdc9d192a P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:17.116169 1293 raft_consensus.cc:2747] T 7442c6ff699141e1aff208ebdc9d192a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:17.116616 858 leader_election.cc:336] T 3b18a7084727490380d26b3485e4e799 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:17.116854 858 leader_election.cc:304] T 3b18a7084727490380d26b3485e4e799 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:17.118431 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 8ed6616ce54548ba83ba299c4ecf3c38 (DEFAULT_TABLE table=default.Table_54591f7e2f0f4649a25e6beac813f588 [id=946ec3e53dcd4de4b8501b3e87e99072]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:17.118942 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ed6616ce54548ba83ba299c4ecf3c38. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:17.120945 1293 raft_consensus.cc:2747] T 3b18a7084727490380d26b3485e4e799 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:17.122264 858 leader_election.cc:336] T d60d571b2c2840a2b20629bb832d1309 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:17.123739 858 leader_election.cc:304] T d60d571b2c2840a2b20629bb832d1309 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:17.125149 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 081e2630e284498aa819f85bf8326315 (DEFAULT_TABLE table=default.Table_537be4a82c0348f6952b911b167d02ad [id=8199118617cb4707bc3ae9b1363750f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:17.125592 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 081e2630e284498aa819f85bf8326315. 1 dirs total, 0 dirs full, 0 dirs failed
W20250901 01:52:17.125288 858 raft_consensus.cc:2684] T d60d571b2c2840a2b20629bb832d1309 P 00c282f779c1477d920f24dc0dc16628: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250901 01:52:17.130570 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_1eb1cf735df04cbdaf0a6a86ef15aa7b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:17.132102 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1eb1cf735df04cbdaf0a6a86ef15aa7b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:17.128356 1306 ts_tablet_manager.cc:1918] T 4d27fb61943c45d4aca67689cc3a9ff3 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250901 01:52:17.133080 858 leader_election.cc:336] T 34042ea5c1b645969c1de41fe587c343 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:17.134928 858 leader_election.cc:304] T 34042ea5c1b645969c1de41fe587c343 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:17.137775 1293 raft_consensus.cc:2747] T 34042ea5c1b645969c1de41fe587c343 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:17.139739 447 master_service.cc:948] ReplaceTablet: received request to replace tablet 8dd587304e6342edbe96c9897b40e5c9 from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:17.139734 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 641a83a3e50d451189c33c7b7b4ca070 (DEFAULT_TABLE table=default.Table_fc0336a00d084f6e98e1d8d32142e66f [id=0e69044796ea4292b1dc100db2f26cee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:17.140148 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 641a83a3e50d451189c33c7b7b4ca070. 1 dirs total, 0 dirs full, 0 dirs failed
W20250901 01:52:17.142298 858 leader_election.cc:336] T 8381305e4c4148808803e255b8a43a90 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:17.143333 858 leader_election.cc:304] T 8381305e4c4148808803e255b8a43a90 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:17.144722 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 2599f98abfdc4871b573edfc2ad97d22 (DEFAULT_TABLE table=default.Table_0e63d21acfcb4c028c9ea6406b6a04e4 [id=81323e6fbe424655b8e185f52657fadc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:17.145148 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2599f98abfdc4871b573edfc2ad97d22. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:17.145149 1293 raft_consensus.cc:2747] T 8381305e4c4148808803e255b8a43a90 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:17.147780 447 master_service.cc:948] ReplaceTablet: received request to replace tablet 0cb3262973934c8689b851adad60afe0 from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:17.148303 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_b362441111a241e19cc1a09bdeefea49"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:17.150278 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b362441111a241e19cc1a09bdeefea49 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:17.152298 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet d58f69fad01f403eab74a47d3695ed83 with delete_type TABLET_DATA_DELETED (replaced by tablet 74b709cb5ec64006b2f12bead8217d80) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:17.152789 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet a774cc6ae3924d299127fd9ffb01ca57 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:16 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:17.153409 1306 log.cc:1199] T 4d27fb61943c45d4aca67689cc3a9ff3 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/4d27fb61943c45d4aca67689cc3a9ff3
W20250901 01:52:17.153537 858 leader_election.cc:336] T 920925ee9b014e14a79ceb7796ff7828 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:17.153915 1306 ts_tablet_manager.cc:1939] T 4d27fb61943c45d4aca67689cc3a9ff3 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:17.153864 858 leader_election.cc:304] T 920925ee9b014e14a79ceb7796ff7828 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:17.154372 858 raft_consensus.cc:2684] T 920925ee9b014e14a79ceb7796ff7828 P 00c282f779c1477d920f24dc0dc16628: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250901 01:52:17.154659 1151 tablet_bootstrap.cc:492] T 2f28451a3ecc481a958588466f7fec77 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:17.154855 1151 ts_tablet_manager.cc:1397] T 2f28451a3ecc481a958588466f7fec77 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.073s user 0.001s sys 0.006s
I20250901 01:52:17.155278 1150 tablet_bootstrap.cc:492] T 82d163681fda47f38d79863bd28f576d P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:17.155436 1150 ts_tablet_manager.cc:1397] T 82d163681fda47f38d79863bd28f576d P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.112s user 0.006s sys 0.005s
I20250901 01:52:17.155798 1151 raft_consensus.cc:357] T 2f28451a3ecc481a958588466f7fec77 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:17.156140 1150 raft_consensus.cc:357] T 82d163681fda47f38d79863bd28f576d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:17.156333 1150 raft_consensus.cc:383] T 82d163681fda47f38d79863bd28f576d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:17.156426 1150 raft_consensus.cc:738] T 82d163681fda47f38d79863bd28f576d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:17.156589 1151 raft_consensus.cc:383] T 2f28451a3ecc481a958588466f7fec77 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:17.156697 1150 consensus_queue.cc:260] T 82d163681fda47f38d79863bd28f576d P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:17.156998 1151 raft_consensus.cc:738] T 2f28451a3ecc481a958588466f7fec77 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
W20250901 01:52:17.156167 992 leader_election.cc:336] T f65ec84958eb488d8b0f812b59b5b3d5 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:17.159140 1151 consensus_queue.cc:260] T 2f28451a3ecc481a958588466f7fec77 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:17.160374 921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f65ec84958eb488d8b0f812b59b5b3d5" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:17.160624 921 raft_consensus.cc:3058] T f65ec84958eb488d8b0f812b59b5b3d5 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:17.161245 1572 raft_consensus.cc:738] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ce2288f784044eea010664e3def133f, State: Initialized, Role: FOLLOWER
I20250901 01:52:17.162281 1572 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 44, Last appended: 2.45, Last appended by leader: 45, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3add7edb97a40ef9f7c6bc011d13227" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 43871 } } peers { permanent_uuid: "d2ae2cb6af0f4b3a8f47ab1cdd6567d3" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42375 } } peers { permanent_uuid: "7ce2288f784044eea010664e3def133f" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 44231 } }
I20250901 01:52:17.163408 921 raft_consensus.cc:2466] T f65ec84958eb488d8b0f812b59b5b3d5 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:17.170399 1658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3add7edb97a40ef9f7c6bc011d13227" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 43871 } } peers { permanent_uuid: "d2ae2cb6af0f4b3a8f47ab1cdd6567d3" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42375 } } peers { permanent_uuid: "7ce2288f784044eea010664e3def133f" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 44231 } } }
I20250901 01:52:17.170997 1151 ts_tablet_manager.cc:1428] T 2f28451a3ecc481a958588466f7fec77 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
I20250901 01:52:17.176688 1150 ts_tablet_manager.cc:1428] T 82d163681fda47f38d79863bd28f576d P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20250901 01:52:17.179303 1658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f [sys.catalog]: This master's current role is: FOLLOWER
I20250901 01:52:17.179107 1572 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f [sys.catalog]: configured and running, proceeding with master startup.
I20250901 01:52:17.188951 992 leader_election.cc:304] T f65ec84958eb488d8b0f812b59b5b3d5 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:17.189283 969 heartbeater.cc:344] Connected to a master server at 127.0.104.60:44231
I20250901 01:52:17.189314 1374 raft_consensus.cc:2802] T f65ec84958eb488d8b0f812b59b5b3d5 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:17.189479 1374 raft_consensus.cc:695] T f65ec84958eb488d8b0f812b59b5b3d5 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:17.189764 1374 consensus_queue.cc:237] T f65ec84958eb488d8b0f812b59b5b3d5 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:17.192773 1150 tablet_bootstrap.cc:492] T b009cc95c819431ea9827a3e453b8da3 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:17.197191 1151 tablet_bootstrap.cc:492] T 3b18a7084727490380d26b3485e4e799 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:17.199604 1150 tablet_bootstrap.cc:654] T b009cc95c819431ea9827a3e453b8da3 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:17.200917 858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:40731, user_credentials={real_user=slave}} blocked reactor thread for 54548.9us
W20250901 01:52:17.201179 858 leader_election.cc:336] T a5f4b0a2d72a4b8095ae4af1fe7c0c5d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:17.201406 858 leader_election.cc:304] T a5f4b0a2d72a4b8095ae4af1fe7c0c5d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:17.201668 858 raft_consensus.cc:2684] T a5f4b0a2d72a4b8095ae4af1fe7c0c5d P 00c282f779c1477d920f24dc0dc16628: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250901 01:52:17.202168 858 leader_election.cc:336] T 7cc1efcb6bff4aba8015d2d3b7f4c923 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:17.202399 1151 tablet_bootstrap.cc:654] T 3b18a7084727490380d26b3485e4e799 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:17.202522 858 leader_election.cc:304] T 7cc1efcb6bff4aba8015d2d3b7f4c923 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:17.202867 1293 raft_consensus.cc:2747] T 7cc1efcb6bff4aba8015d2d3b7f4c923 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:17.203194 858 leader_election.cc:336] T f65ec84958eb488d8b0f812b59b5b3d5 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:17.203390 858 leader_election.cc:304] T f65ec84958eb488d8b0f812b59b5b3d5 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:17.203653 1293 raft_consensus.cc:2747] T f65ec84958eb488d8b0f812b59b5b3d5 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:17.203924 858 leader_election.cc:336] T 88a6ab29d00542aba54f0e4185fbf615 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:17.204095 858 leader_election.cc:304] T 88a6ab29d00542aba54f0e4185fbf615 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:17.204290 858 raft_consensus.cc:2684] T 88a6ab29d00542aba54f0e4185fbf615 P 00c282f779c1477d920f24dc0dc16628: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250901 01:52:17.204819 858 leader_election.cc:336] T 85322a864d154f299f2d670fdd2a7bfb P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:17.205117 440 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.0.104.1:40731, user_credentials={real_user=slave}} blocked reactor thread for 44372.4us
I20250901 01:52:17.214540 858 leader_election.cc:304] T 85322a864d154f299f2d670fdd2a7bfb P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:17.154414 1306 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52846 (request call id 305) took 1118 ms. Trace:
I20250901 01:52:17.215025 1306 rpcz_store.cc:276] 0901 01:52:16.035532 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:16.035968 (+ 436us) service_pool.cc:225] Handling call
0901 01:52:16.933817 (+897849us) ts_tablet_manager.cc:1145] Deleting tablet 4d27fb61943c45d4aca67689cc3a9ff3
0901 01:52:16.933827 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:17.052157 (+118330us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:17.128257 (+ 76100us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:17.154357 (+ 26100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":878598}
I20250901 01:52:17.215829 1306 tablet_replica.cc:331] T b633b461dbf34e13bc6e1cc08e6d20c1 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:17.216184 1306 raft_consensus.cc:2241] T b633b461dbf34e13bc6e1cc08e6d20c1 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:17.216473 1293 raft_consensus.cc:2747] T 85322a864d154f299f2d670fdd2a7bfb P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:17.216840 1306 raft_consensus.cc:2270] T b633b461dbf34e13bc6e1cc08e6d20c1 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:17.217844 1306 ts_tablet_manager.cc:1905] T b633b461dbf34e13bc6e1cc08e6d20c1 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:17.220662 858 leader_election.cc:336] T 756cfe84a517459ca015aecf5cd81571 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:17.222090 858 leader_election.cc:304] T 756cfe84a517459ca015aecf5cd81571 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:17.222409 1293 raft_consensus.cc:2747] T 756cfe84a517459ca015aecf5cd81571 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:17.222807 858 leader_election.cc:336] T 84fe314502444bd88b038b87eead9cb5 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:17.222983 858 leader_election.cc:304] T 84fe314502444bd88b038b87eead9cb5 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:17.223205 858 raft_consensus.cc:2684] T 84fe314502444bd88b038b87eead9cb5 P 00c282f779c1477d920f24dc0dc16628: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250901 01:52:17.221704 992 leader_election.cc:336] T f65ec84958eb488d8b0f812b59b5b3d5 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:17.229832 858 leader_election.cc:336] T 0e698923812841b7a5d37d57da103c5c P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:17.230330 449 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ad45c90ba0da43c9860745e534664a98 consensus state has no leader
I20250901 01:52:17.230692 449 master_service.cc:948] ReplaceTablet: received request to replace tablet 2240302d9178445d8700e35f5ea274f7 from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:17.231947 858 leader_election.cc:304] T 0e698923812841b7a5d37d57da103c5c P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:17.230873 1150 tablet_bootstrap.cc:492] T b009cc95c819431ea9827a3e453b8da3 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:17.232281 1150 ts_tablet_manager.cc:1397] T b009cc95c819431ea9827a3e453b8da3 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.040s user 0.004s sys 0.001s
I20250901 01:52:17.233260 1150 raft_consensus.cc:357] T b009cc95c819431ea9827a3e453b8da3 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:17.233538 1150 raft_consensus.cc:383] T b009cc95c819431ea9827a3e453b8da3 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:17.233718 1150 raft_consensus.cc:738] T b009cc95c819431ea9827a3e453b8da3 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:17.235801 1150 consensus_queue.cc:260] T b009cc95c819431ea9827a3e453b8da3 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:17.239741 1332 ts_tablet_manager.cc:1918] T 6c15cd14831f422eb615038603c79ed0 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:17.239892 1332 log.cc:1199] T 6c15cd14831f422eb615038603c79ed0 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/6c15cd14831f422eb615038603c79ed0
I20250901 01:52:17.240269 1332 ts_tablet_manager.cc:1939] T 6c15cd14831f422eb615038603c79ed0 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:17.243294 1374 raft_consensus.cc:491] T 1f18420261ae41459e1f25b927bd84a9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:17.243367 899 tablet_service.cc:1515] Processing DeleteTablet for tablet d58f69fad01f403eab74a47d3695ed83 with delete_type TABLET_DATA_DELETED (replaced by tablet 74b709cb5ec64006b2f12bead8217d80) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:17.243459 1374 raft_consensus.cc:513] T 1f18420261ae41459e1f25b927bd84a9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:17.243752 899 tablet_service.cc:1515] Processing DeleteTablet for tablet a774cc6ae3924d299127fd9ffb01ca57 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:16 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:17.244076 1374 leader_election.cc:290] T 1f18420261ae41459e1f25b927bd84a9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:17.245836 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_fea398de1f1948809aebf95c387f4b17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:17.246636 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fea398de1f1948809aebf95c387f4b17 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:17.247140 1150 ts_tablet_manager.cc:1428] T b009cc95c819431ea9827a3e453b8da3 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20250901 01:52:17.248510 447 master_service.cc:948] ReplaceTablet: received request to replace tablet d58f69fad01f403eab74a47d3695ed83 from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:17.250133 453 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_54591f7e2f0f4649a25e6beac813f588" } new_table_name: "default.Table_8cd9c1e6a8e74232b32d36451269562a" modify_external_catalogs: true
I20250901 01:52:17.254159 1150 tablet_bootstrap.cc:492] T 5a8c21aee9f84216b6f90538a5bf4c17 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
W20250901 01:52:17.257465 858 raft_consensus.cc:2684] T 0e698923812841b7a5d37d57da103c5c P 00c282f779c1477d920f24dc0dc16628: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250901 01:52:17.263149 858 leader_election.cc:336] T d945b310fe6344af9cc9dd68f84c4bdf P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:17.266202 453 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8ed6616ce54548ba83ba299c4ecf3c38 consensus state has no leader
I20250901 01:52:17.266618 453 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_746f677d34894fc0a027945dd2c04b53" } new_table_name: "default.Table_ed6fd284be6f4ae3a008e21e7ce17930" modify_external_catalogs: true
I20250901 01:52:17.267975 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_d56ff20d129b440cbab7b15c4af6fa46"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:17.269009 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d56ff20d129b440cbab7b15c4af6fa46 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:17.274546 1150 tablet_bootstrap.cc:654] T 5a8c21aee9f84216b6f90538a5bf4c17 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:17.274613 858 leader_election.cc:304] T d945b310fe6344af9cc9dd68f84c4bdf P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:17.274880 858 raft_consensus.cc:2684] T d945b310fe6344af9cc9dd68f84c4bdf P 00c282f779c1477d920f24dc0dc16628: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250901 01:52:17.275523 858 consensus_peers.cc:489] T 24ddc13d1bb04db4a7850acbff091577 P 00c282f779c1477d920f24dc0dc16628 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250901 01:52:17.275717 858 leader_election.cc:336] T 60285f2ba037433999f5455fb9cc7acd P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:17.275852 858 leader_election.cc:304] T 60285f2ba037433999f5455fb9cc7acd P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:17.276105 858 raft_consensus.cc:2684] T 60285f2ba037433999f5455fb9cc7acd P 00c282f779c1477d920f24dc0dc16628: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250901 01:52:17.276749 858 leader_election.cc:336] T 49d30da95b14400b845270122487a2a0 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:17.276919 858 leader_election.cc:304] T 49d30da95b14400b845270122487a2a0 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:17.277225 1293 raft_consensus.cc:2747] T 49d30da95b14400b845270122487a2a0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:17.277268 858 leader_election.cc:336] T b633b461dbf34e13bc6e1cc08e6d20c1 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:17.277575 858 leader_election.cc:304] T b633b461dbf34e13bc6e1cc08e6d20c1 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:17.277817 858 raft_consensus.cc:2684] T b633b461dbf34e13bc6e1cc08e6d20c1 P 00c282f779c1477d920f24dc0dc16628: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250901 01:52:17.278066 858 leader_election.cc:336] T 824229183e024feaa975c7f0387c9c1a P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:17.278219 858 leader_election.cc:304] T 824229183e024feaa975c7f0387c9c1a P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:17.278450 1293 raft_consensus.cc:2747] T 824229183e024feaa975c7f0387c9c1a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:17.284571 453 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8ca5cf4e99014c0d84a3acad4ef713af consensus state has no leader
I20250901 01:52:17.285164 453 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_3df3e67f179f46a8be496e3bfe3cb89f" } modify_external_catalogs: true
I20250901 01:52:17.285422 453 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_3df3e67f179f46a8be496e3bfe3cb89f" } modify_external_catalogs: true
I20250901 01:52:17.286864 1150 tablet_bootstrap.cc:492] T 5a8c21aee9f84216b6f90538a5bf4c17 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:17.287036 1150 ts_tablet_manager.cc:1397] T 5a8c21aee9f84216b6f90538a5bf4c17 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.007s
I20250901 01:52:17.287905 1150 raft_consensus.cc:357] T 5a8c21aee9f84216b6f90538a5bf4c17 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:17.288126 1150 raft_consensus.cc:383] T 5a8c21aee9f84216b6f90538a5bf4c17 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:17.288209 1150 raft_consensus.cc:738] T 5a8c21aee9f84216b6f90538a5bf4c17 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:17.288498 1150 consensus_queue.cc:260] T 5a8c21aee9f84216b6f90538a5bf4c17 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:17.289057 1150 ts_tablet_manager.cc:1428] T 5a8c21aee9f84216b6f90538a5bf4c17 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250901 01:52:17.289400 1150 tablet_bootstrap.cc:492] T abe27e749fa0485195e0064a49d5431e P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:17.289796 452 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_a5d705c4e57a44b8aca8eb68c601e4ae" } modify_external_catalogs: true
I20250901 01:52:17.290547 452 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_a5d705c4e57a44b8aca8eb68c601e4ae" } modify_external_catalogs: true
I20250901 01:52:17.295416 1150 tablet_bootstrap.cc:654] T abe27e749fa0485195e0064a49d5431e P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:17.295840 1293 raft_consensus.cc:491] T 533d14f5903e49599a34ea353ffd8c00 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:17.296034 1293 raft_consensus.cc:513] T 533d14f5903e49599a34ea353ffd8c00 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:17.296780 1293 leader_election.cc:290] T 533d14f5903e49599a34ea353ffd8c00 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:17.304524 1306 ts_tablet_manager.cc:1918] T b633b461dbf34e13bc6e1cc08e6d20c1 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:17.305763 1306 log.cc:1199] T b633b461dbf34e13bc6e1cc08e6d20c1 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/b633b461dbf34e13bc6e1cc08e6d20c1
I20250901 01:52:17.306361 1306 ts_tablet_manager.cc:1939] T b633b461dbf34e13bc6e1cc08e6d20c1 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:17.307885 1374 raft_consensus.cc:491] T 34042ea5c1b645969c1de41fe587c343 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:17.308104 1374 raft_consensus.cc:513] T 34042ea5c1b645969c1de41fe587c343 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:17.304857 858 leader_election.cc:336] T b541a6e14f114ab29a864cca11806316 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:17.308502 858 leader_election.cc:304] T b541a6e14f114ab29a864cca11806316 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:17.308820 858 raft_consensus.cc:2684] T b541a6e14f114ab29a864cca11806316 P 00c282f779c1477d920f24dc0dc16628: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250901 01:52:17.308887 1374 leader_election.cc:290] T 34042ea5c1b645969c1de41fe587c343 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:17.309636 858 leader_election.cc:336] T 4a87e68850c4459eb2c8eafac89c6aa9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:17.310138 858 leader_election.cc:304] T 4a87e68850c4459eb2c8eafac89c6aa9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:17.309986 1306 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52846 (request call id 306) took 1200 ms. Trace:
I20250901 01:52:17.310483 1306 rpcz_store.cc:276] 0901 01:52:16.109538 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:16.109587 (+ 49us) service_pool.cc:225] Handling call
0901 01:52:17.215716 (+1106129us) ts_tablet_manager.cc:1145] Deleting tablet b633b461dbf34e13bc6e1cc08e6d20c1
0901 01:52:17.215732 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:17.258001 (+ 42269us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:17.288221 (+ 30220us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:17.309940 (+ 21719us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":362,"tablet-delete.queue_time_us":1105093}
I20250901 01:52:17.311219 1306 tablet_replica.cc:331] T 12c04e0c1f804179ae23509cdb24d395 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:17.311404 1293 raft_consensus.cc:2747] T 4a87e68850c4459eb2c8eafac89c6aa9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:17.312132 1306 raft_consensus.cc:2241] T 12c04e0c1f804179ae23509cdb24d395 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:17.312366 1306 raft_consensus.cc:2270] T 12c04e0c1f804179ae23509cdb24d395 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:17.314594 1306 ts_tablet_manager.cc:1905] T 12c04e0c1f804179ae23509cdb24d395 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:17.311645 858 leader_election.cc:336] T a774cc6ae3924d299127fd9ffb01ca57 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:17.316175 858 leader_election.cc:304] T a774cc6ae3924d299127fd9ffb01ca57 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:17.316226 440 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.0.104.2:40561, user_credentials={real_user=slave}} blocked reactor thread for 59023.3us
I20250901 01:52:17.316610 1293 raft_consensus.cc:2747] T a774cc6ae3924d299127fd9ffb01ca57 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:17.316977 858 leader_election.cc:336] T 6c05eff658924ca4a9a1e7a37f917d67 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:17.317202 858 leader_election.cc:304] T 6c05eff658924ca4a9a1e7a37f917d67 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:17.317484 1293 raft_consensus.cc:2747] T 6c05eff658924ca4a9a1e7a37f917d67 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:17.317960 858 leader_election.cc:336] T 12c04e0c1f804179ae23509cdb24d395 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:17.318027 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 4d27fb61943c45d4aca67689cc3a9ff3 (table default.Table_0e0c69a04fea4f5fab174e11a35bf628 [id=78de546347e74cc89be668054e292f04]) successfully deleted
I20250901 01:52:17.318499 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet b633b461dbf34e13bc6e1cc08e6d20c1 (table default.Table_52733c9a363b4ed9b6ff11ad8571e421 [id=453153553bea48319e132857f628d48a]) successfully deleted
I20250901 01:52:17.322835 858 leader_election.cc:304] T 12c04e0c1f804179ae23509cdb24d395 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:17.323206 858 raft_consensus.cc:2684] T 12c04e0c1f804179ae23509cdb24d395 P 00c282f779c1477d920f24dc0dc16628: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250901 01:52:17.323522 858 leader_election.cc:336] T 4d27fb61943c45d4aca67689cc3a9ff3 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:17.323699 858 leader_election.cc:304] T 4d27fb61943c45d4aca67689cc3a9ff3 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:17.323954 858 raft_consensus.cc:2684] T 4d27fb61943c45d4aca67689cc3a9ff3 P 00c282f779c1477d920f24dc0dc16628: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250901 01:52:17.324647 858 leader_election.cc:336] T a59ac57aebe24860aa02f53ce0485910 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:17.332067 858 leader_election.cc:304] T a59ac57aebe24860aa02f53ce0485910 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:17.333484 1293 raft_consensus.cc:2747] T a59ac57aebe24860aa02f53ce0485910 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:17.333920 858 leader_election.cc:336] T fcb7b5b7d63d4f0b906fea97cb548de2 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:17.334169 858 leader_election.cc:304] T fcb7b5b7d63d4f0b906fea97cb548de2 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:17.334470 1293 raft_consensus.cc:2747] T fcb7b5b7d63d4f0b906fea97cb548de2 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:17.334544 449 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 2240302d9178445d8700e35f5ea274f7
W20250901 01:52:17.334872 858 leader_election.cc:336] T 154574e17f214cc0ad56e6e6be3af899 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:17.335134 449 catalog_manager.cc:6652] ReplaceTablet: tablet 2240302d9178445d8700e35f5ea274f7 deleted and replaced by tablet b13afacd37474781877b9f2cd20b313a
I20250901 01:52:17.335114 858 leader_election.cc:304] T 154574e17f214cc0ad56e6e6be3af899 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:17.335399 858 raft_consensus.cc:2684] T 154574e17f214cc0ad56e6e6be3af899 P 00c282f779c1477d920f24dc0dc16628: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250901 01:52:17.335587 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_d28d7742171c4465a8b438d2e337b5f7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:17.336340 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d28d7742171c4465a8b438d2e337b5f7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250901 01:52:17.336359 858 leader_election.cc:336] T b29e846dfae44f0e9ce3e5fea3763a4f P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:17.338097 858 leader_election.cc:304] T b29e846dfae44f0e9ce3e5fea3763a4f P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:17.338572 858 raft_consensus.cc:2684] T b29e846dfae44f0e9ce3e5fea3763a4f P 00c282f779c1477d920f24dc0dc16628: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250901 01:52:17.339319 858 consensus_peers.cc:489] T b11e8f77243e4764b96548d1132b8e5a P 00c282f779c1477d920f24dc0dc16628 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:17.344168 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 3b18a7084727490380d26b3485e4e799
I20250901 01:52:17.344763 454 catalog_manager.cc:6652] ReplaceTablet: tablet 3b18a7084727490380d26b3485e4e799 deleted and replaced by tablet b164b4a67a9143ceaff69b0ce49d63fe
W20250901 01:52:17.344990 858 leader_election.cc:336] T 533d14f5903e49599a34ea353ffd8c00 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:17.345212 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_c945e4cbcdb74acfa10533ff0bb3c0b2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:17.345947 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c945e4cbcdb74acfa10533ff0bb3c0b2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:17.331884 1306 ts_tablet_manager.cc:1918] T 12c04e0c1f804179ae23509cdb24d395 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:17.346233 1306 log.cc:1199] T 12c04e0c1f804179ae23509cdb24d395 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/12c04e0c1f804179ae23509cdb24d395
I20250901 01:52:17.348661 1306 ts_tablet_manager.cc:1939] T 12c04e0c1f804179ae23509cdb24d395 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:17.349717 1306 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52846 (request call id 307) took 1237 ms. Trace:
I20250901 01:52:17.349889 1306 rpcz_store.cc:276] 0901 01:52:16.112596 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:16.112649 (+ 53us) service_pool.cc:225] Handling call
0901 01:52:17.311136 (+1198487us) ts_tablet_manager.cc:1145] Deleting tablet 12c04e0c1f804179ae23509cdb24d395
0901 01:52:17.311148 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:17.327583 (+ 16435us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:17.331737 (+ 4154us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:17.349655 (+ 17918us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1197842}
I20250901 01:52:17.352998 1306 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52846 (request call id 310) took 1235 ms. Trace:
I20250901 01:52:17.353185 1306 rpcz_store.cc:276] 0901 01:52:16.117478 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:16.117551 (+ 73us) service_pool.cc:225] Handling call
0901 01:52:17.352786 (+1235235us) ts_tablet_manager.cc:1145] Deleting tablet f06a77ecc841462396a8d5026c7327f4
0901 01:52:17.352800 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:17.352942 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1233643}
I20250901 01:52:17.353822 1306 tablet_replica.cc:331] T 4a87e68850c4459eb2c8eafac89c6aa9 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:17.354130 1306 raft_consensus.cc:2241] T 4a87e68850c4459eb2c8eafac89c6aa9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:17.362375 1306 raft_consensus.cc:2270] T 4a87e68850c4459eb2c8eafac89c6aa9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:17.362802 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_2f266fe78a3c41c2ab134faad48a377d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:17.363396 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2f266fe78a3c41c2ab134faad48a377d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:17.367388 1306 ts_tablet_manager.cc:1905] T 4a87e68850c4459eb2c8eafac89c6aa9 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:17.387703 1306 ts_tablet_manager.cc:1918] T 4a87e68850c4459eb2c8eafac89c6aa9 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:17.387969 1306 log.cc:1199] T 4a87e68850c4459eb2c8eafac89c6aa9 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/4a87e68850c4459eb2c8eafac89c6aa9
I20250901 01:52:17.388484 1306 ts_tablet_manager.cc:1939] T 4a87e68850c4459eb2c8eafac89c6aa9 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:17.389693 1306 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52846 (request call id 308) took 1274 ms. Trace:
I20250901 01:52:17.389902 1306 rpcz_store.cc:276] 0901 01:52:16.115278 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:16.116626 (+ 1348us) service_pool.cc:225] Handling call
0901 01:52:17.353723 (+1237097us) ts_tablet_manager.cc:1145] Deleting tablet 4a87e68850c4459eb2c8eafac89c6aa9
0901 01:52:17.353735 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:17.375483 (+ 21748us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:17.387515 (+ 12032us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:17.389648 (+ 2133us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1228062}
I20250901 01:52:17.392345 454 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_12f764717f384ee4b906dec11b1212f8" } new_table_name: "default.Table_98d60921500349bcb1f8c1e713314b07" modify_external_catalogs: true
I20250901 01:52:17.401657 447 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_db4aaba6e67e436dab76b895e3443bdd" } new_table_name: "default.Table_14bc71ec066c420b8f3f1e5a5305a0ae" modify_external_catalogs: true
I20250901 01:52:17.406296 446 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet f65ec84958eb488d8b0f812b59b5b3d5 (table default.Table_a3ff88e352884abdba91e6d03acbae22 [id=d1cd5288671540c0a95f7a037363014e]). Expected version 2 got 0
I20250901 01:52:17.406891 446 catalog_manager.cc:5582] T 7cc1efcb6bff4aba8015d2d3b7f4c923 P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: UNKNOWN } } }
I20250901 01:52:17.407423 446 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 7cc1efcb6bff4aba8015d2d3b7f4c923 (table default.Table_5610feff8a344dc9be4bc1f0ad8ddd51 [id=78b00856a9b747c19f390b4fdae8ddcf]). Expected version 3 got 0
I20250901 01:52:17.407806 446 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 4410da9e95454ecf95faf80221b0a71c (table default.Table_420b0979d6fd42da9b320456f4ccc9ce [id=d2004563c4374790954327f05c305cc6]). Expected version 1 got 0
I20250901 01:52:17.408114 446 catalog_manager.cc:5582] T c27bb23c150f4565a1d0ba8f231cc386 P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: UNKNOWN } } }
I20250901 01:52:17.409889 446 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 34042ea5c1b645969c1de41fe587c343 (table default.Table_e37cc89ad7ee4d96852b121d67b06248 [id=edbc04078c904f29a16a1bbb354dad92]). Expected version 1 got 0
I20250901 01:52:17.410149 446 catalog_manager.cc:5582] T 756cfe84a517459ca015aecf5cd81571 P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: UNKNOWN } } }
I20250901 01:52:17.410684 1306 tablet_replica.cc:331] T a774cc6ae3924d299127fd9ffb01ca57 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:17.410882 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 467c2bf4a8cb4937b6f627bea28d8ddb
I20250901 01:52:17.411715 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_fb5ee5a5a9514ddaac740c194a820fd2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:17.410959 1306 raft_consensus.cc:2241] T a774cc6ae3924d299127fd9ffb01ca57 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:17.412297 1306 raft_consensus.cc:2270] T a774cc6ae3924d299127fd9ffb01ca57 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250901 01:52:17.412453 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fb5ee5a5a9514ddaac740c194a820fd2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:17.413179 1306 ts_tablet_manager.cc:1905] T a774cc6ae3924d299127fd9ffb01ca57 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:17.414245 441 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 615b2306fca84003b264da9d5f95ddab (table default.Table_0f959237963c4e33932e193285c0eacb [id=cecdd8d6ca6d445c9fd82decd2a95366], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 615b2306fca84003b264da9d5f95ddab consensus state has no leader
I20250901 01:52:17.416471 451 master_service.cc:948] ReplaceTablet: received request to replace tablet 98cca86164db4252b1b49de88747822a from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:17.421298 452 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet eeae5294bd474f368f5766c260ccf3fe
I20250901 01:52:17.426513 452 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 606fac5c6efe48d1877104b01c23ba5b (table default.Table_7c12d66048a6470e8c227dc1d9f45fa4 [id=91472f220fe44297b79d3eec8b9b80e8]). Expected version 1 got 0
I20250901 01:52:17.426894 1374 raft_consensus.cc:491] T ceb724d859d24d569598e18389dae3b3 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:17.427157 1374 raft_consensus.cc:513] T ceb724d859d24d569598e18389dae3b3 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:17.427773 1293 raft_consensus.cc:491] T 82d163681fda47f38d79863bd28f576d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:17.428000 1374 leader_election.cc:290] T ceb724d859d24d569598e18389dae3b3 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:17.427982 1293 raft_consensus.cc:513] T 82d163681fda47f38d79863bd28f576d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:17.428908 1293 leader_election.cc:290] T 82d163681fda47f38d79863bd28f576d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:17.429484 452 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet b009cc95c819431ea9827a3e453b8da3 (table default.Table_203547c9458e43b3b48ea051d56892c6 [id=cc26c89db2484a6fad919231116285f2]). Expected version 1 got 0
W20250901 01:52:17.430660 858 leader_election.cc:336] T 82d163681fda47f38d79863bd28f576d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:17.443041 1306 ts_tablet_manager.cc:1918] T a774cc6ae3924d299127fd9ffb01ca57 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:17.443300 1306 log.cc:1199] T a774cc6ae3924d299127fd9ffb01ca57 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/a774cc6ae3924d299127fd9ffb01ca57
I20250901 01:52:17.448949 1306 ts_tablet_manager.cc:1939] T a774cc6ae3924d299127fd9ffb01ca57 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
W20250901 01:52:17.458966 454 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a54af99bc10045e8b0b6868e5aed98b4 consensus state has no leader
I20250901 01:52:17.459301 449 master_service.cc:948] ReplaceTablet: received request to replace tablet 82d163681fda47f38d79863bd28f576d from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:17.459501 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_c931e9dfbef04fd3b49aa4f43db3d981"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:17.460014 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_7be95f26c04746828f38a19ddb67afc0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:17.460753 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7be95f26c04746828f38a19ddb67afc0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:17.461886 452 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_bda1777b1e9c409ea39c00de92706b50" } new_table_name: "default.Table_139f6d174d794a6e9a618e90587d0c0d" modify_external_catalogs: true
W20250901 01:52:17.460112 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c931e9dfbef04fd3b49aa4f43db3d981 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:17.485365 1670 tablet.cc:1722] T 467c2bf4a8cb4937b6f627bea28d8ddb P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:17.490945 1150 tablet_bootstrap.cc:492] T abe27e749fa0485195e0064a49d5431e P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:17.491181 1150 ts_tablet_manager.cc:1397] T abe27e749fa0485195e0064a49d5431e P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.202s user 0.035s sys 0.031s
I20250901 01:52:17.492153 1150 raft_consensus.cc:357] T abe27e749fa0485195e0064a49d5431e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:17.493186 453 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet c58edaa76ebf47dfb1d9d5fab6de7dfa
I20250901 01:52:17.493353 1150 raft_consensus.cc:383] T abe27e749fa0485195e0064a49d5431e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:17.493454 1150 raft_consensus.cc:738] T abe27e749fa0485195e0064a49d5431e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:17.493996 1150 consensus_queue.cc:260] T abe27e749fa0485195e0064a49d5431e P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:17.494247 447 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6741c56499b44f648fb032c8cf50205b consensus state has no leader
I20250901 01:52:17.496843 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet e9b179dba2cc49428ba8090c962d6136
I20250901 01:52:17.499547 1150 ts_tablet_manager.cc:1428] T abe27e749fa0485195e0064a49d5431e P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250901 01:52:17.499884 1150 tablet_bootstrap.cc:492] T c9a4c80bec1d41458974a0c449aecaf7 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:17.500092 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_e21873578fc14cc3b67e08478ec42192"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:17.500671 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e21873578fc14cc3b67e08478ec42192 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:17.502333 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_3b46a0a1f77a4cea842e19b3c89de475"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:17.502859 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3b46a0a1f77a4cea842e19b3c89de475 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:17.503355 1150 tablet_bootstrap.cc:654] T c9a4c80bec1d41458974a0c449aecaf7 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:17.504101 450 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_98d60921500349bcb1f8c1e713314b07" } new_table_name: "default.Table_489b87a28dac48d093c241f301d5cccc" modify_external_catalogs: true
I20250901 01:52:17.509876 1150 tablet_bootstrap.cc:492] T c9a4c80bec1d41458974a0c449aecaf7 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:17.510466 1150 ts_tablet_manager.cc:1397] T c9a4c80bec1d41458974a0c449aecaf7 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.004s
I20250901 01:52:17.511510 1150 raft_consensus.cc:357] T c9a4c80bec1d41458974a0c449aecaf7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:17.512516 992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:40561, user_credentials={real_user=slave}} blocked reactor thread for 278133us
I20250901 01:52:17.513372 1150 raft_consensus.cc:383] T c9a4c80bec1d41458974a0c449aecaf7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:17.513652 1150 raft_consensus.cc:738] T c9a4c80bec1d41458974a0c449aecaf7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:17.514129 1150 consensus_queue.cc:260] T c9a4c80bec1d41458974a0c449aecaf7 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:17.518580 1150 ts_tablet_manager.cc:1428] T c9a4c80bec1d41458974a0c449aecaf7 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250901 01:52:17.522281 1150 tablet_bootstrap.cc:492] T 2ff914352c7c43de800f3d54ecf63072 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:17.524313 1150 tablet_bootstrap.cc:654] T 2ff914352c7c43de800f3d54ecf63072 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:17.524336 441 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 6c15cd14831f422eb615038603c79ed0 (table default.Table_5e65c5f068b84995a2d7a76fa65dda0f [id=b9d81ddd38fe47838877b010be2b40d0]) successfully deleted
W20250901 01:52:17.524683 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet d58f69fad01f403eab74a47d3695ed83 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d58f69fad01f403eab74a47d3695ed83 already in progress: creating tablet
W20250901 01:52:17.524945 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet a774cc6ae3924d299127fd9ffb01ca57 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a774cc6ae3924d299127fd9ffb01ca57 already in progress: creating tablet
I20250901 01:52:17.525792 920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "34042ea5c1b645969c1de41fe587c343" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:17.526544 920 raft_consensus.cc:2466] T 34042ea5c1b645969c1de41fe587c343 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:17.530036 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet f3962b471bdb49aea6e9fa95f1f9dfe2
I20250901 01:52:17.534647 1150 tablet_bootstrap.cc:492] T 2ff914352c7c43de800f3d54ecf63072 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:17.534924 1150 ts_tablet_manager.cc:1397] T 2ff914352c7c43de800f3d54ecf63072 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.003s
I20250901 01:52:17.535995 1150 raft_consensus.cc:357] T 2ff914352c7c43de800f3d54ecf63072 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:17.536547 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 2240302d9178445d8700e35f5ea274f7 with delete_type TABLET_DATA_DELETED (replaced by tablet b13afacd37474781877b9f2cd20b313a) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:17.537184 1150 raft_consensus.cc:383] T 2ff914352c7c43de800f3d54ecf63072 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:17.537252 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 3b18a7084727490380d26b3485e4e799 with delete_type TABLET_DATA_DELETED (replaced by tablet b164b4a67a9143ceaff69b0ce49d63fe) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:17.537452 1150 raft_consensus.cc:738] T 2ff914352c7c43de800f3d54ecf63072 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:17.537931 1150 consensus_queue.cc:260] T 2ff914352c7c43de800f3d54ecf63072 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:17.538034 1030 tablet_service.cc:1468] Processing CreateTablet for tablet c58edaa76ebf47dfb1d9d5fab6de7dfa (DEFAULT_TABLE table=default.Table_3df3e67f179f46a8be496e3bfe3cb89f [id=3fa0578ede344c0a8ff5d00b3703ad74]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:17.538012 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_7c23fc28bbae4efe982c240b87f40956"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:17.538501 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c58edaa76ebf47dfb1d9d5fab6de7dfa. 1 dirs total, 0 dirs full, 0 dirs failed
W20250901 01:52:17.539719 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7c23fc28bbae4efe982c240b87f40956 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:17.544741 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet 467c2bf4a8cb4937b6f627bea28d8ddb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:17 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:17.545037 1332 tablet_replica.cc:331] T 467c2bf4a8cb4937b6f627bea28d8ddb P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:17.545326 1332 raft_consensus.cc:2241] T 467c2bf4a8cb4937b6f627bea28d8ddb P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:17.545684 1332 raft_consensus.cc:2270] T 467c2bf4a8cb4937b6f627bea28d8ddb P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:17.546849 1332 ts_tablet_manager.cc:1905] T 467c2bf4a8cb4937b6f627bea28d8ddb P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:17.550896 1150 ts_tablet_manager.cc:1428] T 2ff914352c7c43de800f3d54ecf63072 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
I20250901 01:52:17.552309 1150 tablet_bootstrap.cc:492] T 98cca86164db4252b1b49de88747822a P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:17.553776 1332 ts_tablet_manager.cc:1918] T 467c2bf4a8cb4937b6f627bea28d8ddb P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 01:52:17.553975 1332 log.cc:1199] T 467c2bf4a8cb4937b6f627bea28d8ddb P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/467c2bf4a8cb4937b6f627bea28d8ddb
I20250901 01:52:17.554564 1332 ts_tablet_manager.cc:1939] T 467c2bf4a8cb4937b6f627bea28d8ddb P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:17.556454 1150 tablet_bootstrap.cc:654] T 98cca86164db4252b1b49de88747822a P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:17.556335 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 89d40f0331114490b51de3ba6d773571 (DEFAULT_TABLE table=default.Table_88d759cd5be3472088b92a5efe832a70 [id=8ee0aa1ba24042dd8ded004644913b9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:17.556794 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89d40f0331114490b51de3ba6d773571. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:17.561242 455 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 2ff914352c7c43de800f3d54ecf63072 (table default.Table_1186b4165684445da50e713ed6a227f9 [id=74d89178d7594613ba29d2bb01358e80]). Expected version 1 got 0
I20250901 01:52:17.564020 1023 tablet_service.cc:1468] Processing CreateTablet for tablet ea846f88b3534beebd0001b7f17b5ca9 (DEFAULT_TABLE table=default.Table_2b74c001cb4f4d82ab314557f4f09422 [id=85b64046d2fa4f1d871ab43e5c800fb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:17.564491 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea846f88b3534beebd0001b7f17b5ca9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:17.567896 1150 tablet_bootstrap.cc:492] T 98cca86164db4252b1b49de88747822a P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:17.568063 1150 ts_tablet_manager.cc:1397] T 98cca86164db4252b1b49de88747822a P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.002s
I20250901 01:52:17.568241 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_aac69ecb4bd042ad83e7ea0fc72f6ef9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:17.568854 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aac69ecb4bd042ad83e7ea0fc72f6ef9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:17.568961 1150 raft_consensus.cc:357] T 98cca86164db4252b1b49de88747822a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:17.569172 1150 raft_consensus.cc:383] T 98cca86164db4252b1b49de88747822a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:17.569244 1150 raft_consensus.cc:738] T 98cca86164db4252b1b49de88747822a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:17.569211 446 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_e098ab0a5d4045cfb18b7182ffda1b99" } new_table_name: "default.Table_95e1338a49a04f22b679e363e253262e" modify_external_catalogs: true
I20250901 01:52:17.572618 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 71941b815d6840478c5acc1cbb187cc9 (DEFAULT_TABLE table=default.Table_f027ccc695a74d1fb4595d9614f30381 [id=3ac80ef18f1044c0b22018ae09997fd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:17.572646 1038 tablet_service.cc:1468] Processing CreateTablet for tablet f13b972e25214f168421439fe3943e8a (DEFAULT_TABLE table=default.Table_d5a6ac03c9ff4969ada556d1318b64f0 [id=c1483e9d0db64d468b12344500f9cac8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:17.573104 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71941b815d6840478c5acc1cbb187cc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:17.573333 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet c58edaa76ebf47dfb1d9d5fab6de7dfa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:17 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:17.573650 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f13b972e25214f168421439fe3943e8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:17.576460 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 98cca86164db4252b1b49de88747822a
I20250901 01:52:17.576952 451 catalog_manager.cc:6652] ReplaceTablet: tablet 98cca86164db4252b1b49de88747822a deleted and replaced by tablet fa7fb88ded21432a8757f37ec3db5e71
I20250901 01:52:17.581049 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet e9b179dba2cc49428ba8090c962d6136 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:17 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:17.581017 1025 tablet_service.cc:1468] Processing CreateTablet for tablet f6a8b425968e49deaecfa3adacfd3f1f (DEFAULT_TABLE table=default.Table_c2ed44ae561c4b88b22e8975a61b77aa [id=004b578e7e5d47ed9dd12d89076b0f28]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:17.581409 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6a8b425968e49deaecfa3adacfd3f1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:17.583057 451 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 2ff914352c7c43de800f3d54ecf63072 (table default.Table_1186b4165684445da50e713ed6a227f9 [id=74d89178d7594613ba29d2bb01358e80]). Expected version 1 got 0
I20250901 01:52:17.583875 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_39ba5e484e9e4820a478f44df19e769f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:17.584450 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_39ba5e484e9e4820a478f44df19e769f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:17.584563 1150 consensus_queue.cc:260] T 98cca86164db4252b1b49de88747822a P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:17.588454 899 tablet_service.cc:1515] Processing DeleteTablet for tablet 2240302d9178445d8700e35f5ea274f7 with delete_type TABLET_DATA_DELETED (replaced by tablet b13afacd37474781877b9f2cd20b313a) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:17.589205 1020 tablet_service.cc:1468] Processing CreateTablet for tablet e87d1281f35648b0b115b67c95e244e7 (DEFAULT_TABLE table=default.Table_add9fa2d6db444b6896dd970425259ae [id=4453959e273c473fadd737f56f7db3dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:17.589718 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e87d1281f35648b0b115b67c95e244e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:17.591919 1150 ts_tablet_manager.cc:1428] T 98cca86164db4252b1b49de88747822a P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.024s user 0.003s sys 0.000s
I20250901 01:52:17.592845 1023 tablet_service.cc:1468] Processing CreateTablet for tablet aec951f2cc84499b9162b9fa24d5440e (DEFAULT_TABLE table=default.Table_8770c960d4bd4c49b043ffe889b4b87b [id=55c31cd42fac43e6b8eddc048f565d0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:17.593343 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aec951f2cc84499b9162b9fa24d5440e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:17.592643 899 tablet_service.cc:1515] Processing DeleteTablet for tablet 3b18a7084727490380d26b3485e4e799 with delete_type TABLET_DATA_DELETED (replaced by tablet b164b4a67a9143ceaff69b0ce49d63fe) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:17.594671 1306 tablet_replica.cc:331] T 3b18a7084727490380d26b3485e4e799 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:17.594859 1306 raft_consensus.cc:2241] T 3b18a7084727490380d26b3485e4e799 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:17.595046 1306 raft_consensus.cc:2270] T 3b18a7084727490380d26b3485e4e799 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:17.595072 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet eeae5294bd474f368f5766c260ccf3fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:17 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:17.595610 1306 ts_tablet_manager.cc:1905] T 3b18a7084727490380d26b3485e4e799 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:17.596208 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 500e3c33598b45a4afa983da7cb0845a (DEFAULT_TABLE table=default.Table_8ac222c93e36436c9a3ecb2c489f6d74 [id=44b75816b8b24af0a9530d4a12c2416e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:17.596705 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 500e3c33598b45a4afa983da7cb0845a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:17.593163 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 467c2bf4a8cb4937b6f627bea28d8ddb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:17 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:17.593665 902 tablet_service.cc:1515] Processing DeleteTablet for tablet eeae5294bd474f368f5766c260ccf3fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:17 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:17.594293 889 tablet_service.cc:1468] Processing CreateTablet for tablet c58edaa76ebf47dfb1d9d5fab6de7dfa (DEFAULT_TABLE table=default.Table_3df3e67f179f46a8be496e3bfe3cb89f [id=3fa0578ede344c0a8ff5d00b3703ad74]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:17.599181 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c58edaa76ebf47dfb1d9d5fab6de7dfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:17.603327 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_85b9e2f3f8f7400aa833201daae93dc3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:17.603873 1033 tablet_service.cc:1515] Processing DeleteTablet for tablet a5f4b0a2d72a4b8095ae4af1fe7c0c5d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:11 UTC) from {username='slave'} at 127.0.0.1:44246
W20250901 01:52:17.603976 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_85b9e2f3f8f7400aa833201daae93dc3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:17.604251 1033 tablet_service.cc:1515] Processing DeleteTablet for tablet 22691943a134400c8df4617240e5d3ef with delete_type TABLET_DATA_DELETED (replaced by tablet 438cd35ff3f74c49b908d072560077db) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:17.604619 1033 tablet_service.cc:1515] Processing DeleteTablet for tablet 1f18420261ae41459e1f25b927bd84a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:13 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:17.604943 1033 tablet_service.cc:1515] Processing DeleteTablet for tablet ceb724d859d24d569598e18389dae3b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:14 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:17.605500 1033 tablet_service.cc:1468] Processing CreateTablet for tablet f3962b471bdb49aea6e9fa95f1f9dfe2 (DEFAULT_TABLE table=default.Table_e2ea7aa90231448888c6cfead7e3d9c1 [id=c264ed0bf1d84bc9bd5c2bb0d1e8a45b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:17.605901 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3962b471bdb49aea6e9fa95f1f9dfe2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:17.607227 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 5e2472ee5d6e4d3bbf1f24027ba531ca (DEFAULT_TABLE table=default.Table_52733c9a363b4ed9b6ff11ad8571e421 [id=453153553bea48319e132857f628d48a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:17.607695 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e2472ee5d6e4d3bbf1f24027ba531ca. 1 dirs total, 0 dirs full, 0 dirs failed
W20250901 01:52:17.610773 452 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ed1b41340ce742a88c3e4fd85cd25ca6 consensus state has no leader
I20250901 01:52:17.612651 1021 tablet_service.cc:1468] Processing CreateTablet for tablet f9b293bf6a17453b9ea9892a6970d0b0 (DEFAULT_TABLE table=default.Table_8d0db1143f1547c2a7d41775b4d9ef09 [id=b67bfe518a094e0781ef931524b55990]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:17.613090 1552 raft_consensus.cc:491] T a5f4b0a2d72a4b8095ae4af1fe7c0c5d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:17.613272 1552 raft_consensus.cc:513] T a5f4b0a2d72a4b8095ae4af1fe7c0c5d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:17.614089 1552 leader_election.cc:290] T a5f4b0a2d72a4b8095ae4af1fe7c0c5d P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:17.614423 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9b293bf6a17453b9ea9892a6970d0b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:17.616554 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet f3962b471bdb49aea6e9fa95f1f9dfe2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:17 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:17.616519 452 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_97f8d5c204174c918a795498e47d404d" } modify_external_catalogs: true
I20250901 01:52:17.616735 452 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_97f8d5c204174c918a795498e47d404d" } modify_external_catalogs: true
I20250901 01:52:17.617076 1038 tablet_service.cc:1468] Processing CreateTablet for tablet 3d90c6ef06d1436d8e2bb1333e7d9985 (DEFAULT_TABLE table=default.Table_5facf725e0fc44cda7d77f8511030fa5 [id=892696e8040d4f2db8620d1d4ac90e0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:17.617483 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d90c6ef06d1436d8e2bb1333e7d9985. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:17.610428 1306 ts_tablet_manager.cc:1918] T 3b18a7084727490380d26b3485e4e799 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:17.618686 1306 log.cc:1199] T 3b18a7084727490380d26b3485e4e799 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/3b18a7084727490380d26b3485e4e799
I20250901 01:52:17.619149 1306 ts_tablet_manager.cc:1939] T 3b18a7084727490380d26b3485e4e799 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:17.619807 1306 tablet_replica.cc:331] T 467c2bf4a8cb4937b6f627bea28d8ddb P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:17.620476 1150 tablet_bootstrap.cc:492] T f50f33de1b654663b9ef8a91e3436f11 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:17.624507 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 7a607fac57994001b2304e5d81c9decc (DEFAULT_TABLE table=default.Table_048a628c8a35498f94012e430b3c29bc [id=5014fdf96714467f98fd130c3b66f37a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:17.621222 889 tablet_service.cc:1468] Processing CreateTablet for tablet 89d40f0331114490b51de3ba6d773571 (DEFAULT_TABLE table=default.Table_88d759cd5be3472088b92a5efe832a70 [id=8ee0aa1ba24042dd8ded004644913b9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:17.622181 902 tablet_service.cc:1468] Processing CreateTablet for tablet ea846f88b3534beebd0001b7f17b5ca9 (DEFAULT_TABLE table=default.Table_2b74c001cb4f4d82ab314557f4f09422 [id=85b64046d2fa4f1d871ab43e5c800fb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:17.622815 890 tablet_service.cc:1468] Processing CreateTablet for tablet aec951f2cc84499b9162b9fa24d5440e (DEFAULT_TABLE table=default.Table_8770c960d4bd4c49b043ffe889b4b87b [id=55c31cd42fac43e6b8eddc048f565d0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:17.624912 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a607fac57994001b2304e5d81c9decc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:17.625097 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea846f88b3534beebd0001b7f17b5ca9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:17.626653 454 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_18b93109e1224ea59b5c93825b3b27d6" } modify_external_catalogs: true
I20250901 01:52:17.626858 454 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_18b93109e1224ea59b5c93825b3b27d6" } modify_external_catalogs: true
I20250901 01:52:17.628913 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aec951f2cc84499b9162b9fa24d5440e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:17.638255 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89d40f0331114490b51de3ba6d773571. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:17.642699 890 tablet_service.cc:1468] Processing CreateTablet for tablet e87d1281f35648b0b115b67c95e244e7 (DEFAULT_TABLE table=default.Table_add9fa2d6db444b6896dd970425259ae [id=4453959e273c473fadd737f56f7db3dd]), partition=RANGE (key) PARTITION UNBOUNDED
W20250901 01:52:17.651841 450 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a54af99bc10045e8b0b6868e5aed98b4 consensus state has no leader
I20250901 01:52:17.657193 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e87d1281f35648b0b115b67c95e244e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:17.659586 448 master_service.cc:948] ReplaceTablet: received request to replace tablet 467c2bf4a8cb4937b6f627bea28d8ddb from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:17.660880 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_8b1086c9a4c44239851c586ea7267e3c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:17.661511 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8b1086c9a4c44239851c586ea7267e3c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:17.667393 455 master_service.cc:948] ReplaceTablet: received request to replace tablet c49d1143d54b4c319739488906b0e271 from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:17.669154 448 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_3ffe49903e864e8e8cc92d82f4c14a3a" } new_table_name: "default.Table_eecfab7239cc40c5a2f1fb6ac6d35dc9" modify_external_catalogs: true
I20250901 01:52:17.650698 1682 tablet.cc:1722] T 467c2bf4a8cb4937b6f627bea28d8ddb P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250901 01:52:17.673122 446 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 49aaaec611784e6bac131efd58dbca51 consensus state has no leader
I20250901 01:52:17.674278 1306 raft_consensus.cc:2241] T 467c2bf4a8cb4937b6f627bea28d8ddb P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:17.674779 1306 raft_consensus.cc:2270] T 467c2bf4a8cb4937b6f627bea28d8ddb P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:17.675673 1306 ts_tablet_manager.cc:1905] T 467c2bf4a8cb4937b6f627bea28d8ddb P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:17.678890 1552 raft_consensus.cc:491] T 4410da9e95454ecf95faf80221b0a71c P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:17.680691 1552 raft_consensus.cc:513] T 4410da9e95454ecf95faf80221b0a71c P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:17.681341 1552 leader_election.cc:290] T 4410da9e95454ecf95faf80221b0a71c P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:17.681757 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 7954393c8c1c4457ab8bb723c03a9c8b (DEFAULT_TABLE table=default.Table_853b0fae8015415b91325d10a633ccf6 [id=543f2074598d4c29972e564be1123fca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:17.683492 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7954393c8c1c4457ab8bb723c03a9c8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:17.685324 1151 tablet_bootstrap.cc:492] T 3b18a7084727490380d26b3485e4e799 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:17.685494 1151 ts_tablet_manager.cc:1397] T 3b18a7084727490380d26b3485e4e799 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.488s user 0.022s sys 0.030s
I20250901 01:52:17.686376 1151 raft_consensus.cc:357] T 3b18a7084727490380d26b3485e4e799 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:17.686586 1151 raft_consensus.cc:383] T 3b18a7084727490380d26b3485e4e799 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:17.686686 1151 raft_consensus.cc:738] T 3b18a7084727490380d26b3485e4e799 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:17.686915 1151 consensus_queue.cc:260] T 3b18a7084727490380d26b3485e4e799 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:17.687541 1151 ts_tablet_manager.cc:1428] T 3b18a7084727490380d26b3485e4e799 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250901 01:52:17.687767 1332 tablet_replica.cc:331] T eeae5294bd474f368f5766c260ccf3fe P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:17.687948 1332 raft_consensus.cc:2241] T eeae5294bd474f368f5766c260ccf3fe P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:17.688148 1332 raft_consensus.cc:2270] T eeae5294bd474f368f5766c260ccf3fe P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:17.688855 1332 ts_tablet_manager.cc:1905] T eeae5294bd474f368f5766c260ccf3fe P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:17.691824 902 tablet_service.cc:1468] Processing CreateTablet for tablet 71941b815d6840478c5acc1cbb187cc9 (DEFAULT_TABLE table=default.Table_f027ccc695a74d1fb4595d9614f30381 [id=3ac80ef18f1044c0b22018ae09997fd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:17.692607 899 tablet_service.cc:1515] Processing DeleteTablet for tablet 533d14f5903e49599a34ea353ffd8c00 with delete_type TABLET_DATA_DELETED (replaced by tablet a54af99bc10045e8b0b6868e5aed98b4) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:17.692847 451 master_service.cc:948] ReplaceTablet: received request to replace tablet e249474dcbf642189560c91ec3a99fdf from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:17.693135 899 tablet_service.cc:1515] Processing DeleteTablet for tablet 82d163681fda47f38d79863bd28f576d with delete_type TABLET_DATA_DELETED (replaced by tablet bf8f0237233a49219cd1c41cf808bdba) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:17.693650 899 tablet_service.cc:1515] Processing DeleteTablet for tablet 5a8c21aee9f84216b6f90538a5bf4c17 with delete_type TABLET_DATA_DELETED (replaced by tablet 42c1156e8dd84c6fa1396b46de13eb6f) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:17.694371 899 tablet_service.cc:1515] Processing DeleteTablet for tablet c58edaa76ebf47dfb1d9d5fab6de7dfa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:17 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:17.697592 1658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3add7edb97a40ef9f7c6bc011d13227. Latest consensus state: current_term: 2 leader_uuid: "e3add7edb97a40ef9f7c6bc011d13227" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3add7edb97a40ef9f7c6bc011d13227" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 43871 } } peers { permanent_uuid: "d2ae2cb6af0f4b3a8f47ab1cdd6567d3" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42375 } } peers { permanent_uuid: "7ce2288f784044eea010664e3def133f" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 44231 } } }
I20250901 01:52:17.697830 1658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f [sys.catalog]: This master's current role is: FOLLOWER
I20250901 01:52:17.699767 1103 heartbeater.cc:344] Connected to a master server at 127.0.104.60:44231
I20250901 01:52:17.700196 1503 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54796 (request call id 0) took 2486 ms. Trace:
I20250901 01:52:17.709183 1150 tablet_bootstrap.cc:654] T f50f33de1b654663b9ef8a91e3436f11 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:17.710615 899 tablet_service.cc:1515] Processing DeleteTablet for tablet e9b179dba2cc49428ba8090c962d6136 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:17 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:17.711298 899 tablet_service.cc:1468] Processing CreateTablet for tablet f13b972e25214f168421439fe3943e8a (DEFAULT_TABLE table=default.Table_d5a6ac03c9ff4969ada556d1318b64f0 [id=c1483e9d0db64d468b12344500f9cac8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:17.712100 904 tablet_service.cc:1468] Processing CreateTablet for tablet f6a8b425968e49deaecfa3adacfd3f1f (DEFAULT_TABLE table=default.Table_c2ed44ae561c4b88b22e8975a61b77aa [id=004b578e7e5d47ed9dd12d89076b0f28]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:17.713019 898 tablet_service.cc:1515] Processing DeleteTablet for tablet f3962b471bdb49aea6e9fa95f1f9dfe2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:17 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:17.713521 898 tablet_service.cc:1515] Processing DeleteTablet for tablet abe27e749fa0485195e0064a49d5431e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:15 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:17.714138 898 tablet_service.cc:1468] Processing CreateTablet for tablet 7a607fac57994001b2304e5d81c9decc (DEFAULT_TABLE table=default.Table_048a628c8a35498f94012e430b3c29bc [id=5014fdf96714467f98fd130c3b66f37a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:17.715036 897 tablet_service.cc:1468] Processing CreateTablet for tablet 3d90c6ef06d1436d8e2bb1333e7d9985 (DEFAULT_TABLE table=default.Table_5facf725e0fc44cda7d77f8511030fa5 [id=892696e8040d4f2db8620d1d4ac90e0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:17.716949 446 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_ad2bde750a6f4c9aad14a28a9a0848c9" } modify_external_catalogs: true
I20250901 01:52:17.717021 900 tablet_service.cc:1468] Processing CreateTablet for tablet 5e2472ee5d6e4d3bbf1f24027ba531ca (DEFAULT_TABLE table=default.Table_52733c9a363b4ed9b6ff11ad8571e421 [id=453153553bea48319e132857f628d48a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:17.717154 446 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_ad2bde750a6f4c9aad14a28a9a0848c9" } modify_external_catalogs: true
I20250901 01:52:17.717944 891 tablet_service.cc:1515] Processing DeleteTablet for tablet c9a4c80bec1d41458974a0c449aecaf7 with delete_type TABLET_DATA_DELETED (replaced by tablet 121803e79ab64ec0b948b448646b9d0a) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:17.718551 891 tablet_service.cc:1468] Processing CreateTablet for tablet f9b293bf6a17453b9ea9892a6970d0b0 (DEFAULT_TABLE table=default.Table_8d0db1143f1547c2a7d41775b4d9ef09 [id=b67bfe518a094e0781ef931524b55990]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:17.718638 903 tablet_service.cc:1468] Processing CreateTablet for tablet 500e3c33598b45a4afa983da7cb0845a (DEFAULT_TABLE table=default.Table_8ac222c93e36436c9a3ecb2c489f6d74 [id=44b75816b8b24af0a9530d4a12c2416e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:17.719156 453 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_ae4267b6728a4e60a333464621e3dc8f" } modify_external_catalogs: true
I20250901 01:52:17.719306 453 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_ae4267b6728a4e60a333464621e3dc8f" } modify_external_catalogs: true
I20250901 01:52:17.719357 896 tablet_service.cc:1468] Processing CreateTablet for tablet f3962b471bdb49aea6e9fa95f1f9dfe2 (DEFAULT_TABLE table=default.Table_e2ea7aa90231448888c6cfead7e3d9c1 [id=c264ed0bf1d84bc9bd5c2bb0d1e8a45b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:17.719767 447 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_14bc71ec066c420b8f3f1e5a5305a0ae" } new_table_name: "default.Table_eb9eaa97a468420bb672b720834c6ddd" modify_external_catalogs: true
I20250901 01:52:17.720196 894 tablet_service.cc:1468] Processing CreateTablet for tablet 7954393c8c1c4457ab8bb723c03a9c8b (DEFAULT_TABLE table=default.Table_853b0fae8015415b91325d10a633ccf6 [id=543f2074598d4c29972e564be1123fca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:17.720916 452 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 0dbdcb76eeba4fefa56e7835c23c0316
I20250901 01:52:17.721314 1151 tablet_bootstrap.cc:492] T 58c0e7a2e90e48528bc2bc7ec889f608 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:17.722363 1658 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e3add7edb97a40ef9f7c6bc011d13227)
I20250901 01:52:17.723382 1658 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3add7edb97a40ef9f7c6bc011d13227" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 43871 } } peers { permanent_uuid: "d2ae2cb6af0f4b3a8f47ab1cdd6567d3" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42375 } } peers { permanent_uuid: "7ce2288f784044eea010664e3def133f" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 44231 } }
I20250901 01:52:17.723575 1023 tablet_service.cc:1468] Processing CreateTablet for tablet afc002ae2486476b9208e1cf8215157c (DEFAULT_TABLE table=default.Table_ad2bde750a6f4c9aad14a28a9a0848c9 [id=da00d16114194d41826ef0da1f9423a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:17.726104 1658 leader_election.cc:290] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e3add7edb97a40ef9f7c6bc011d13227 (127.0.104.62:43871), d2ae2cb6af0f4b3a8f47ab1cdd6567d3 (127.0.104.61:42375)
I20250901 01:52:17.726415 1038 tablet_service.cc:1468] Processing CreateTablet for tablet 33f637f6a8134215bb1995ec38440073 (DEFAULT_TABLE table=default.Table_983ec443070b4ffc8508ea825e30471a [id=557afe29976d40f6bfb6ae4ca348f887]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:17.726635 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 20f7bfab18134e079e91c52c89468cc3
I20250901 01:52:17.726593 1033 tablet_service.cc:1468] Processing CreateTablet for tablet 482a9a2cb81f448bb2823892e35920d4 (DEFAULT_TABLE table=default.Table_c298ecfa989347b0b7a4d7227611539f [id=b94e5ce799ee4046982550aace3a08d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:17.726775 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afc002ae2486476b9208e1cf8215157c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:17.738101 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33f637f6a8134215bb1995ec38440073. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:17.741015 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 482a9a2cb81f448bb2823892e35920d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:17.744304 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 98cca86164db4252b1b49de88747822a with delete_type TABLET_DATA_DELETED (replaced by tablet fa7fb88ded21432a8757f37ec3db5e71) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:17.749194 454 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_39ba5e484e9e4820a478f44df19e769f" } modify_external_catalogs: true
I20250901 01:52:17.749442 454 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_39ba5e484e9e4820a478f44df19e769f" } modify_external_catalogs: true
I20250901 01:52:17.752031 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_8ca4c54e4c424b888922086913acf096"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:17.752684 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8ca4c54e4c424b888922086913acf096 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:17.754026 1024 tablet_service.cc:1468] Processing CreateTablet for tablet ed1b41340ce742a88c3e4fd85cd25ca6 (DEFAULT_TABLE table=default.Table_bda1777b1e9c409ea39c00de92706b50 [id=0baa4993b72e4a8d94ee3059c9d2db6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:17.782323 1696 catalog_manager.cc:797] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250901 01:52:17.782939 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 6789505328b1409ea288e8b4eaf94935 (DEFAULT_TABLE table=default.Table_657603f2bb3c43bc85028218860da978 [id=013cf6104d1a47e592f90f6ba89653aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:17.783339 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6789505328b1409ea288e8b4eaf94935. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:17.784364 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 615b2306fca84003b264da9d5f95ddab (DEFAULT_TABLE table=default.Table_0f959237963c4e33932e193285c0eacb [id=cecdd8d6ca6d445c9fd82decd2a95366]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:17.784749 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 615b2306fca84003b264da9d5f95ddab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:17.786373 1037 tablet_service.cc:1468] Processing CreateTablet for tablet 05630a72ddae4de0b890dece2feacc31 (DEFAULT_TABLE table=default.Table_ee242c34410e4c6887405d04a4b26124 [id=c9a2f96383664a4e8524c286cd9e4149]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:17.786733 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05630a72ddae4de0b890dece2feacc31. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:17.788339 1020 tablet_service.cc:1468] Processing CreateTablet for tablet ad1a6be9fcc045d3bb4e71a2f6bd5ccd (DEFAULT_TABLE table=default.Table_5e84ae9a9c2a4084a1f9c5ffb4594de2 [id=9bebb2e486ab48888e6b523281ac1e88]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:17.788738 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad1a6be9fcc045d3bb4e71a2f6bd5ccd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:17.792721 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 14b466b2a5a249aa841f1b8b06301edc (DEFAULT_TABLE table=default.Table_ae42ba019ee14c69b1a16d1177f3642c [id=78cd71321e9a480ca2d7d88327f157ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:17.793114 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14b466b2a5a249aa841f1b8b06301edc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:17.796537 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 56475ba491a24301a534304811f30140 (DEFAULT_TABLE table=default.Table_b0e916c76a7a4766aa0887cc8691488f [id=cf61137a852c48748a29f833f1860da7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:17.797274 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f13b972e25214f168421439fe3943e8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:17.797514 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7954393c8c1c4457ab8bb723c03a9c8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:17.800427 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_8f7db2ae75a24dfab71c2a95c6a0a404"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:17.801023 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8f7db2ae75a24dfab71c2a95c6a0a404 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:17.802430 889 tablet_service.cc:1468] Processing CreateTablet for tablet afc002ae2486476b9208e1cf8215157c (DEFAULT_TABLE table=default.Table_ad2bde750a6f4c9aad14a28a9a0848c9 [id=da00d16114194d41826ef0da1f9423a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:17.805922 1026 tablet_service.cc:1468] Processing CreateTablet for tablet ff013ea889d44cba8797834e4611b489 (DEFAULT_TABLE table=default.Table_0f91070385df441b96af4d5406a94724 [id=4747cdd47a954e0ab19609c9526b688d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:17.806401 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56475ba491a24301a534304811f30140. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:17.806932 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e2472ee5d6e4d3bbf1f24027ba531ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:17.807190 904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6a8b425968e49deaecfa3adacfd3f1f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250901 01:52:17.808068 448 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e6c10426f60f43e8bde69d9b1b9c8eb6 consensus state has no leader
I20250901 01:52:17.808336 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9b293bf6a17453b9ea9892a6970d0b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:17.808450 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet c49d1143d54b4c319739488906b0e271
I20250901 01:52:17.808564 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71941b815d6840478c5acc1cbb187cc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:17.808935 455 catalog_manager.cc:6652] ReplaceTablet: tablet c49d1143d54b4c319739488906b0e271 deleted and replaced by tablet 69f61ec283484f9d8b3eebcea27f741b
I20250901 01:52:17.809186 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d90c6ef06d1436d8e2bb1333e7d9985. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:17.823655 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet e249474dcbf642189560c91ec3a99fdf
I20250901 01:52:17.823781 446 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet afc002ae2486476b9208e1cf8215157c
I20250901 01:52:17.824201 451 catalog_manager.cc:6652] ReplaceTablet: tablet e249474dcbf642189560c91ec3a99fdf deleted and replaced by tablet 7f46e9ddadaf4ff986d8c991178daa0c
I20250901 01:52:17.823555 1151 tablet_bootstrap.cc:654] T 58c0e7a2e90e48528bc2bc7ec889f608 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:17.825404 1293 raft_consensus.cc:491] T 606fac5c6efe48d1877104b01c23ba5b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250901 01:52:17.826104 447 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6741c56499b44f648fb032c8cf50205b consensus state has no leader
I20250901 01:52:17.826601 453 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 9473f0ef978c4bfaa41fb7b0729e8863
I20250901 01:52:17.827728 1293 raft_consensus.cc:513] T 606fac5c6efe48d1877104b01c23ba5b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:17.828397 1293 leader_election.cc:290] T 606fac5c6efe48d1877104b01c23ba5b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:17.828899 454 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 76f81d5b7c5e4e2689ecd97314f99fd9
I20250901 01:52:17.831864 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_4185b4f73a2743e08f24a26878cf73ca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:17.832553 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4185b4f73a2743e08f24a26878cf73ca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250901 01:52:17.834827 858 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111) [suppressed 72 similar messages]
I20250901 01:52:17.835866 903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 500e3c33598b45a4afa983da7cb0845a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:17.836494 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3962b471bdb49aea6e9fa95f1f9dfe2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:17.839219 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a607fac57994001b2304e5d81c9decc. 1 dirs total, 0 dirs full, 0 dirs failed
W20250901 01:52:17.841195 858 leader_election.cc:336] T 606fac5c6efe48d1877104b01c23ba5b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:17.844897 455 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_88d759cd5be3472088b92a5efe832a70" } new_table_name: "default.Table_4dacb3b1d2e94e42ac8937eb6eb58375" modify_external_catalogs: true
I20250901 01:52:17.846432 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_4401d83f75f8464da49d5b811d7dd89d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:17.847034 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4401d83f75f8464da49d5b811d7dd89d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:17.850142 1293 raft_consensus.cc:491] T c9a4c80bec1d41458974a0c449aecaf7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:17.850302 1293 raft_consensus.cc:513] T c9a4c80bec1d41458974a0c449aecaf7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:17.852111 1293 leader_election.cc:290] T c9a4c80bec1d41458974a0c449aecaf7 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:17.855228 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_364cf1adeb2c4102aad338f2e228fb0b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:17.855870 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_364cf1adeb2c4102aad338f2e228fb0b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:17.856556 453 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_48252c70dcfd4a9e8dbd3b8efba88da6" } new_table_name: "default.Table_8a88621295cf403da5cf0fc50afbb345" modify_external_catalogs: true
I20250901 01:52:17.858810 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_5a32379f33204b9e9b2dfe82995134dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:17.859277 858 leader_election.cc:336] T c9a4c80bec1d41458974a0c449aecaf7 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:17.859310 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5a32379f33204b9e9b2dfe82995134dc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:17.858498 534 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7ce2288f784044eea010664e3def133f" candidate_term: 3 candidate_status { last_received { term: 2 index: 324 } } ignore_live_leader: false dest_uuid: "d2ae2cb6af0f4b3a8f47ab1cdd6567d3" is_pre_election: true
I20250901 01:52:17.700577 1503 rpcz_store.cc:276] 0901 01:52:15.213456 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:15.213530 (+ 74us) service_pool.cc:225] Handling call
0901 01:52:17.185060 (+1971530us) raft_consensus.cc:1469] Updating replica for 279 ops
0901 01:52:17.221574 (+ 36514us) raft_consensus.cc:1532] Early marking committed up to index 45
0901 01:52:17.228001 (+ 6427us) raft_consensus.cc:1537] Triggering prepare for 279 ops
0901 01:52:17.228372 (+ 371us) write_op.cc:270] Start()
0901 01:52:17.228445 (+ 73us) write_op.cc:276] Timestamp: P: 1756691528456888 usec, L: 0
0901 01:52:17.231721 (+ 3276us) write_op.cc:270] Start()
0901 01:52:17.231773 (+ 52us) write_op.cc:276] Timestamp: P: 1756691528459375 usec, L: 0
0901 01:52:17.262308 (+ 30535us) write_op.cc:270] Start()
0901 01:52:17.262337 (+ 29us) write_op.cc:276] Timestamp: P: 1756691528551575 usec, L: 0
0901 01:52:17.263830 (+ 1493us) write_op.cc:270] Start()
0901 01:52:17.263855 (+ 25us) write_op.cc:276] Timestamp: P: 1756691528552590 usec, L: 0
0901 01:52:17.264325 (+ 470us) write_op.cc:270] Start()
0901 01:52:17.264347 (+ 22us) write_op.cc:276] Timestamp: P: 1756691528553155 usec, L: 0
0901 01:52:17.269218 (+ 4871us) write_op.cc:270] Start()
0901 01:52:17.269266 (+ 48us) write_op.cc:276] Timestamp: P: 1756691528553741 usec, L: 0
0901 01:52:17.269816 (+ 550us) write_op.cc:270] Start()
0901 01:52:17.269856 (+ 40us) write_op.cc:276] Timestamp: P: 1756691528554284 usec, L: 0
0901 01:52:17.270393 (+ 537us) write_op.cc:270] Start()
0901 01:52:17.270427 (+ 34us) write_op.cc:276] Timestamp: P: 1756691528554828 usec, L: 0
0901 01:52:17.270873 (+ 446us) write_op.cc:270] Start()
0901 01:52:17.270894 (+ 21us) write_op.cc:276] Timestamp: P: 1756691528555286 usec, L: 0
0901 01:52:17.271363 (+ 469us) write_op.cc:270] Start()
0901 01:52:17.271382 (+ 19us) write_op.cc:276] Timestamp: P: 1756691528574238 usec, L: 0
0901 01:52:17.271883 (+ 501us) write_op.cc:270] Start()
0901 01:52:17.271903 (+ 20us) write_op.cc:276] Timestamp: P: 1756691528575117 usec, L: 0
0901 01:52:17.272290 (+ 387us) write_op.cc:270] Start()
0901 01:52:17.272335 (+ 45us) write_op.cc:276] Timestamp: P: 1756691528582034 usec, L: 0
0901 01:52:17.272820 (+ 485us) write_op.cc:270] Start()
0901 01:52:17.272841 (+ 21us) write_op.cc:276] Timestamp: P: 1756691528613161 usec, L: 0
0901 01:52:17.286243 (+ 13402us) write_op.cc:270] Start()
0901 01:52:17.286352 (+ 109us) write_op.cc:276] Timestamp: P: 1756691528617646 usec, L: 0
0901 01:52:17.293800 (+ 7448us) write_op.cc:270] Start()
0901 01:52:17.293840 (+ 40us) write_op.cc:276] Timestamp: P: 1756691528618629 usec, L: 0
0901 01:52:17.294017 (+ 177us) write_op.cc:270] Start()
0901 01:52:17.294050 (+ 33us) write_op.cc:276] Timestamp: P: 1756691528620336 usec, L: 0
0901 01:52:17.294245 (+ 195us) write_op.cc:270] Start()
0901 01:52:17.294259 (+ 14us) write_op.cc:276] Timestamp: P: 1756691528677978 usec, L: 0
0901 01:52:17.294392 (+ 133us) write_op.cc:270] Start()
0901 01:52:17.294413 (+ 21us) write_op.cc:276] Timestamp: P: 1756691528687829 usec, L: 0
0901 01:52:17.294542 (+ 129us) write_op.cc:270] Start()
0901 01:52:17.294563 (+ 21us) write_op.cc:276] Timestamp: P: 1756691528691527 usec, L: 0
0901 01:52:17.294734 (+ 171us) write_op.cc:270] Start()
0901 01:52:17.294755 (+ 21us) write_op.cc:276] Timestamp: P: 1756691528705278 usec, L: 0
0901 01:52:17.294935 (+ 180us) write_op.cc:270] Start()
0901 01:52:17.294957 (+ 22us) write_op.cc:276] Timestamp: P: 1756691528706932 usec, L: 0
0901 01:52:17.295125 (+ 168us) write_op.cc:270] Start()
0901 01:52:17.295148 (+ 23us) write_op.cc:276] Timestamp: P: 1756691528730101 usec, L: 0
0901 01:52:17.297301 (+ 2153us) write_op.cc:270] Start()
0901 01:52:17.297340 (+ 39us) write_op.cc:276] Timestamp: P: 1756691528760687 usec, L: 0
0901 01:52:17.297519 (+ 179us) write_op.cc:270] Start()
0901 01:52:17.297543 (+ 24us) write_op.cc:276] Timestamp: P: 1756691528764613 usec, L: 0
0901 01:52:17.297742 (+ 199us) write_op.cc:270] Start()
0901 01:52:17.297768 (+ 26us) write_op.cc:276] Timestamp: P: 1756691528767221 usec, L: 0
0901 01:52:17.297958 (+ 190us) write_op.cc:270] Start()
0901 01:52:17.297981 (+ 23us) write_op.cc:276] Timestamp: P: 1756691528776343 usec, L: 0
0901 01:52:17.298115 (+ 134us) write_op.cc:270] Start()
0901 01:52:17.298136 (+ 21us) write_op.cc:276] Timestamp: P: 1756691528781196 usec, L: 0
0901 01:52:17.298276 (+ 140us) write_op.cc:270] Start()
0901 01:52:17.298297 (+ 21us) write_op.cc:276] Timestamp: P: 1756691528783700 usec, L: 0
0901 01:52:17.298451 (+ 154us) write_op.cc:270] Start()
0901 01:52:17.298475 (+ 24us) write_op.cc:276] Timestamp: P: 1756691528795831 usec, L: 0
0901 01:52:17.298634 (+ 159us) write_op.cc:270] Start()
0901 01:52:17.298672 (+ 38us) write_op.cc:276] Timestamp: P: 1756691528796485 usec, L: 0
0901 01:52:17.298832 (+ 160us) write_op.cc:270] Start()
0901 01:52:17.298854 (+ 22us) write_op.cc:276] Timestamp: P: 1756691528796995 usec, L: 0
0901 01:52:17.298988 (+ 134us) write_op.cc:270] Start()
0901 01:52:17.299010 (+ 22us) write_op.cc:276] Timestamp: P: 1756691528811802 usec, L: 0
0901 01:52:17.299162 (+ 152us) write_op.cc:270] Start()
0901 01:52:17.299184 (+ 22us) write_op.cc:276] Timestamp: P: 1756691528812443 usec, L: 0
0901 01:52:17.299346 (+ 162us) write_op.cc:270] Start()
0901 01:52:17.299368 (+ 22us) write_op.cc:276] Timestamp: P: 1756691528827288 usec, L: 0
0901 01:52:17.299529 (+ 161us) write_op.cc:270] Start()
0901 01:52:17.299552 (+ 23us) write_op.cc:276] Timestamp: P: 1756691528827893 usec, L: 0
0901 01:52:17.299716 (+ 164us) write_op.cc:270] Start()
0901 01:52:17.299771 (+ 55us) write_op.cc:276] Timestamp: P: 1756691528831407 usec, L: 0
0901 01:52:17.299932 (+ 161us) write_op.cc:270] Start()
0901 01:52:17.299955 (+ 23us) write_op.cc:276] Timestamp: P: 1756691528838774 usec, L: 0
0901 01:52:17.300090 (+ 135us) write_op.cc:270] Start()
0901 01:52:17.300112 (+ 22us) write_op.cc:276] Timestamp: P: 1756691528847286 usec, L: 0
0901 01:52:17.300250 (+ 138us) write_op.cc:270] Start()
0901 01:52:17.300272 (+ 22us) write_op.cc:276] Timestamp: P: 1756691528851071 usec, L: 0
0901 01:52:17.300450 (+ 178us) write_op.cc:270] Start()
0901 01:52:17.300475 (+ 25us) write_op.cc:276] Timestamp: P: 1756691528918989 usec, L: 0
0901 01:52:17.300632 (+ 157us) write_op.cc:270] Start()
0901 01:52:17.300654 (+ 22us) write_op.cc:276] Timestamp: P: 1756691528959561 usec, L: 0
0901 01:52:17.300863 (+ 209us) write_op.cc:270] Start()
0901 01:52:17.300886 (+ 23us) write_op.cc:276] Timestamp: P: 1756691528972385 usec, L: 0
0901 01:52:17.301037 (+ 151us) write_op.cc:270] Start()
0901 01:52:17.301060 (+ 23us) write_op.cc:276] Timestamp: P: 1756691528980362 usec, L: 0
0901 01:52:17.301212 (+ 152us) write_op.cc:270] Start()
0901 01:52:17.301234 (+ 22us) write_op.cc:276] Timestamp: P: 1756691528982875 usec, L: 0
0901 01:52:17.301380 (+ 146us) write_op.cc:270] Start()
0901 01:52:17.301402 (+ 22us) write_op.cc:276] Timestamp: P: 1756691529018704 usec, L: 0
0901 01:52:17.301555 (+ 153us) write_op.cc:270] Start()
0901 01:52:17.301577 (+ 22us) write_op.cc:276] Timestamp: P: 1756691529082757 usec, L: 0
0901 01:52:17.301725 (+ 148us) write_op.cc:270] Start()
0901 01:52:17.301748 (+ 23us) write_op.cc:276] Timestamp: P: 1756691529178072 usec, L: 0
0901 01:52:17.301889 (+ 141us) write_op.cc:270] Start()
0901 01:52:17.301911 (+ 22us) write_op.cc:276] Timestamp: P: 1756691529254537 usec, L: 0
0901 01:52:17.302052 (+ 141us) write_op.cc:270] Start()
0901 01:52:17.302068 (+ 16us) write_op.cc:276] Timestamp: P: 1756691529268184 usec, L: 0
0901 01:52:17.302270 (+ 202us) write_op.cc:270] Start()
0901 01:52:17.302295 (+ 25us) write_op.cc:276] Timestamp: P: 1756691529269385 usec, L: 0
0901 01:52:17.302469 (+ 174us) write_op.cc:270] Start()
0901 01:52:17.302493 (+ 24us) write_op.cc:276] Timestamp: P: 1756691529270365 usec, L: 0
0901 01:52:17.302631 (+ 138us) write_op.cc:270] Start()
0901 01:52:17.302654 (+ 23us) write_op.cc:276] Timestamp: P: 1756691529271447 usec, L: 0
0901 01:52:17.302805 (+ 151us) write_op.cc:270] Start()
0901 01:52:17.302827 (+ 22us) write_op.cc:276] Timestamp: P: 1756691529272508 usec, L: 0
0901 01:52:17.302985 (+ 158us) write_op.cc:270] Start()
0901 01:52:17.303007 (+ 22us) write_op.cc:276] Timestamp: P: 1756691529385035 usec, L: 0
0901 01:52:17.303160 (+ 153us) write_op.cc:270] Start()
0901 01:52:17.303182 (+ 22us) write_op.cc:276] Timestamp: P: 1756691529387097 usec, L: 0
0901 01:52:17.303319 (+ 137us) write_op.cc:270] Start()
0901 01:52:17.303340 (+ 21us) write_op.cc:276] Timestamp: P: 1756691529408999 usec, L: 0
0901 01:52:17.303474 (+ 134us) write_op.cc:270] Start()
0901 01:52:17.303497 (+ 23us) write_op.cc:276] Timestamp: P: 1756691529429375 usec, L: 0
0901 01:52:17.303635 (+ 138us) write_op.cc:270] Start()
0901 01:52:17.303656 (+ 21us) write_op.cc:276] Timestamp: P: 1756691529453259 usec, L: 0
0901 01:52:17.303805 (+ 149us) write_op.cc:270] Start()
0901 01:52:17.303828 (+ 23us) write_op.cc:276] Timestamp: P: 1756691529474144 usec, L: 0
0901 01:52:17.303983 (+ 155us) write_op.cc:270] Start()
0901 01:52:17.304046 (+ 63us) write_op.cc:276] Timestamp: P: 1756691529484522 usec, L: 0
0901 01:52:17.304190 (+ 144us) write_op.cc:270] Start()
0901 01:52:17.304211 (+ 21us) write_op.cc:276] Timestamp: P: 1756691529503815 usec, L: 0
0901 01:52:17.304350 (+ 139us) write_op.cc:270] Start()
0901 01:52:17.304371 (+ 21us) write_op.cc:276] Timestamp: P: 1756691529515699 usec, L: 0
0901 01:52:17.336828 (+ 32457us) write_op.cc:270] Start()
0901 01:52:17.336899 (+ 71us) write_op.cc:276] Timestamp: P: 1756691529530387 usec, L: 0
0901 01:52:17.337099 (+ 200us) write_op.cc:270] Start()
0901 01:52:17.337123 (+ 24us) write_op.cc:276] Timestamp: P: 1756691529540261 usec, L: 0
0901 01:52:17.337296 (+ 173us) write_op.cc:270] Start()
0901 01:52:17.337319 (+ 23us) write_op.cc:276] Timestamp: P: 1756691529561142 usec, L: 0
0901 01:52:17.337478 (+ 159us) write_op.cc:270] Start()
0901 01:52:17.337503 (+ 25us) write_op.cc:276] Timestamp: P: 1756691529641299 usec, L: 0
0901 01:52:17.339580 (+ 2077us) write_op.cc:270] Start()
0901 01:52:17.339626 (+ 46us) write_op.cc:276] Timestamp: P: 1756691529675216 usec, L: 0
0901 01:52:17.339854 (+ 228us) write_op.cc:270] Start()
0901 01:52:17.339880 (+ 26us) write_op.cc:276] Timestamp: P: 1756691529690543 usec, L: 0
0901 01:52:17.340036 (+ 156us) write_op.cc:270] Start()
0901 01:52:17.340060 (+ 24us) write_op.cc:276] Timestamp: P: 1756691529714485 usec, L: 0
0901 01:52:17.340211 (+ 151us) write_op.cc:270] Start()
0901 01:52:17.340235 (+ 24us) write_op.cc:276] Timestamp: P: 1756691529736925 usec, L: 0
0901 01:52:17.340377 (+ 142us) write_op.cc:270] Start()
0901 01:52:17.341529 (+ 1152us) write_op.cc:276] Timestamp: P: 1756691529769010 usec, L: 0
0901 01:52:17.341748 (+ 219us) write_op.cc:270] Start()
0901 01:52:17.341785 (+ 37us) write_op.cc:276] Timestamp: P: 1756691529786331 usec, L: 0
0901 01:52:17.341950 (+ 165us) write_op.cc:270] Start()
0901 01:52:17.341974 (+ 24us) write_op.cc:276] Timestamp: P: 1756691529831594 usec, L: 0
0901 01:52:17.342111 (+ 137us) write_op.cc:270] Start()
0901 01:52:17.343066 (+ 955us) write_op.cc:276] Timestamp: P: 1756691529851193 usec, L: 0
0901 01:52:17.343285 (+ 219us) write_op.cc:270] Start()
0901 01:52:17.343311 (+ 26us) write_op.cc:276] Timestamp: P: 1756691529862766 usec, L: 0
0901 01:52:17.343478 (+ 167us) write_op.cc:270] Start()
0901 01:52:17.343501 (+ 23us) write_op.cc:276] Timestamp: P: 1756691529900719 usec, L: 0
0901 01:52:17.343673 (+ 172us) write_op.cc:270] Start()
0901 01:52:17.343696 (+ 23us) write_op.cc:276] Timestamp: P: 1756691529924928 usec, L: 0
0901 01:52:17.343844 (+ 148us) write_op.cc:270] Start()
0901 01:52:17.343902 (+ 58us) write_op.cc:276] Timestamp: P: 1756691529948261 usec, L: 0
0901 01:52:17.344072 (+ 170us) write_op.cc:270] Start()
0901 01:52:17.344094 (+ 22us) write_op.cc:276] Timestamp: P: 1756691529989437 usec, L: 0
0901 01:52:17.344236 (+ 142us) write_op.cc:270] Start()
0901 01:52:17.344258 (+ 22us) write_op.cc:276] Timestamp: P: 1756691530012878 usec, L: 0
0901 01:52:17.345367 (+ 1109us) write_op.cc:270] Start()
0901 01:52:17.345400 (+ 33us) write_op.cc:276] Timestamp: P: 1756691530037122 usec, L: 0
0901 01:52:17.345717 (+ 317us) write_op.cc:270] Start()
0901 01:52:17.345744 (+ 27us) write_op.cc:276] Timestamp: P: 1756691530053639 usec, L: 0
0901 01:52:17.345910 (+ 166us) write_op.cc:270] Start()
0901 01:52:17.345933 (+ 23us) write_op.cc:276] Timestamp: P: 1756691530056505 usec, L: 0
0901 01:52:17.346437 (+ 504us) write_op.cc:270] Start()
0901 01:52:17.346492 (+ 55us) write_op.cc:276] Timestamp: P: 1756691530057511 usec, L: 0
0901 01:52:17.346672 (+ 180us) write_op.cc:270] Start()
0901 01:52:17.346697 (+ 25us) write_op.cc:276] Timestamp: P: 1756691530100431 usec, L: 0
0901 01:52:17.346857 (+ 160us) write_op.cc:270] Start()
0901 01:52:17.346880 (+ 23us) write_op.cc:276] Timestamp: P: 1756691530114245 usec, L: 0
0901 01:52:17.347013 (+ 133us) write_op.cc:270] Start()
0901 01:52:17.347035 (+ 22us) write_op.cc:276] Timestamp: P: 1756691530134688 usec, L: 0
0901 01:52:17.347175 (+ 140us) write_op.cc:270] Start()
0901 01:52:17.347197 (+ 22us) write_op.cc:276] Timestamp: P: 1756691530296679 usec, L: 0
0901 01:52:17.347331 (+ 134us) write_op.cc:270] Start()
0901 01:52:17.347353 (+ 22us) write_op.cc:276] Timestamp: P: 1756691530342056 usec, L: 0
0901 01:52:17.347508 (+ 155us) write_op.cc:270] Start()
0901 01:52:17.347531 (+ 23us) write_op.cc:276] Timestamp: P: 1756691530379177 usec, L: 0
0901 01:52:17.347673 (+ 142us) write_op.cc:270] Start()
0901 01:52:17.347702 (+ 29us) write_op.cc:276] Timestamp: P: 1756691530430940 usec, L: 0
0901 01:52:17.347846 (+ 144us) write_op.cc:270] Start()
0901 01:52:17.347868 (+ 22us) write_op.cc:276] Timestamp: P: 1756691530496956 usec, L: 0
0901 01:52:17.348028 (+ 160us) write_op.cc:270] Start()
0901 01:52:17.348049 (+ 21us) write_op.cc:276] Timestamp: P: 1756691530518931 usec, L: 0
0901 01:52:17.350669 (+ 2620us) write_op.cc:270] Start()
0901 01:52:17.350696 (+ 27us) write_op.cc:276] Timestamp: P: 1756691530524939 usec, L: 0
0901 01:52:17.350863 (+ 167us) write_op.cc:270] Start()
0901 01:52:17.350889 (+ 26us) write_op.cc:276] Timestamp: P: 1756691530527031 usec, L: 0
0901 01:52:17.351067 (+ 178us) write_op.cc:270] Start()
0901 01:52:17.351091 (+ 24us) write_op.cc:276] Timestamp: P: 1756691530528701 usec, L: 0
0901 01:52:17.351259 (+ 168us) write_op.cc:270] Start()
0901 01:52:17.351282 (+ 23us) write_op.cc:276] Timestamp: P: 1756691530530246 usec, L: 0
0901 01:52:17.351444 (+ 162us) write_op.cc:270] Start()
0901 01:52:17.351477 (+ 33us) write_op.cc:276] Timestamp: P: 1756691531449135 usec, L: 0
0901 01:52:17.351625 (+ 148us) write_op.cc:270] Start()
0901 01:52:17.351648 (+ 23us) write_op.cc:276] Timestamp: P: 1756691531470900 usec, L: 0
0901 01:52:17.351791 (+ 143us) write_op.cc:270] Start()
0901 01:52:17.351814 (+ 23us) write_op.cc:276] Timestamp: P: 1756691531494815 usec, L: 0
0901 01:52:17.351954 (+ 140us) write_op.cc:270] Start()
0901 01:52:17.351978 (+ 24us) write_op.cc:276] Timestamp: P: 1756691531509850 usec, L: 0
0901 01:52:17.352154 (+ 176us) write_op.cc:270] Start()
0901 01:52:17.352176 (+ 22us) write_op.cc:276] Timestamp: P: 1756691531521371 usec, L: 0
0901 01:52:17.352312 (+ 136us) write_op.cc:270] Start()
0901 01:52:17.352345 (+ 33us) write_op.cc:276] Timestamp: P: 1756691531523465 usec, L: 0
0901 01:52:17.367775 (+ 15430us) write_op.cc:270] Start()
0901 01:52:17.367824 (+ 49us) write_op.cc:276] Timestamp: P: 1756691531545445 usec, L: 0
0901 01:52:17.368034 (+ 210us) write_op.cc:270] Start()
0901 01:52:17.368059 (+ 25us) write_op.cc:276] Timestamp: P: 1756691531551781 usec, L: 0
0901 01:52:17.368201 (+ 142us) write_op.cc:270] Start()
0901 01:52:17.368223 (+ 22us) write_op.cc:276] Timestamp: P: 1756691532017441 usec, L: 0
0901 01:52:17.368385 (+ 162us) write_op.cc:270] Start()
0901 01:52:17.370636 (+ 2251us) write_op.cc:276] Timestamp: P: 1756691532033795 usec, L: 0
0901 01:52:17.370921 (+ 285us) write_op.cc:270] Start()
0901 01:52:17.370955 (+ 34us) write_op.cc:276] Timestamp: P: 1756691532037710 usec, L: 0
0901 01:52:17.371107 (+ 152us) write_op.cc:270] Start()
0901 01:52:17.371131 (+ 24us) write_op.cc:276] Timestamp: P: 1756691532060073 usec, L: 0
0901 01:52:17.371274 (+ 143us) write_op.cc:270] Start()
0901 01:52:17.371296 (+ 22us) write_op.cc:276] Timestamp: P: 1756691532066951 usec, L: 0
0901 01:52:17.371441 (+ 145us) write_op.cc:270] Start()
0901 01:52:17.371467 (+ 26us) write_op.cc:276] Timestamp: P: 1756691532067797 usec, L: 0
0901 01:52:17.371629 (+ 162us) write_op.cc:270] Start()
0901 01:52:17.371652 (+ 23us) write_op.cc:276] Timestamp: P: 1756691532068533 usec, L: 0
0901 01:52:17.371793 (+ 141us) write_op.cc:270] Start()
0901 01:52:17.371815 (+ 22us) write_op.cc:276] Timestamp: P: 1756691532069199 usec, L: 0
0901 01:52:17.371945 (+ 130us) write_op.cc:270] Start()
0901 01:52:17.371968 (+ 23us) write_op.cc:276] Timestamp: P: 1756691532156228 usec, L: 0
0901 01:52:17.372106 (+ 138us) write_op.cc:270] Start()
0901 01:52:17.372135 (+ 29us) write_op.cc:276] Timestamp: P: 1756691532282378 usec, L: 0
0901 01:52:17.372278 (+ 143us) write_op.cc:270] Start()
0901 01:52:17.372299 (+ 21us) write_op.cc:276] Timestamp: P: 1756691532392425 usec, L: 0
0901 01:52:17.372484 (+ 185us) write_op.cc:270] Start()
0901 01:52:17.372509 (+ 25us) write_op.cc:276] Timestamp: P: 1756691532393203 usec, L: 0
0901 01:52:17.372694 (+ 185us) write_op.cc:270] Start()
0901 01:52:17.372717 (+ 23us) write_op.cc:276] Timestamp: P: 1756691532583460 usec, L: 0
0901 01:52:17.372876 (+ 159us) write_op.cc:270] Start()
0901 01:52:17.372901 (+ 25us) write_op.cc:276] Timestamp: P: 1756691532585673 usec, L: 0
0901 01:52:17.376493 (+ 3592us) write_op.cc:270] Start()
0901 01:52:17.376544 (+ 51us) write_op.cc:276] Timestamp: P: 1756691532586773 usec, L: 0
0901 01:52:17.376837 (+ 293us) write_op.cc:270] Start()
0901 01:52:17.376865 (+ 28us) write_op.cc:276] Timestamp: P: 1756691532593295 usec, L: 0
0901 01:52:17.377073 (+ 208us) write_op.cc:270] Start()
0901 01:52:17.377097 (+ 24us) write_op.cc:276] Timestamp: P: 1756691532595367 usec, L: 0
0901 01:52:17.377275 (+ 178us) write_op.cc:270] Start()
0901 01:52:17.377298 (+ 23us) write_op.cc:276] Timestamp: P: 1756691532604532 usec, L: 0
0901 01:52:17.377464 (+ 166us) write_op.cc:270] Start()
0901 01:52:17.377488 (+ 24us) write_op.cc:276] Timestamp: P: 1756691532605243 usec, L: 0
0901 01:52:17.377656 (+ 168us) write_op.cc:270] Start()
0901 01:52:17.377680 (+ 24us) write_op.cc:276] Timestamp: P: 1756691532605918 usec, L: 0
0901 01:52:17.377831 (+ 151us) write_op.cc:270] Start()
0901 01:52:17.377853 (+ 22us) write_op.cc:276] Timestamp: P: 1756691532606463 usec, L: 0
0901 01:52:17.377995 (+ 142us) write_op.cc:270] Start()
0901 01:52:17.378017 (+ 22us) write_op.cc:276] Timestamp: P: 1756691532616471 usec, L: 0
0901 01:52:17.378222 (+ 205us) write_op.cc:270] Start()
0901 01:52:17.378248 (+ 26us) write_op.cc:276] Timestamp: P: 1756691532649252 usec, L: 0
0901 01:52:17.378461 (+ 213us) write_op.cc:270] Start()
0901 01:52:17.378486 (+ 25us) write_op.cc:276] Timestamp: P: 1756691532650432 usec, L: 0
0901 01:52:17.378654 (+ 168us) write_op.cc:270] Start()
0901 01:52:17.378678 (+ 24us) write_op.cc:276] Timestamp: P: 1756691532652691 usec, L: 0
0901 01:52:17.378827 (+ 149us) write_op.cc:270] Start()
0901 01:52:17.378907 (+ 80us) write_op.cc:276] Timestamp: P: 1756691532668255 usec, L: 0
0901 01:52:17.379060 (+ 153us) write_op.cc:270] Start()
0901 01:52:17.379082 (+ 22us) write_op.cc:276] Timestamp: P: 1756691532673453 usec, L: 0
0901 01:52:17.379218 (+ 136us) write_op.cc:270] Start()
0901 01:52:17.379239 (+ 21us) write_op.cc:276] Timestamp: P: 1756691532676301 usec, L: 0
0901 01:52:17.379413 (+ 174us) write_op.cc:270] Start()
0901 01:52:17.379437 (+ 24us) write_op.cc:276] Timestamp: P: 1756691532683105 usec, L: 0
0901 01:52:17.379606 (+ 169us) write_op.cc:270] Start()
0901 01:52:17.379641 (+ 35us) write_op.cc:276] Timestamp: P: 1756691532788985 usec, L: 0
0901 01:52:17.379788 (+ 147us) write_op.cc:270] Start()
0901 01:52:17.379811 (+ 23us) write_op.cc:276] Timestamp: P: 1756691532789632 usec, L: 0
0901 01:52:17.379951 (+ 140us) write_op.cc:270] Start()
0901 01:52:17.379975 (+ 24us) write_op.cc:276] Timestamp: P: 1756691532800904 usec, L: 0
0901 01:52:17.380114 (+ 139us) write_op.cc:270] Start()
0901 01:52:17.380136 (+ 22us) write_op.cc:276] Timestamp: P: 1756691532801538 usec, L: 0
0901 01:52:17.380285 (+ 149us) write_op.cc:270] Start()
0901 01:52:17.380311 (+ 26us) write_op.cc:276] Timestamp: P: 1756691532802123 usec, L: 0
0901 01:52:17.380561 (+ 250us) write_op.cc:270] Start()
0901 01:52:17.380587 (+ 26us) write_op.cc:276] Timestamp: P: 1756691532812363 usec, L: 0
0901 01:52:17.380762 (+ 175us) write_op.cc:270] Start()
0901 01:52:17.380781 (+ 19us) write_op.cc:276] Timestamp: P: 1756691532838388 usec, L: 0
0901 01:52:17.380888 (+ 107us) write_op.cc:270] Start()
0901 01:52:17.380910 (+ 22us) write_op.cc:276] Timestamp: P: 1756691532839093 usec, L: 0
0901 01:52:17.381044 (+ 134us) write_op.cc:270] Start()
0901 01:52:17.381066 (+ 22us) write_op.cc:276] Timestamp: P: 1756691532857185 usec, L: 0
0901 01:52:17.381202 (+ 136us) write_op.cc:270] Start()
0901 01:52:17.381224 (+ 22us) write_op.cc:276] Timestamp: P: 1756691532858420 usec, L: 0
0901 01:52:17.381405 (+ 181us) write_op.cc:270] Start()
0901 01:52:17.381429 (+ 24us) write_op.cc:276] Timestamp: P: 1756691532880093 usec, L: 0
0901 01:52:17.381569 (+ 140us) write_op.cc:270] Start()
0901 01:52:17.381591 (+ 22us) write_op.cc:276] Timestamp: P: 1756691532880708 usec, L: 0
0901 01:52:17.381766 (+ 175us) write_op.cc:270] Start()
0901 01:52:17.381788 (+ 22us) write_op.cc:276] Timestamp: P: 1756691532881248 usec, L: 0
0901 01:52:17.381957 (+ 169us) write_op.cc:270] Start()
0901 01:52:17.381980 (+ 23us) write_op.cc:276] Timestamp: P: 1756691532893597 usec, L: 0
0901 01:52:17.382150 (+ 170us) write_op.cc:270] Start()
0901 01:52:17.382173 (+ 23us) write_op.cc:276] Timestamp: P: 1756691532894268 usec, L: 0
0901 01:52:17.382317 (+ 144us) write_op.cc:270] Start()
0901 01:52:17.382339 (+ 22us) write_op.cc:276] Timestamp: P: 1756691532894799 usec, L: 0
0901 01:52:17.382474 (+ 135us) write_op.cc:270] Start()
0901 01:52:17.382498 (+ 24us) write_op.cc:276] Timestamp: P: 1756691532895351 usec, L: 0
0901 01:52:17.382660 (+ 162us) write_op.cc:270] Start()
0901 01:52:17.382684 (+ 24us) write_op.cc:276] Timestamp: P: 1756691532913788 usec, L: 0
0901 01:52:17.382846 (+ 162us) write_op.cc:270] Start()
0901 01:52:17.382869 (+ 23us) write_op.cc:276] Timestamp: P: 1756691532925267 usec, L: 0
0901 01:52:17.383010 (+ 141us) write_op.cc:270] Start()
0901 01:52:17.383033 (+ 23us) write_op.cc:276] Timestamp: P: 1756691532990838 usec, L: 0
0901 01:52:17.383193 (+ 160us) write_op.cc:270] Start()
0901 01:52:17.383215 (+ 22us) write_op.cc:276] Timestamp: P: 1756691533004419 usec, L: 0
0901 01:52:17.383378 (+ 163us) write_op.cc:270] Start()
0901 01:52:17.383397 (+ 19us) write_op.cc:276] Timestamp: P: 1756691533005095 usec, L: 0
0901 01:52:17.383546 (+ 149us) write_op.cc:270] Start()
0901 01:52:17.383570 (+ 24us) write_op.cc:276] Timestamp: P: 1756691533014747 usec, L: 0
0901 01:52:17.383723 (+ 153us) write_op.cc:270] Start()
0901 01:52:17.383744 (+ 21us) write_op.cc:276] Timestamp: P: 1756691533026912 usec, L: 0
0901 01:52:17.383885 (+ 141us) write_op.cc:270] Start()
0901 01:52:17.383907 (+ 22us) write_op.cc:276] Timestamp: P: 1756691533034677 usec, L: 0
0901 01:52:17.384044 (+ 137us) write_op.cc:270] Start()
0901 01:52:17.384066 (+ 22us) write_op.cc:276] Timestamp: P: 1756691533035317 usec, L: 0
0901 01:52:17.384199 (+ 133us) write_op.cc:270] Start()
0901 01:52:17.384221 (+ 22us) write_op.cc:276] Timestamp: P: 1756691533035987 usec, L: 0
0901 01:52:17.384371 (+ 150us) write_op.cc:270] Start()
0901 01:52:17.384391 (+ 20us) write_op.cc:276] Timestamp: P: 1756691533036897 usec, L: 0
0901 01:52:17.401147 (+ 16756us) write_op.cc:270] Start()
0901 01:52:17.401187 (+ 40us) write_op.cc:276] Timestamp: P: 1756691533037655 usec, L: 0
0901 01:52:17.401333 (+ 146us) write_op.cc:270] Start()
0901 01:52:17.401352 (+ 19us) write_op.cc:276] Timestamp: P: 1756691533048306 usec, L: 0
0901 01:52:17.401487 (+ 135us) write_op.cc:270] Start()
0901 01:52:17.401506 (+ 19us) write_op.cc:276] Timestamp: P: 1756691533049652 usec, L: 0
0901 01:52:17.401616 (+ 110us) write_op.cc:270] Start()
0901 01:52:17.401718 (+ 102us) write_op.cc:276] Timestamp: P: 1756691533063404 usec, L: 0
0901 01:52:17.405840 (+ 4122us) write_op.cc:270] Start()
0901 01:52:17.406180 (+ 340us) write_op.cc:276] Timestamp: P: 1756691533094845 usec, L: 0
0901 01:52:17.407642 (+ 1462us) write_op.cc:270] Start()
0901 01:52:17.407696 (+ 54us) write_op.cc:276] Timestamp: P: 1756691533104965 usec, L: 0
0901 01:52:17.410137 (+ 2441us) write_op.cc:270] Start()
0901 01:52:17.410171 (+ 34us) write_op.cc:276] Timestamp: P: 1756691533134267 usec, L: 0
0901 01:52:17.410355 (+ 184us) write_op.cc:270] Start()
0901 01:52:17.410374 (+ 19us) write_op.cc:276] Timestamp: P: 1756691533136626 usec, L: 0
0901 01:52:17.411084 (+ 710us) write_op.cc:270] Start()
0901 01:52:17.411111 (+ 27us) write_op.cc:276] Timestamp: P: 1756691533184655 usec, L: 0
0901 01:52:17.411271 (+ 160us) write_op.cc:270] Start()
0901 01:52:17.411292 (+ 21us) write_op.cc:276] Timestamp: P: 1756691533186073 usec, L: 0
0901 01:52:17.411410 (+ 118us) write_op.cc:270] Start()
0901 01:52:17.411427 (+ 17us) write_op.cc:276] Timestamp: P: 1756691533198912 usec, L: 0
0901 01:52:17.411571 (+ 144us) write_op.cc:270] Start()
0901 01:52:17.411588 (+ 17us) write_op.cc:276] Timestamp: P: 1756691533210872 usec, L: 0
0901 01:52:17.411703 (+ 115us) write_op.cc:270] Start()
0901 01:52:17.411719 (+ 16us) write_op.cc:276] Timestamp: P: 1756691533249981 usec, L: 0
0901 01:52:17.411825 (+ 106us) write_op.cc:270] Start()
0901 01:52:17.411849 (+ 24us) write_op.cc:276] Timestamp: P: 1756691533250874 usec, L: 0
0901 01:52:17.414443 (+ 2594us) write_op.cc:270] Start()
0901 01:52:17.414483 (+ 40us) write_op.cc:276] Timestamp: P: 1756691533251491 usec, L: 0
0901 01:52:17.414690 (+ 207us) write_op.cc:270] Start()
0901 01:52:17.414714 (+ 24us) write_op.cc:276] Timestamp: P: 1756691533252801 usec, L: 0
0901 01:52:17.414904 (+ 190us) write_op.cc:270] Start()
0901 01:52:17.414928 (+ 24us) write_op.cc:276] Timestamp: P: 1756691533268158 usec, L: 0
0901 01:52:17.415081 (+ 153us) write_op.cc:270] Start()
0901 01:52:17.415104 (+ 23us) write_op.cc:276] Timestamp: P: 1756691533272222 usec, L: 0
0901 01:52:17.415263 (+ 159us) write_op.cc:270] Start()
0901 01:52:17.415285 (+ 22us) write_op.cc:276] Timestamp: P: 1756691533273551 usec, L: 0
0901 01:52:17.415423 (+ 138us) write_op.cc:270] Start()
0901 01:52:17.415446 (+ 23us) write_op.cc:276] Timestamp: P: 1756691533279856 usec, L: 0
0901 01:52:17.415594 (+ 148us) write_op.cc:270] Start()
0901 01:52:17.415616 (+ 22us) write_op.cc:276] Timestamp: P: 1756691533297984 usec, L: 0
0901 01:52:17.415765 (+ 149us) write_op.cc:270] Start()
0901 01:52:17.415787 (+ 22us) write_op.cc:276] Timestamp: P: 1756691533307054 usec, L: 0
0901 01:52:17.415921 (+ 134us) write_op.cc:270] Start()
0901 01:52:17.415937 (+ 16us) write_op.cc:276] Timestamp: P: 1756691533318158 usec, L: 0
0901 01:52:17.416080 (+ 143us) write_op.cc:270] Start()
0901 01:52:17.416103 (+ 23us) write_op.cc:276] Timestamp: P: 1756691533321354 usec, L: 0
0901 01:52:17.416271 (+ 168us) write_op.cc:270] Start()
0901 01:52:17.416295 (+ 24us) write_op.cc:276] Timestamp: P: 1756691533322063 usec, L: 0
0901 01:52:17.422602 (+ 6307us) write_op.cc:270] Start()
0901 01:52:17.422660 (+ 58us) write_op.cc:276] Timestamp: P: 1756691533322707 usec, L: 0
0901 01:52:17.422864 (+ 204us) write_op.cc:270] Start()
0901 01:52:17.422888 (+ 24us) write_op.cc:276] Timestamp: P: 1756691533332604 usec, L: 0
0901 01:52:17.423033 (+ 145us) write_op.cc:270] Start()
0901 01:52:17.423074 (+ 41us) write_op.cc:276] Timestamp: P: 1756691533333294 usec, L: 0
0901 01:52:17.423260 (+ 186us) write_op.cc:270] Start()
0901 01:52:17.423283 (+ 23us) write_op.cc:276] Timestamp: P: 1756691533333948 usec, L: 0
0901 01:52:17.423424 (+ 141us) write_op.cc:270] Start()
0901 01:52:17.423447 (+ 23us) write_op.cc:276] Timestamp: P: 1756691533372499 usec, L: 0
0901 01:52:17.423600 (+ 153us) write_op.cc:270] Start()
0901 01:52:17.423630 (+ 30us) write_op.cc:276] Timestamp: P: 1756691533386928 usec, L: 0
0901 01:52:17.423765 (+ 135us) write_op.cc:270] Start()
0901 01:52:17.423788 (+ 23us) write_op.cc:276] Timestamp: P: 1756691533391863 usec, L: 0
0901 01:52:17.423934 (+ 146us) write_op.cc:270] Start()
0901 01:52:17.423956 (+ 22us) write_op.cc:276] Timestamp: P: 1756691533402132 usec, L: 0
0901 01:52:17.424096 (+ 140us) write_op.cc:270] Start()
0901 01:52:17.424119 (+ 23us) write_op.cc:276] Timestamp: P: 1756691533409998 usec, L: 0
0901 01:52:17.424248 (+ 129us) write_op.cc:270] Start()
0901 01:52:17.424270 (+ 22us) write_op.cc:276] Timestamp: P: 175669
I20250901 01:52:17.863494 1150 tablet_bootstrap.cc:492] T f50f33de1b654663b9ef8a91e3436f11 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:17.863703 1150 ts_tablet_manager.cc:1397] T f50f33de1b654663b9ef8a91e3436f11 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.243s user 0.004s sys 0.000s
I20250901 01:52:17.865597 465 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7ce2288f784044eea010664e3def133f" candidate_term: 3 candidate_status { last_received { term: 2 index: 324 } } ignore_live_leader: false dest_uuid: "e3add7edb97a40ef9f7c6bc011d13227" is_pre_election: true
I20250901 01:52:17.866307 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afc002ae2486476b9208e1cf8215157c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:17.867079 1472 leader_election.cc:304] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ce2288f784044eea010664e3def133f; no voters: d2ae2cb6af0f4b3a8f47ab1cdd6567d3, e3add7edb97a40ef9f7c6bc011d13227
I20250901 01:52:17.868583 1658 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250901 01:52:17.869220 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_3fc2615b8f444281a2c3d5a748d58499"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:17.869729 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3fc2615b8f444281a2c3d5a748d58499 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:17.874949 447 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_947ec0e1a97a40b59be1e234bcaf4e0d" } new_table_name: "default.Table_589c302e02474fc3bc5d3a290a8bd54a" modify_external_catalogs: true
W20250901 01:52:17.877154 455 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 89d40f0331114490b51de3ba6d773571 consensus state has no leader
I20250901 01:52:17.884907 455 master_service.cc:948] ReplaceTablet: received request to replace tablet d77aadd7291647a0bae7c421f244c252 from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:17.889699 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff013ea889d44cba8797834e4611b489. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:17.893563 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_123e5dc85bca4350b541ce0f0e9e7d3e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:17.894228 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_123e5dc85bca4350b541ce0f0e9e7d3e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:17.895710 1648 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "e3add7edb97a40ef9f7c6bc011d13227" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3add7edb97a40ef9f7c6bc011d13227" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 43871 } } peers { permanent_uuid: "d2ae2cb6af0f4b3a8f47ab1cdd6567d3" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42375 } } peers { permanent_uuid: "7ce2288f784044eea010664e3def133f" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 44231 } } }
I20250901 01:52:17.895915 1648 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227 [sys.catalog]: This master's current role is: LEADER
I20250901 01:52:17.902175 1035 tablet_service.cc:1468] Processing CreateTablet for tablet d10c193452954c198bba686169b8025b (DEFAULT_TABLE table=default.Table_2872910642aa48eca0350b2792008200 [id=6c0576410d7340d6aa493a747e95565b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:17.904031 451 master_service.cc:948] ReplaceTablet: received request to replace tablet 9d886fa63f8f4996b087997b542e8bb1 from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:17.906665 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 12c04e0c1f804179ae23509cdb24d395 (table default.Table_3d223b93128649e49a295d940feef90a [id=348190236c694eab9a318040bb19c1f2]) successfully deleted
I20250901 01:52:17.916301 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 4a87e68850c4459eb2c8eafac89c6aa9 (table default.Table_0f959237963c4e33932e193285c0eacb [id=cecdd8d6ca6d445c9fd82decd2a95366]) successfully deleted
I20250901 01:52:17.916503 451 master_service.cc:948] ReplaceTablet: received request to replace tablet 8381305e4c4148808803e255b8a43a90 from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:17.918010 1033 tablet_service.cc:1468] Processing CreateTablet for tablet 1a60c10e42104502ac2d3b80b8126ceb (DEFAULT_TABLE table=default.Table_53e30205c5fd488eb3b48c685a159d04 [id=60837b865ee7415ea42a85a953d837a5]), partition=RANGE (key) PARTITION UNBOUNDED
W20250901 01:52:17.918143 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 1286c0ab85704837a99b6e5a62184d77 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1286c0ab85704837a99b6e5a62184d77 already in progress: creating tablet
I20250901 01:52:17.918401 899 tablet_service.cc:1515] Processing DeleteTablet for tablet 98cca86164db4252b1b49de88747822a with delete_type TABLET_DATA_DELETED (replaced by tablet fa7fb88ded21432a8757f37ec3db5e71) from {username='slave'} at 127.0.0.1:52846
W20250901 01:52:17.918454 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 25ce0fe8c9544f828694184d19d13df9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 25ce0fe8c9544f828694184d19d13df9 already in progress: creating tablet
W20250901 01:52:17.918684 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 0cb3262973934c8689b851adad60afe0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0cb3262973934c8689b851adad60afe0 already in progress: creating tablet
W20250901 01:52:17.918869 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 579b9cdf819f4609a532e96bec093fd5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 579b9cdf819f4609a532e96bec093fd5 already in progress: creating tablet
I20250901 01:52:17.920576 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet afc002ae2486476b9208e1cf8215157c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:17 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:17.918998 1036 tablet_service.cc:1468] Processing CreateTablet for tablet 0dbdcb76eeba4fefa56e7835c23c0316 (DEFAULT_TABLE table=default.Table_97f8d5c204174c918a795498e47d404d [id=d74c256799d64024b869f8076b857f63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:17.922025 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed1b41340ce742a88c3e4fd85cd25ca6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:17.922574 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet e249474dcbf642189560c91ec3a99fdf with delete_type TABLET_DATA_DELETED (replaced by tablet 7f46e9ddadaf4ff986d8c991178daa0c) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:17.922748 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 20f7bfab18134e079e91c52c89468cc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:17 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:17.922853 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet c49d1143d54b4c319739488906b0e271 with delete_type TABLET_DATA_DELETED (replaced by tablet 69f61ec283484f9d8b3eebcea27f741b) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:17.924536 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_27b1a96341864be385ff5523f4f7cf2b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:17.926560 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_27b1a96341864be385ff5523f4f7cf2b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:17.926751 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 0dbdcb76eeba4fefa56e7835c23c0316 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:17 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:17.926935 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet 9473f0ef978c4bfaa41fb7b0729e8863 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:17 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:17.924913 450 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_de8ae42a0de043129ceab6caec04d192" } modify_external_catalogs: true
I20250901 01:52:17.927951 450 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_de8ae42a0de043129ceab6caec04d192" } modify_external_catalogs: true
I20250901 01:52:17.928372 1150 raft_consensus.cc:357] T f50f33de1b654663b9ef8a91e3436f11 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:17.929615 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a60c10e42104502ac2d3b80b8126ceb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:17.932610 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0dbdcb76eeba4fefa56e7835c23c0316. 1 dirs total, 0 dirs full, 0 dirs failed
W20250901 01:52:17.933770 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 4855448edb4b40c184838aa7f81c112b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4855448edb4b40c184838aa7f81c112b already in progress: creating tablet
I20250901 01:52:17.933904 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "82d163681fda47f38d79863bd28f576d" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:17.934320 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "606fac5c6efe48d1877104b01c23ba5b" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:17.934567 1058 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "533d14f5903e49599a34ea353ffd8c00" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:17.934159 903 tablet_service.cc:1468] Processing CreateTablet for tablet 33f637f6a8134215bb1995ec38440073 (DEFAULT_TABLE table=default.Table_983ec443070b4ffc8508ea825e30471a [id=557afe29976d40f6bfb6ae4ca348f887]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:17.935523 1150 raft_consensus.cc:383] T f50f33de1b654663b9ef8a91e3436f11 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:17.935933 903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33f637f6a8134215bb1995ec38440073. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:17.936703 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d10c193452954c198bba686169b8025b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250901 01:52:17.934100 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 831e42de95854541a49ae6b3e039d3e9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 831e42de95854541a49ae6b3e039d3e9 already in progress: creating tablet
I20250901 01:52:17.938865 1150 raft_consensus.cc:738] T f50f33de1b654663b9ef8a91e3436f11 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
W20250901 01:52:17.938335 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 8dd587304e6342edbe96c9897b40e5c9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8dd587304e6342edbe96c9897b40e5c9 already in progress: creating tablet
I20250901 01:52:17.939126 1150 consensus_queue.cc:260] T f50f33de1b654663b9ef8a91e3436f11 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
W20250901 01:52:17.942263 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet b7b802f976f3471faaaecc964ec87e48 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b7b802f976f3471faaaecc964ec87e48 already in progress: creating tablet
I20250901 01:52:17.942623 1150 ts_tablet_manager.cc:1428] T f50f33de1b654663b9ef8a91e3436f11 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.077s user 0.002s sys 0.000s
I20250901 01:52:17.944830 891 tablet_service.cc:1468] Processing CreateTablet for tablet 482a9a2cb81f448bb2823892e35920d4 (DEFAULT_TABLE table=default.Table_c298ecfa989347b0b7a4d7227611539f [id=b94e5ce799ee4046982550aace3a08d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:17.946858 452 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_048a628c8a35498f94012e430b3c29bc" } modify_external_catalogs: true
I20250901 01:52:17.947080 452 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_048a628c8a35498f94012e430b3c29bc" } modify_external_catalogs: true
I20250901 01:52:17.948166 1150 tablet_bootstrap.cc:492] T e6c10426f60f43e8bde69d9b1b9c8eb6 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:17.965483 1150 tablet_bootstrap.cc:654] T e6c10426f60f43e8bde69d9b1b9c8eb6 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:17.960340 1306 ts_tablet_manager.cc:1918] T 467c2bf4a8cb4937b6f627bea28d8ddb P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 01:52:17.966773 1306 log.cc:1199] T 467c2bf4a8cb4937b6f627bea28d8ddb P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/467c2bf4a8cb4937b6f627bea28d8ddb
I20250901 01:52:17.967361 1306 ts_tablet_manager.cc:1939] T 467c2bf4a8cb4937b6f627bea28d8ddb P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:17.973239 1058 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c9a4c80bec1d41458974a0c449aecaf7" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:17.973773 453 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6c0abbfc8d904dce989ce51961e4237d has no consensus state
I20250901 01:52:17.975103 1293 raft_consensus.cc:491] T b009cc95c819431ea9827a3e453b8da3 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:17.975327 1293 raft_consensus.cc:513] T b009cc95c819431ea9827a3e453b8da3 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:17.976037 1293 leader_election.cc:290] T b009cc95c819431ea9827a3e453b8da3 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:17.979434 1332 ts_tablet_manager.cc:1918] T eeae5294bd474f368f5766c260ccf3fe P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250901 01:52:17.979594 1332 log.cc:1199] T eeae5294bd474f368f5766c260ccf3fe P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/eeae5294bd474f368f5766c260ccf3fe
W20250901 01:52:17.982299 858 leader_election.cc:336] T b009cc95c819431ea9827a3e453b8da3 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:17.987341 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 482a9a2cb81f448bb2823892e35920d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:18.002596 1306 tablet_replica.cc:331] T eeae5294bd474f368f5766c260ccf3fe P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:18.002838 1306 raft_consensus.cc:2241] T eeae5294bd474f368f5766c260ccf3fe P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:18.003055 1306 raft_consensus.cc:2270] T eeae5294bd474f368f5766c260ccf3fe P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:18.003602 1306 ts_tablet_manager.cc:1905] T eeae5294bd474f368f5766c260ccf3fe P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:18.005051 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet d58f69fad01f403eab74a47d3695ed83 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d58f69fad01f403eab74a47d3695ed83 already in progress: creating tablet
W20250901 01:52:18.005446 440 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.2:40561, user_credentials={real_user=slave}} blocked reactor thread for 61593.8us
I20250901 01:52:18.005954 1332 ts_tablet_manager.cc:1939] T eeae5294bd474f368f5766c260ccf3fe P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:18.014158 454 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_8ca4c54e4c424b888922086913acf096" } new_table_name: "default.Table_dc9c5d18c7ec478592a76dbd76424b88" modify_external_catalogs: true
I20250901 01:52:18.015756 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_b4fbc34da806475c912052d0fab3fc53"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:18.016330 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b4fbc34da806475c912052d0fab3fc53 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:18.024549 1293 raft_consensus.cc:491] T abe27e749fa0485195e0064a49d5431e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:18.024746 1293 raft_consensus.cc:513] T abe27e749fa0485195e0064a49d5431e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:18.025561 1293 leader_election.cc:290] T abe27e749fa0485195e0064a49d5431e P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:18.029114 858 leader_election.cc:336] T abe27e749fa0485195e0064a49d5431e P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:18.029275 1150 tablet_bootstrap.cc:492] T e6c10426f60f43e8bde69d9b1b9c8eb6 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:18.029474 1150 ts_tablet_manager.cc:1397] T e6c10426f60f43e8bde69d9b1b9c8eb6 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.085s user 0.005s sys 0.001s
I20250901 01:52:18.033154 1150 raft_consensus.cc:357] T e6c10426f60f43e8bde69d9b1b9c8eb6 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:18.033349 1150 raft_consensus.cc:383] T e6c10426f60f43e8bde69d9b1b9c8eb6 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:18.033413 1150 raft_consensus.cc:738] T e6c10426f60f43e8bde69d9b1b9c8eb6 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:18.036281 1150 consensus_queue.cc:260] T e6c10426f60f43e8bde69d9b1b9c8eb6 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:18.036885 1150 ts_tablet_manager.cc:1428] T e6c10426f60f43e8bde69d9b1b9c8eb6 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250901 01:52:18.037217 1150 tablet_bootstrap.cc:492] T 66a48a8ebe154b2cac89c234eaa378ff P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:18.038579 1306 ts_tablet_manager.cc:1918] T eeae5294bd474f368f5766c260ccf3fe P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250901 01:52:18.038705 1306 log.cc:1199] T eeae5294bd474f368f5766c260ccf3fe P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/eeae5294bd474f368f5766c260ccf3fe
I20250901 01:52:18.039165 1306 ts_tablet_manager.cc:1939] T eeae5294bd474f368f5766c260ccf3fe P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:18.041028 449 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_4ce20bf6ad224378861e083f6a480edf" } modify_external_catalogs: true
I20250901 01:52:18.041224 1150 tablet_bootstrap.cc:654] T 66a48a8ebe154b2cac89c234eaa378ff P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:18.041249 449 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_4ce20bf6ad224378861e083f6a480edf" } modify_external_catalogs: true
I20250901 01:52:18.050164 1293 raft_consensus.cc:491] T 2ff914352c7c43de800f3d54ecf63072 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:18.050369 1293 raft_consensus.cc:513] T 2ff914352c7c43de800f3d54ecf63072 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:18.051810 1293 leader_election.cc:290] T 2ff914352c7c43de800f3d54ecf63072 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:18.053264 858 leader_election.cc:336] T 2ff914352c7c43de800f3d54ecf63072 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:18.054905 447 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5adf6b492640417ca071c94702b48cb9 consensus state has no leader
I20250901 01:52:18.064563 1332 tablet_replica.cc:331] T a5f4b0a2d72a4b8095ae4af1fe7c0c5d P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:18.067971 1483 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "00c282f779c1477d920f24dc0dc16628" instance_seqno: 1756691527381543) as {username='slave'} at 127.0.104.2:54603; Asking this server to re-register.
I20250901 01:52:18.071012 969 heartbeater.cc:461] Registering TS with master...
I20250901 01:52:18.074472 1483 ts_manager.cc:194] Registered new tserver with Master: 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:18.005846 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet a774cc6ae3924d299127fd9ffb01ca57 (table default.Table_1159dbb70cc940df8b509db6e70c350a [id=1d0bc46740444f1fbf2129270e806259]) successfully deleted
W20250901 01:52:18.077569 440 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.2:40561, user_credentials={real_user=slave}} blocked reactor thread for 71733us
W20250901 01:52:18.077911 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 2240302d9178445d8700e35f5ea274f7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2240302d9178445d8700e35f5ea274f7 already in progress: creating tablet
I20250901 01:52:18.079293 1151 tablet_bootstrap.cc:492] T 58c0e7a2e90e48528bc2bc7ec889f608 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:18.079396 1669 raft_consensus.cc:491] T 2f28451a3ecc481a958588466f7fec77 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:18.081120 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 3b18a7084727490380d26b3485e4e799 (table default.Table_da2c70a8f2fe49a0a8184e88c0a9e0ea [id=c1b1e9e3876244f68d31c63ce7bdebf1]) successfully deleted
I20250901 01:52:18.082242 1669 raft_consensus.cc:513] T 2f28451a3ecc481a958588466f7fec77 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:18.084969 1306 tablet_replica.cc:331] T 533d14f5903e49599a34ea353ffd8c00 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:18.085218 1306 raft_consensus.cc:2241] T 533d14f5903e49599a34ea353ffd8c00 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:18.085460 1306 raft_consensus.cc:2270] T 533d14f5903e49599a34ea353ffd8c00 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:18.085791 448 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_c945e4cbcdb74acfa10533ff0bb3c0b2" } new_table_name: "default.Table_82d91a1ba2eb4265896824028dec8d3a" modify_external_catalogs: true
I20250901 01:52:18.086063 1306 ts_tablet_manager.cc:1905] T 533d14f5903e49599a34ea353ffd8c00 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:18.088279 454 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 435b1cca5a7d4d49959a310cb250b2a0 has no consensus state
I20250901 01:52:18.089270 1151 ts_tablet_manager.cc:1397] T 58c0e7a2e90e48528bc2bc7ec889f608 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.358s user 0.009s sys 0.004s
I20250901 01:52:18.089345 453 master_service.cc:948] ReplaceTablet: received request to replace tablet 6cf09e8f299548dba2cce16d7b565bad from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:18.089466 1669 leader_election.cc:290] T 2f28451a3ecc481a958588466f7fec77 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:18.089821 1332 raft_consensus.cc:2241] T a5f4b0a2d72a4b8095ae4af1fe7c0c5d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:18.090288 1332 raft_consensus.cc:2270] T a5f4b0a2d72a4b8095ae4af1fe7c0c5d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:18.091233 455 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_6f470109ee524f10a48d1080ec15a199" } modify_external_catalogs: true
I20250901 01:52:18.091352 447 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_e21873578fc14cc3b67e08478ec42192" } modify_external_catalogs: true
I20250901 01:52:18.091444 455 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_6f470109ee524f10a48d1080ec15a199" } modify_external_catalogs: true
I20250901 01:52:18.091534 447 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_e21873578fc14cc3b67e08478ec42192" } modify_external_catalogs: true
I20250901 01:52:18.092808 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_b7596c107f084e0fa1f68d1466314c6f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:18.093395 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b7596c107f084e0fa1f68d1466314c6f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250901 01:52:18.096248 448 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d1f2d98712ff4ea6aec665512de068bd has no consensus state
I20250901 01:52:18.096649 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_b7b2788a85494a56a4f6e87de2130cae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:18.097141 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b7b2788a85494a56a4f6e87de2130cae in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:18.101902 1306 ts_tablet_manager.cc:1918] T 533d14f5903e49599a34ea353ffd8c00 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:18.102064 1306 log.cc:1199] T 533d14f5903e49599a34ea353ffd8c00 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/533d14f5903e49599a34ea353ffd8c00
I20250901 01:52:18.102140 1332 ts_tablet_manager.cc:1905] T a5f4b0a2d72a4b8095ae4af1fe7c0c5d P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:18.102473 1306 ts_tablet_manager.cc:1939] T 533d14f5903e49599a34ea353ffd8c00 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:18.103000 1306 tablet_replica.cc:331] T 82d163681fda47f38d79863bd28f576d P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:18.104337 454 master_service.cc:948] ReplaceTablet: received request to replace tablet 7a2f6e2c7ee2402ea310c058e046416c from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:18.104823 1306 raft_consensus.cc:2241] T 82d163681fda47f38d79863bd28f576d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:18.105031 1306 raft_consensus.cc:2270] T 82d163681fda47f38d79863bd28f576d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:18.105984 1151 raft_consensus.cc:357] T 58c0e7a2e90e48528bc2bc7ec889f608 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:18.107002 1306 ts_tablet_manager.cc:1905] T 82d163681fda47f38d79863bd28f576d P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:18.107959 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_e533a2595cc9420f9ab7a36bda6dc4a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:18.108587 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e533a2595cc9420f9ab7a36bda6dc4a3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:18.111025 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 467c2bf4a8cb4937b6f627bea28d8ddb (table default.Table_a0e83c6959d64160a5fdec67e0e49ba3 [id=f3735a7cf9634c11abc2c8121e2ed930]) successfully deleted
I20250901 01:52:18.111393 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet eeae5294bd474f368f5766c260ccf3fe (table default.Table_a5d705c4e57a44b8aca8eb68c601e4ae [id=a48c2b3f8c4847cf9f9cf29548db5c50]) successfully deleted
I20250901 01:52:18.111585 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 533d14f5903e49599a34ea353ffd8c00 (table default.Table_489b87a28dac48d093c241f301d5cccc [id=5fdd352d5e5e47a3a2bd8fd608a263dd]) successfully deleted
I20250901 01:52:18.113131 1151 raft_consensus.cc:383] T 58c0e7a2e90e48528bc2bc7ec889f608 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:18.113343 1151 raft_consensus.cc:738] T 58c0e7a2e90e48528bc2bc7ec889f608 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:18.122278 1151 consensus_queue.cc:260] T 58c0e7a2e90e48528bc2bc7ec889f608 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:18.124733 1306 ts_tablet_manager.cc:1918] T 82d163681fda47f38d79863bd28f576d P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:18.124861 1306 log.cc:1199] T 82d163681fda47f38d79863bd28f576d P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/82d163681fda47f38d79863bd28f576d
I20250901 01:52:18.125190 1306 ts_tablet_manager.cc:1939] T 82d163681fda47f38d79863bd28f576d P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:18.125811 1306 tablet_replica.cc:331] T 5a8c21aee9f84216b6f90538a5bf4c17 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:18.126017 1306 raft_consensus.cc:2241] T 5a8c21aee9f84216b6f90538a5bf4c17 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:18.126164 1306 raft_consensus.cc:2270] T 5a8c21aee9f84216b6f90538a5bf4c17 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:18.126695 1306 ts_tablet_manager.cc:1905] T 5a8c21aee9f84216b6f90538a5bf4c17 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:18.130549 1483 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" instance_seqno: 1756691527859899) as {username='slave'} at 127.0.104.3:51295; Asking this server to re-register.
I20250901 01:52:18.133041 1306 ts_tablet_manager.cc:1918] T 5a8c21aee9f84216b6f90538a5bf4c17 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:18.133226 1306 log.cc:1199] T 5a8c21aee9f84216b6f90538a5bf4c17 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/5a8c21aee9f84216b6f90538a5bf4c17
I20250901 01:52:18.133869 1306 ts_tablet_manager.cc:1939] T 5a8c21aee9f84216b6f90538a5bf4c17 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:18.135630 1306 tablet_replica.cc:331] T abe27e749fa0485195e0064a49d5431e P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:18.135912 1306 raft_consensus.cc:2241] T abe27e749fa0485195e0064a49d5431e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:18.136149 1306 raft_consensus.cc:2270] T abe27e749fa0485195e0064a49d5431e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:18.136389 1103 heartbeater.cc:461] Registering TS with master...
I20250901 01:52:18.136916 1306 ts_tablet_manager.cc:1905] T abe27e749fa0485195e0064a49d5431e P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:18.141016 1483 ts_manager.cc:194] Registered new tserver with Master: fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:18.143666 1332 ts_tablet_manager.cc:1918] T a5f4b0a2d72a4b8095ae4af1fe7c0c5d P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:18.143873 1332 log.cc:1199] T a5f4b0a2d72a4b8095ae4af1fe7c0c5d P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/a5f4b0a2d72a4b8095ae4af1fe7c0c5d
I20250901 01:52:18.144114 1306 ts_tablet_manager.cc:1918] T abe27e749fa0485195e0064a49d5431e P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:18.144232 1306 log.cc:1199] T abe27e749fa0485195e0064a49d5431e P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/abe27e749fa0485195e0064a49d5431e
I20250901 01:52:18.144505 1332 ts_tablet_manager.cc:1939] T a5f4b0a2d72a4b8095ae4af1fe7c0c5d P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:18.144613 1306 ts_tablet_manager.cc:1939] T abe27e749fa0485195e0064a49d5431e P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:18.145167 1306 tablet_replica.cc:331] T c9a4c80bec1d41458974a0c449aecaf7 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:18.145303 1332 tablet_replica.cc:331] T 22691943a134400c8df4617240e5d3ef P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:18.145556 1332 raft_consensus.cc:2241] T 22691943a134400c8df4617240e5d3ef P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:18.145706 1306 raft_consensus.cc:2241] T c9a4c80bec1d41458974a0c449aecaf7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:18.145862 1306 raft_consensus.cc:2270] T c9a4c80bec1d41458974a0c449aecaf7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:18.145859 1332 raft_consensus.cc:2270] T 22691943a134400c8df4617240e5d3ef P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:18.146363 1306 ts_tablet_manager.cc:1905] T c9a4c80bec1d41458974a0c449aecaf7 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:18.146538 1332 ts_tablet_manager.cc:1905] T 22691943a134400c8df4617240e5d3ef P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:18.152449 1151 ts_tablet_manager.cc:1428] T 58c0e7a2e90e48528bc2bc7ec889f608 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.035s user 0.001s sys 0.001s
I20250901 01:52:18.155133 1306 ts_tablet_manager.cc:1918] T c9a4c80bec1d41458974a0c449aecaf7 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:18.155318 1306 log.cc:1199] T c9a4c80bec1d41458974a0c449aecaf7 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/c9a4c80bec1d41458974a0c449aecaf7
I20250901 01:52:18.155754 1306 ts_tablet_manager.cc:1939] T c9a4c80bec1d41458974a0c449aecaf7 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:18.156474 1306 tablet_replica.cc:331] T 98cca86164db4252b1b49de88747822a P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:18.156659 1306 raft_consensus.cc:2241] T 98cca86164db4252b1b49de88747822a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:18.156826 1306 raft_consensus.cc:2270] T 98cca86164db4252b1b49de88747822a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:18.157351 1306 ts_tablet_manager.cc:1905] T 98cca86164db4252b1b49de88747822a P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:18.159409 1151 tablet_bootstrap.cc:492] T fcb7b5b7d63d4f0b906fea97cb548de2 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:18.161276 1151 tablet_bootstrap.cc:654] T fcb7b5b7d63d4f0b906fea97cb548de2 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:18.165082 1306 ts_tablet_manager.cc:1918] T 98cca86164db4252b1b49de88747822a P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:18.165253 1306 log.cc:1199] T 98cca86164db4252b1b49de88747822a P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/98cca86164db4252b1b49de88747822a
I20250901 01:52:18.165724 1306 ts_tablet_manager.cc:1939] T 98cca86164db4252b1b49de88747822a P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:18.120464 448 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_1eb1cf735df04cbdaf0a6a86ef15aa7b" } new_table_name: "default.Table_c4148158ed7b48a3946559ba978e72fd" modify_external_catalogs: true
W20250901 01:52:18.166887 440 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.2:40561, user_credentials={real_user=slave}} blocked reactor thread for 55311.1us
I20250901 01:52:18.168108 889 tablet_service.cc:1468] Processing CreateTablet for tablet 6789505328b1409ea288e8b4eaf94935 (DEFAULT_TABLE table=default.Table_657603f2bb3c43bc85028218860da978 [id=013cf6104d1a47e592f90f6ba89653aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:18.168569 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6789505328b1409ea288e8b4eaf94935. 1 dirs total, 0 dirs full, 0 dirs failed
W20250901 01:52:18.174016 448 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cb7d7e1796c0456f8de98fb812b9e374 has no consensus state
I20250901 01:52:18.174484 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_583918346a424cb293d2c35bfa4a5b5f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:18.166620 992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:40731, user_credentials={real_user=slave}} blocked reactor thread for 210571us
I20250901 01:52:18.181249 903 tablet_service.cc:1468] Processing CreateTablet for tablet ed1b41340ce742a88c3e4fd85cd25ca6 (DEFAULT_TABLE table=default.Table_bda1777b1e9c409ea39c00de92706b50 [id=0baa4993b72e4a8d94ee3059c9d2db6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:18.181839 903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed1b41340ce742a88c3e4fd85cd25ca6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:18.186489 894 tablet_service.cc:1468] Processing CreateTablet for tablet ad1a6be9fcc045d3bb4e71a2f6bd5ccd (DEFAULT_TABLE table=default.Table_5e84ae9a9c2a4084a1f9c5ffb4594de2 [id=9bebb2e486ab48888e6b523281ac1e88]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:18.186707 897 tablet_service.cc:1468] Processing CreateTablet for tablet 14b466b2a5a249aa841f1b8b06301edc (DEFAULT_TABLE table=default.Table_ae42ba019ee14c69b1a16d1177f3642c [id=78cd71321e9a480ca2d7d88327f157ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:18.186971 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad1a6be9fcc045d3bb4e71a2f6bd5ccd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:18.187237 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14b466b2a5a249aa841f1b8b06301edc. 1 dirs total, 0 dirs full, 0 dirs failed
W20250901 01:52:18.190153 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_583918346a424cb293d2c35bfa4a5b5f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250901 01:52:18.191473 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 2240302d9178445d8700e35f5ea274f7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2240302d9178445d8700e35f5ea274f7 already in progress: creating tablet
W20250901 01:52:18.191953 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 3b18a7084727490380d26b3485e4e799 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b18a7084727490380d26b3485e4e799 already in progress: creating tablet
I20250901 01:52:18.192310 441 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 467c2bf4a8cb4937b6f627bea28d8ddb (table default.Table_a0e83c6959d64160a5fdec67e0e49ba3 [id=f3735a7cf9634c11abc2c8121e2ed930]) successfully deleted
W20250901 01:52:18.195268 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet c58edaa76ebf47dfb1d9d5fab6de7dfa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c58edaa76ebf47dfb1d9d5fab6de7dfa already in progress: creating tablet
W20250901 01:52:18.195829 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet e9b179dba2cc49428ba8090c962d6136 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e9b179dba2cc49428ba8090c962d6136 already in progress: creating tablet
I20250901 01:52:18.199028 1669 raft_consensus.cc:491] T 7442c6ff699141e1aff208ebdc9d192a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:18.199077 448 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_762edea68aae4d33ba89049358cfb236" } new_table_name: "default.Table_0e73f80424cd48408f18c2eff1411706" modify_external_catalogs: true
I20250901 01:52:18.199229 1669 raft_consensus.cc:513] T 7442c6ff699141e1aff208ebdc9d192a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:18.199368 992 leader_election.cc:343] T 1f18420261ae41459e1f25b927bd84a9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 1f18420261ae41459e1f25b927bd84a9
I20250901 01:52:18.199698 992 leader_election.cc:304] T 34042ea5c1b645969c1de41fe587c343 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:18.199882 1669 leader_election.cc:290] T 7442c6ff699141e1aff208ebdc9d192a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:18.200063 1669 raft_consensus.cc:2802] T 34042ea5c1b645969c1de41fe587c343 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:18.200148 1669 raft_consensus.cc:491] T 34042ea5c1b645969c1de41fe587c343 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:18.200248 1669 raft_consensus.cc:3058] T 34042ea5c1b645969c1de41fe587c343 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:18.201702 1552 consensus_queue.cc:1035] T 7cc1efcb6bff4aba8015d2d3b7f4c923 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:18.203336 1669 raft_consensus.cc:513] T 34042ea5c1b645969c1de41fe587c343 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:18.203954 1669 leader_election.cc:290] T 34042ea5c1b645969c1de41fe587c343 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:18.207710 448 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet becbaae4fd7e44faa679e9479465a7c4 has no consensus state
I20250901 01:52:18.209844 889 tablet_service.cc:1468] Processing CreateTablet for tablet 615b2306fca84003b264da9d5f95ddab (DEFAULT_TABLE table=default.Table_0f959237963c4e33932e193285c0eacb [id=cecdd8d6ca6d445c9fd82decd2a95366]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:18.210353 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_d228a4fe971b454ebe99e712ed80abdb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:18.210942 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d228a4fe971b454ebe99e712ed80abdb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:18.199024 1332 ts_tablet_manager.cc:1918] T 22691943a134400c8df4617240e5d3ef P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:18.218168 1332 log.cc:1199] T 22691943a134400c8df4617240e5d3ef P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/22691943a134400c8df4617240e5d3ef
I20250901 01:52:18.219868 1332 ts_tablet_manager.cc:1939] T 22691943a134400c8df4617240e5d3ef P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:18.209998 891 tablet_service.cc:1468] Processing CreateTablet for tablet 05630a72ddae4de0b890dece2feacc31 (DEFAULT_TABLE table=default.Table_ee242c34410e4c6887405d04a4b26124 [id=c9a2f96383664a4e8524c286cd9e4149]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:18.220963 1332 tablet_replica.cc:331] T 1f18420261ae41459e1f25b927bd84a9 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:18.210306 921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4410da9e95454ecf95faf80221b0a71c" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:18.221151 921 raft_consensus.cc:2466] T 4410da9e95454ecf95faf80221b0a71c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:18.221189 1332 raft_consensus.cc:2241] T 1f18420261ae41459e1f25b927bd84a9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:18.221346 1332 raft_consensus.cc:2270] T 1f18420261ae41459e1f25b927bd84a9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:18.221918 1151 tablet_bootstrap.cc:492] T fcb7b5b7d63d4f0b906fea97cb548de2 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:18.221963 1332 ts_tablet_manager.cc:1905] T 1f18420261ae41459e1f25b927bd84a9 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:18.222270 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 615b2306fca84003b264da9d5f95ddab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:18.228571 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05630a72ddae4de0b890dece2feacc31. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:18.228811 1151 ts_tablet_manager.cc:1397] T fcb7b5b7d63d4f0b906fea97cb548de2 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.069s user 0.001s sys 0.003s
I20250901 01:52:18.230592 1151 raft_consensus.cc:357] T fcb7b5b7d63d4f0b906fea97cb548de2 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:18.231506 1151 raft_consensus.cc:383] T fcb7b5b7d63d4f0b906fea97cb548de2 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:18.231804 1151 raft_consensus.cc:738] T fcb7b5b7d63d4f0b906fea97cb548de2 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:18.232077 1151 consensus_queue.cc:260] T fcb7b5b7d63d4f0b906fea97cb548de2 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:18.233235 859 leader_election.cc:343] T 82d163681fda47f38d79863bd28f576d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:18.233458 859 leader_election.cc:304] T 82d163681fda47f38d79863bd28f576d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:18.233783 859 raft_consensus.cc:2684] T 82d163681fda47f38d79863bd28f576d P 00c282f779c1477d920f24dc0dc16628: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250901 01:52:18.234172 859 leader_election.cc:343] T 606fac5c6efe48d1877104b01c23ba5b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:18.234401 859 leader_election.cc:304] T 606fac5c6efe48d1877104b01c23ba5b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:18.234741 1293 raft_consensus.cc:2747] T 606fac5c6efe48d1877104b01c23ba5b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:18.235203 859 leader_election.cc:343] T 533d14f5903e49599a34ea353ffd8c00 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:18.235453 859 leader_election.cc:304] T 533d14f5903e49599a34ea353ffd8c00 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:18.235684 859 raft_consensus.cc:2684] T 533d14f5903e49599a34ea353ffd8c00 P 00c282f779c1477d920f24dc0dc16628: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250901 01:52:18.244215 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_0ece0b3619db43da9d3bb2b0a72a7ef9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:18.244884 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0ece0b3619db43da9d3bb2b0a72a7ef9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:18.256505 448 master_service.cc:948] ReplaceTablet: received request to replace tablet a59ac57aebe24860aa02f53ce0485910 from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:18.264185 446 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet e6c10426f60f43e8bde69d9b1b9c8eb6 (table default.Table_eecfab7239cc40c5a2f1fb6ac6d35dc9 [id=d3dbeb4258424133844e20b181ee44e4]). Expected version 1 got 0
I20250901 01:52:18.265422 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_91214af783ff4aa3b6c3cbdb0358307e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:18.265967 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_91214af783ff4aa3b6c3cbdb0358307e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:18.268357 449 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 50e3da53433f4c14b57db8a733fa53bc
I20250901 01:52:18.269069 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_8deed8083e3f4f4f8533e42022a8633d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:18.269544 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8deed8083e3f4f4f8533e42022a8633d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:18.282177 1150 tablet_bootstrap.cc:492] T 66a48a8ebe154b2cac89c234eaa378ff P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:18.282393 1150 ts_tablet_manager.cc:1397] T 66a48a8ebe154b2cac89c234eaa378ff P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.245s user 0.026s sys 0.040s
I20250901 01:52:18.284898 1151 ts_tablet_manager.cc:1428] T fcb7b5b7d63d4f0b906fea97cb548de2 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.029s user 0.002s sys 0.000s
I20250901 01:52:18.283445 1150 raft_consensus.cc:357] T 66a48a8ebe154b2cac89c234eaa378ff P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:18.286522 1150 raft_consensus.cc:383] T 66a48a8ebe154b2cac89c234eaa378ff P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:18.286664 1150 raft_consensus.cc:738] T 66a48a8ebe154b2cac89c234eaa378ff P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:18.286332 1710 consensus_queue.cc:1035] T 756cfe84a517459ca015aecf5cd81571 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.050s
I20250901 01:52:18.289755 447 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet b81a12b38ac64efabd44aeb192d181a0
W20250901 01:52:18.289872 992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:40561, user_credentials={real_user=slave}} blocked reactor thread for 52324.8us
W20250901 01:52:18.290174 992 leader_election.cc:343] T ceb724d859d24d569598e18389dae3b3 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: ceb724d859d24d569598e18389dae3b3
W20250901 01:52:18.290452 992 leader_election.cc:343] T a5f4b0a2d72a4b8095ae4af1fe7c0c5d P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: a5f4b0a2d72a4b8095ae4af1fe7c0c5d
I20250901 01:52:18.290629 447 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_4e1d0964cecf42e99b61dc169027cee1" } new_table_name: "default.Table_14cf61059c5145b0a02159aae2a3e457" modify_external_catalogs: true
I20250901 01:52:18.291532 452 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 7a607fac57994001b2304e5d81c9decc
I20250901 01:52:18.293563 1552 consensus_queue.cc:1035] T c27bb23c150f4565a1d0ba8f231cc386 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:18.293643 992 leader_election.cc:304] T 4410da9e95454ecf95faf80221b0a71c P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
W20250901 01:52:18.294027 859 leader_election.cc:343] T c9a4c80bec1d41458974a0c449aecaf7 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:18.294164 1712 raft_consensus.cc:2802] T 4410da9e95454ecf95faf80221b0a71c P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:18.294160 859 leader_election.cc:304] T c9a4c80bec1d41458974a0c449aecaf7 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:18.294274 1712 raft_consensus.cc:491] T 4410da9e95454ecf95faf80221b0a71c P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250901 01:52:18.294410 859 raft_consensus.cc:2684] T c9a4c80bec1d41458974a0c449aecaf7 P 00c282f779c1477d920f24dc0dc16628: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250901 01:52:18.294384 1712 raft_consensus.cc:3058] T 4410da9e95454ecf95faf80221b0a71c P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
W20250901 01:52:18.294661 992 leader_election.cc:336] T 1f18420261ae41459e1f25b927bd84a9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:18.294817 992 leader_election.cc:304] T 1f18420261ae41459e1f25b927bd84a9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:18.294819 891 tablet_service.cc:1468] Processing CreateTablet for tablet 56475ba491a24301a534304811f30140 (DEFAULT_TABLE table=default.Table_b0e916c76a7a4766aa0887cc8691488f [id=cf61137a852c48748a29f833f1860da7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:18.295257 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56475ba491a24301a534304811f30140. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:18.295522 1714 tablet.cc:1722] T 85322a864d154f299f2d670fdd2a7bfb P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
W20250901 01:52:18.296594 991 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250901 01:52:18.296608 1151 tablet_bootstrap.cc:492] T 186ec810d02e4748aad15228b5866ac1 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:18.297742 1150 consensus_queue.cc:260] T 66a48a8ebe154b2cac89c234eaa378ff P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:18.298539 889 tablet_service.cc:1468] Processing CreateTablet for tablet d10c193452954c198bba686169b8025b (DEFAULT_TABLE table=default.Table_2872910642aa48eca0350b2792008200 [id=6c0576410d7340d6aa493a747e95565b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:18.299042 1293 raft_consensus.cc:491] T e6c10426f60f43e8bde69d9b1b9c8eb6 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:18.299196 1293 raft_consensus.cc:513] T e6c10426f60f43e8bde69d9b1b9c8eb6 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:18.299875 1293 leader_election.cc:290] T e6c10426f60f43e8bde69d9b1b9c8eb6 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:18.299960 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d10c193452954c198bba686169b8025b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250901 01:52:18.304450 858 leader_election.cc:336] T e6c10426f60f43e8bde69d9b1b9c8eb6 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:18.304868 1150 ts_tablet_manager.cc:1428] T 66a48a8ebe154b2cac89c234eaa378ff P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.022s user 0.001s sys 0.002s
I20250901 01:52:18.305227 1150 tablet_bootstrap.cc:492] T 629db80a8ab444e582628578155638fd P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:18.308315 1150 tablet_bootstrap.cc:654] T 629db80a8ab444e582628578155638fd P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:18.310563 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_442e11b9211c4b209c68f653d7bfc313"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:18.317024 992 raft_consensus.cc:2684] T 1f18420261ae41459e1f25b927bd84a9 P fd27ab6665304aaea53adbd102dd4f58: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250901 01:52:18.321801 1150 tablet_bootstrap.cc:492] T 629db80a8ab444e582628578155638fd P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:18.322033 1150 ts_tablet_manager.cc:1397] T 629db80a8ab444e582628578155638fd P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20250901 01:52:18.323945 894 tablet_service.cc:1468] Processing CreateTablet for tablet 0dbdcb76eeba4fefa56e7835c23c0316 (DEFAULT_TABLE table=default.Table_97f8d5c204174c918a795498e47d404d [id=d74c256799d64024b869f8076b857f63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:18.324074 902 tablet_service.cc:1515] Processing DeleteTablet for tablet c49d1143d54b4c319739488906b0e271 with delete_type TABLET_DATA_DELETED (replaced by tablet 69f61ec283484f9d8b3eebcea27f741b) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:18.324568 897 tablet_service.cc:1515] Processing DeleteTablet for tablet 0dbdcb76eeba4fefa56e7835c23c0316 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:17 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:18.324746 1293 raft_consensus.cc:491] T 66a48a8ebe154b2cac89c234eaa378ff P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:18.324811 903 tablet_service.cc:1515] Processing DeleteTablet for tablet 20f7bfab18134e079e91c52c89468cc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:17 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:18.324898 1293 raft_consensus.cc:513] T 66a48a8ebe154b2cac89c234eaa378ff P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:18.325026 904 tablet_service.cc:1515] Processing DeleteTablet for tablet e249474dcbf642189560c91ec3a99fdf with delete_type TABLET_DATA_DELETED (replaced by tablet 7f46e9ddadaf4ff986d8c991178daa0c) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:18.325196 900 tablet_service.cc:1515] Processing DeleteTablet for tablet afc002ae2486476b9208e1cf8215157c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:17 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:18.325415 898 tablet_service.cc:1515] Processing DeleteTablet for tablet 9473f0ef978c4bfaa41fb7b0729e8863 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:17 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:18.325562 1150 raft_consensus.cc:357] T 629db80a8ab444e582628578155638fd P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:18.325734 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0dbdcb76eeba4fefa56e7835c23c0316. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:18.325806 1150 raft_consensus.cc:383] T 629db80a8ab444e582628578155638fd P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:18.325896 1150 raft_consensus.cc:738] T 629db80a8ab444e582628578155638fd P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:18.326156 1150 consensus_queue.cc:260] T 629db80a8ab444e582628578155638fd P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:18.327843 858 leader_election.cc:336] T 66a48a8ebe154b2cac89c234eaa378ff P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:18.328204 441 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet eeae5294bd474f368f5766c260ccf3fe (table default.Table_a5d705c4e57a44b8aca8eb68c601e4ae [id=a48c2b3f8c4847cf9f9cf29548db5c50]) successfully deleted
I20250901 01:52:18.328457 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 8381305e4c4148808803e255b8a43a90
I20250901 01:52:18.328925 889 tablet_service.cc:1468] Processing CreateTablet for tablet 1a60c10e42104502ac2d3b80b8126ceb (DEFAULT_TABLE table=default.Table_53e30205c5fd488eb3b48c685a159d04 [id=60837b865ee7415ea42a85a953d837a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:18.329284 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a60c10e42104502ac2d3b80b8126ceb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:18.329577 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet af2c4bcbb00741f789edee109252d8cb
I20250901 01:52:18.330247 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_f4482a2fa4cc4cfd9658b47dda711f2d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:18.332319 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f4482a2fa4cc4cfd9658b47dda711f2d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:18.325734 1293 leader_election.cc:290] T 66a48a8ebe154b2cac89c234eaa378ff P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:18.332862 992 consensus_peers.cc:489] T 756cfe84a517459ca015aecf5cd81571 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250901 01:52:18.333017 992 leader_election.cc:336] T 34042ea5c1b645969c1de41fe587c343 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:18.333060 1150 ts_tablet_manager.cc:1428] T 629db80a8ab444e582628578155638fd P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.011s user 0.001s sys 0.002s
I20250901 01:52:18.333422 1150 tablet_bootstrap.cc:492] T c95798f0daf4437db18a291c4464522e P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:18.333899 453 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 6cf09e8f299548dba2cce16d7b565bad
I20250901 01:52:18.335259 1150 tablet_bootstrap.cc:654] T c95798f0daf4437db18a291c4464522e P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:18.336441 898 tablet_service.cc:1515] Processing DeleteTablet for tablet f50f33de1b654663b9ef8a91e3436f11 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:13 UTC) from {username='slave'} at 127.0.0.1:52846
W20250901 01:52:18.336485 992 consensus_peers.cc:489] T 7cc1efcb6bff4aba8015d2d3b7f4c923 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250901 01:52:18.336617 992 leader_election.cc:336] T ceb724d859d24d569598e18389dae3b3 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:18.336687 1306 tablet_replica.cc:331] T f50f33de1b654663b9ef8a91e3436f11 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:18.336915 1306 raft_consensus.cc:2241] T f50f33de1b654663b9ef8a91e3436f11 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:18.336993 453 catalog_manager.cc:6652] ReplaceTablet: tablet 6cf09e8f299548dba2cce16d7b565bad deleted and replaced by tablet 0e4009963cfe42d6ba164fc828897f09
I20250901 01:52:18.337114 1306 raft_consensus.cc:2270] T f50f33de1b654663b9ef8a91e3436f11 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:18.337313 449 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_b7596c107f084e0fa1f68d1466314c6f" } new_table_name: "default.Table_6e461d9cb0ae4dd095975bbc21533794" modify_external_catalogs: true
I20250901 01:52:18.337802 1306 ts_tablet_manager.cc:1905] T f50f33de1b654663b9ef8a91e3436f11 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:18.339915 992 leader_election.cc:304] T ceb724d859d24d569598e18389dae3b3 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
W20250901 01:52:18.340320 992 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111) [suppressed 81 similar messages]
I20250901 01:52:18.341694 1669 raft_consensus.cc:2747] T ceb724d859d24d569598e18389dae3b3 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:18.341675 921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7442c6ff699141e1aff208ebdc9d192a" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:18.341905 921 raft_consensus.cc:2466] T 7442c6ff699141e1aff208ebdc9d192a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:18.342235 1151 tablet_bootstrap.cc:654] T 186ec810d02e4748aad15228b5866ac1 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:18.343011 921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "34042ea5c1b645969c1de41fe587c343" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:18.343209 921 raft_consensus.cc:3058] T 34042ea5c1b645969c1de41fe587c343 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:18.346226 921 raft_consensus.cc:2466] T 34042ea5c1b645969c1de41fe587c343 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:18.347231 446 master_service.cc:948] ReplaceTablet: received request to replace tablet e25923b2daf9429683b2d44a75a56189 from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:18.347605 1306 ts_tablet_manager.cc:1918] T f50f33de1b654663b9ef8a91e3436f11 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:18.347749 1306 log.cc:1199] T f50f33de1b654663b9ef8a91e3436f11 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/f50f33de1b654663b9ef8a91e3436f11
I20250901 01:52:18.348156 1306 ts_tablet_manager.cc:1939] T f50f33de1b654663b9ef8a91e3436f11 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:18.348557 441 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet a5f4b0a2d72a4b8095ae4af1fe7c0c5d (table default.Table_457273ab524846e6954d3e60f698e178 [id=a2546d228d4d42bb9829c56443a4c867]) successfully deleted
I20250901 01:52:18.349726 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 7a2f6e2c7ee2402ea310c058e046416c
I20250901 01:52:18.350217 454 catalog_manager.cc:6652] ReplaceTablet: tablet 7a2f6e2c7ee2402ea310c058e046416c deleted and replaced by tablet 5f5396045979450193d461ba1334a5c9
W20250901 01:52:18.351190 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_442e11b9211c4b209c68f653d7bfc313 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:18.351127 1712 raft_consensus.cc:513] T 4410da9e95454ecf95faf80221b0a71c P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:18.351816 1712 leader_election.cc:290] T 4410da9e95454ecf95faf80221b0a71c P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:18.351852 1033 tablet_service.cc:1515] Processing DeleteTablet for tablet 50e3da53433f4c14b57db8a733fa53bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:18 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:18.351817 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_14e7f1e7dd314cdcaee09ac2fdf28993"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:18.353005 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_14e7f1e7dd314cdcaee09ac2fdf28993 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:18.353048 1033 tablet_service.cc:1468] Processing CreateTablet for tablet 1495de70f0844f82a0eea99c059f076b (DEFAULT_TABLE table=default.Table_aac69ecb4bd042ad83e7ea0fc72f6ef9 [id=70a5ce71d16b47d4bc6dae97be4548e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:18.353541 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1495de70f0844f82a0eea99c059f076b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:18.353633 453 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_adb18e9d4ac64721b777cd20a2cd0800" } new_table_name: "default.Table_15c9f4f02eb24474936a43d705f90f7b" modify_external_catalogs: true
I20250901 01:52:18.354171 1332 ts_tablet_manager.cc:1918] T 1f18420261ae41459e1f25b927bd84a9 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:18.354308 1332 log.cc:1199] T 1f18420261ae41459e1f25b927bd84a9 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/1f18420261ae41459e1f25b927bd84a9
I20250901 01:52:18.354753 1332 ts_tablet_manager.cc:1939] T 1f18420261ae41459e1f25b927bd84a9 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:18.355679 1332 tablet_replica.cc:331] T ceb724d859d24d569598e18389dae3b3 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:18.355887 1332 raft_consensus.cc:2241] T ceb724d859d24d569598e18389dae3b3 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:18.356086 1332 raft_consensus.cc:2270] T ceb724d859d24d569598e18389dae3b3 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250901 01:52:18.358587 447 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet eaa521c52a2f4408b59770bd2a05907c consensus state has no leader
I20250901 01:52:18.360244 1712 raft_consensus.cc:491] T 58c0e7a2e90e48528bc2bc7ec889f608 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:18.360459 1712 raft_consensus.cc:513] T 58c0e7a2e90e48528bc2bc7ec889f608 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:18.361037 1712 leader_election.cc:290] T 58c0e7a2e90e48528bc2bc7ec889f608 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:18.361449 1332 ts_tablet_manager.cc:1905] T ceb724d859d24d569598e18389dae3b3 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:18.362771 1038 tablet_service.cc:1468] Processing CreateTablet for tablet 48a211e174054866b030cdfb0cf947f4 (DEFAULT_TABLE table=default.Table_7c23fc28bbae4efe982c240b87f40956 [id=65d035b79f004c3d9ac3b0cd6a515d45]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:18.362859 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 82d163681fda47f38d79863bd28f576d (table default.Table_dfd01d6b61ac43538c37f74330579cbd [id=b067d050196e4185b76c016c93f27c02]) successfully deleted
I20250901 01:52:18.363081 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 5a8c21aee9f84216b6f90538a5bf4c17 (table default.Table_d6119d1604674319914fa8cb698f16c5 [id=40d05d90b0354334840ad17764f5b6bb]) successfully deleted
I20250901 01:52:18.363226 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48a211e174054866b030cdfb0cf947f4. 1 dirs total, 0 dirs full, 0 dirs failed
W20250901 01:52:18.363284 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet c58edaa76ebf47dfb1d9d5fab6de7dfa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c58edaa76ebf47dfb1d9d5fab6de7dfa already in progress: creating tablet
W20250901 01:52:18.363466 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet e9b179dba2cc49428ba8090c962d6136 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e9b179dba2cc49428ba8090c962d6136 already in progress: creating tablet
W20250901 01:52:18.363685 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet f3962b471bdb49aea6e9fa95f1f9dfe2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f3962b471bdb49aea6e9fa95f1f9dfe2 already in progress: creating tablet
I20250901 01:52:18.363838 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet abe27e749fa0485195e0064a49d5431e (table default.Table_30943b75394441a5a55b195055d1e0cc [id=7e7a4771910546f7ba8c4369c768c943]) successfully deleted
I20250901 01:52:18.363981 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet c9a4c80bec1d41458974a0c449aecaf7 (table default.Table_117c6c43b552476dbb8b7b5ccdce47b7 [id=a4c947d85200437395c66bb4f7e607af]) successfully deleted
I20250901 01:52:18.364107 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 98cca86164db4252b1b49de88747822a (table default.Table_22d8d38d0b624912a0d8968fb764420e [id=808a8849c99e485b8be7cd17397874ec]) successfully deleted
W20250901 01:52:18.365193 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet c49d1143d54b4c319739488906b0e271 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c49d1143d54b4c319739488906b0e271 already in progress: creating tablet
W20250901 01:52:18.365447 440 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 0dbdcb76eeba4fefa56e7835c23c0316 because the tablet was not found. No further retry: Not found: Tablet not found: 0dbdcb76eeba4fefa56e7835c23c0316
W20250901 01:52:18.365617 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 20f7bfab18134e079e91c52c89468cc3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 20f7bfab18134e079e91c52c89468cc3 already in progress: creating tablet
W20250901 01:52:18.365762 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet e249474dcbf642189560c91ec3a99fdf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e249474dcbf642189560c91ec3a99fdf already in progress: creating tablet
W20250901 01:52:18.365907 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet afc002ae2486476b9208e1cf8215157c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet afc002ae2486476b9208e1cf8215157c already in progress: creating tablet
W20250901 01:52:18.366043 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 9473f0ef978c4bfaa41fb7b0729e8863 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9473f0ef978c4bfaa41fb7b0729e8863 already in progress: creating tablet
I20250901 01:52:18.366268 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet f50f33de1b654663b9ef8a91e3436f11 (table default.Table_4f1c4695ad9e4f24a31d3522281a41a5 [id=9af016a062a64218ad3b62e131d8b65d]) successfully deleted
I20250901 01:52:18.367851 1150 tablet_bootstrap.cc:492] T c95798f0daf4437db18a291c4464522e P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:18.367997 1150 ts_tablet_manager.cc:1397] T c95798f0daf4437db18a291c4464522e P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.000s
I20250901 01:52:18.368503 451 catalog_manager.cc:6652] ReplaceTablet: tablet 8381305e4c4148808803e255b8a43a90 deleted and replaced by tablet 8e49863da5814e43b442e6081ee911b9
I20250901 01:52:18.368983 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_df46d9604f6a4a19b4d7c3424f48d90a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:18.369246 1150 raft_consensus.cc:357] T c95798f0daf4437db18a291c4464522e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:18.369496 1150 raft_consensus.cc:383] T c95798f0daf4437db18a291c4464522e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:18.369585 1150 raft_consensus.cc:738] T c95798f0daf4437db18a291c4464522e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:18.369841 1150 consensus_queue.cc:260] T c95798f0daf4437db18a291c4464522e P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:18.370041 1029 tablet_service.cc:1468] Processing CreateTablet for tablet b81a12b38ac64efabd44aeb192d181a0 (DEFAULT_TABLE table=default.Table_e21873578fc14cc3b67e08478ec42192 [id=a404c32b3e8146f288ce7ea3a017ab96]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:18.370497 1150 ts_tablet_manager.cc:1428] T c95798f0daf4437db18a291c4464522e P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250901 01:52:18.370517 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b81a12b38ac64efabd44aeb192d181a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:18.370803 1150 tablet_bootstrap.cc:492] T ef58f477ebeb49639ed7d071e1a62c10 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:18.371718 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 4902062977ea42149e3466e999117322 (DEFAULT_TABLE table=default.Table_fb5ee5a5a9514ddaac740c194a820fd2 [id=48b7b856dca841eba100ba8acf02b7e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:18.372128 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4902062977ea42149e3466e999117322. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:18.372671 1150 tablet_bootstrap.cc:654] T ef58f477ebeb49639ed7d071e1a62c10 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:18.372749 1036 tablet_service.cc:1468] Processing CreateTablet for tablet 4cde5f4f02044b7bbd7d43b49aac4ae8 (DEFAULT_TABLE table=default.Table_c931e9dfbef04fd3b49aa4f43db3d981 [id=77ae3376281349e087a3b3cee41f2a09]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:18.373139 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cde5f4f02044b7bbd7d43b49aac4ae8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:18.378280 441 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 22691943a134400c8df4617240e5d3ef (table default.Table_62ececf8293e4e22bad9e9568f496bbe [id=5a33cfdf60684867bb5f36d84ee80468]) successfully deleted
W20250901 01:52:18.378856 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_df46d9604f6a4a19b4d7c3424f48d90a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:18.380160 1035 tablet_service.cc:1468] Processing CreateTablet for tablet a9b7f2e6548043a7865e25387cdd06c9 (DEFAULT_TABLE table=default.Table_d28d7742171c4465a8b438d2e337b5f7 [id=5bb33f9da2dc4a9cbf17b31cd14fdd23]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:18.380556 1038 tablet_service.cc:1515] Processing DeleteTablet for tablet b81a12b38ac64efabd44aeb192d181a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:18 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:18.380617 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9b7f2e6548043a7865e25387cdd06c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:18.380867 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 7a607fac57994001b2304e5d81c9decc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:17 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:18.382493 1055 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66a48a8ebe154b2cac89c234eaa378ff" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:18.382553 1054 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e6c10426f60f43e8bde69d9b1b9c8eb6" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:18.382870 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abe27e749fa0485195e0064a49d5431e" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:18.382970 1714 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:44246 (request call id 327) took 1229 ms. Trace:
I20250901 01:52:18.383069 1714 rpcz_store.cc:276] 0901 01:52:17.153118 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:17.153156 (+ 38us) service_pool.cc:225] Handling call
0901 01:52:18.382933 (+1229777us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0901 01:52:17.158767 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0901 01:52:17.158852 (+ 85us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0901 01:52:17.158859 (+ 7us) alter_schema_op.cc:66] Acquired schema lock
0901 01:52:17.158866 (+ 7us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0901 01:52:17.158894 (+ 28us) alter_schema_op.cc:128] START. Timestamp: 1756691537158860
0901 01:52:17.158897 (+ 3us) op_driver.cc:348] REPLICATION: starting
0901 01:52:17.159018 (+ 121us) log.cc:844] Serialized 173 byte log entry
0901 01:52:18.260549 (+1101531us) op_driver.cc:464] REPLICATION: finished
0901 01:52:18.295466 (+ 34917us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0901 01:52:18.295919 (+ 453us) log.cc:844] Serialized 5 byte log entry
0901 01:52:18.380127 (+ 84208us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:18.382528 (+ 2401us) log.cc:844] Serialized 14 byte log entry
0901 01:52:18.382898 (+ 370us) alter_schema_op.cc:73] Released schema lock
0901 01:52:18.382904 (+ 6us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":34846,"prepare.queue_time_us":5172,"prepare.run_cpu_time_us":100836,"prepare.run_wall_time_us":324098,"raft.queue_time_us":14166,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":201,"replication_time_us":1101596,"thread_start_us":493094,"threads_started":5,"wal-append.queue_time_us":144229,"wal-append.run_cpu_time_us":660,"wal-append.run_wall_time_us":1011172}]]}
I20250901 01:52:18.383978 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet af2c4bcbb00741f789edee109252d8cb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:18 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:18.384160 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 8381305e4c4148808803e255b8a43a90 with delete_type TABLET_DATA_DELETED (replaced by tablet 8e49863da5814e43b442e6081ee911b9) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:18.384370 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 6cf09e8f299548dba2cce16d7b565bad with delete_type TABLET_DATA_DELETED (replaced by tablet 0e4009963cfe42d6ba164fc828897f09) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:18.384572 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 7a2f6e2c7ee2402ea310c058e046416c with delete_type TABLET_DATA_DELETED (replaced by tablet 5f5396045979450193d461ba1334a5c9) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:18.384781 1058 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b009cc95c819431ea9827a3e453b8da3" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:18.385087 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2ff914352c7c43de800f3d54ecf63072" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:18.387346 1151 tablet_bootstrap.cc:492] T 186ec810d02e4748aad15228b5866ac1 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:18.387527 1151 ts_tablet_manager.cc:1397] T 186ec810d02e4748aad15228b5866ac1 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.102s user 0.003s sys 0.003s
I20250901 01:52:18.388307 1151 raft_consensus.cc:357] T 186ec810d02e4748aad15228b5866ac1 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:18.388576 1151 raft_consensus.cc:383] T 186ec810d02e4748aad15228b5866ac1 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:18.388662 1151 raft_consensus.cc:738] T 186ec810d02e4748aad15228b5866ac1 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:18.388909 1151 consensus_queue.cc:260] T 186ec810d02e4748aad15228b5866ac1 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:18.393440 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet a59ac57aebe24860aa02f53ce0485910
I20250901 01:52:18.393957 448 catalog_manager.cc:6652] ReplaceTablet: tablet a59ac57aebe24860aa02f53ce0485910 deleted and replaced by tablet 93bcac1fd95d44dc927abfca81692921
I20250901 01:52:18.394325 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_7dd48000e9b24337b75ceb4c53b6be25"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:18.394896 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7dd48000e9b24337b75ceb4c53b6be25 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:18.405215 1712 raft_consensus.cc:491] T 186ec810d02e4748aad15228b5866ac1 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:18.405431 1712 raft_consensus.cc:513] T 186ec810d02e4748aad15228b5866ac1 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:18.406051 1712 leader_election.cc:290] T 186ec810d02e4748aad15228b5866ac1 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:18.407478 1151 ts_tablet_manager.cc:1428] T 186ec810d02e4748aad15228b5866ac1 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20250901 01:52:18.411741 1151 tablet_bootstrap.cc:492] T 3df908c3096a4df7bddd8ac88b49cfe0 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:18.413569 1151 tablet_bootstrap.cc:654] T 3df908c3096a4df7bddd8ac88b49cfe0 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:18.415246 447 catalog_manager.cc:5582] T f65ec84958eb488d8b0f812b59b5b3d5 P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: UNKNOWN } } }
I20250901 01:52:18.419648 1332 ts_tablet_manager.cc:1918] T ceb724d859d24d569598e18389dae3b3 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:18.419777 1332 log.cc:1199] T ceb724d859d24d569598e18389dae3b3 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/ceb724d859d24d569598e18389dae3b3
W20250901 01:52:18.423303 439 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet eaa521c52a2f4408b59770bd2a05907c (table default.Table_14cf61059c5145b0a02159aae2a3e457 [id=a33b9b981bd64d7981829d663b0b48b5], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet eaa521c52a2f4408b59770bd2a05907c consensus state has no leader
I20250901 01:52:18.423596 1332 ts_tablet_manager.cc:1939] T ceb724d859d24d569598e18389dae3b3 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:18.423885 1150 tablet_bootstrap.cc:492] T ef58f477ebeb49639ed7d071e1a62c10 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:18.424048 1150 ts_tablet_manager.cc:1397] T ef58f477ebeb49639ed7d071e1a62c10 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.053s user 0.014s sys 0.005s
I20250901 01:52:18.425349 1293 raft_consensus.cc:491] T c95798f0daf4437db18a291c4464522e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:18.425559 1293 raft_consensus.cc:513] T c95798f0daf4437db18a291c4464522e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:18.425776 1150 raft_consensus.cc:357] T ef58f477ebeb49639ed7d071e1a62c10 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:18.425948 1150 raft_consensus.cc:383] T ef58f477ebeb49639ed7d071e1a62c10 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:18.426018 1150 raft_consensus.cc:738] T ef58f477ebeb49639ed7d071e1a62c10 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:18.426231 1293 leader_election.cc:290] T c95798f0daf4437db18a291c4464522e P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:18.426234 1150 consensus_queue.cc:260] T ef58f477ebeb49639ed7d071e1a62c10 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:18.427774 1150 ts_tablet_manager.cc:1428] T ef58f477ebeb49639ed7d071e1a62c10 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250901 01:52:18.428073 1150 tablet_bootstrap.cc:492] T b73e362321284df1b18ef755dfe930a5 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:18.430073 1150 tablet_bootstrap.cc:654] T b73e362321284df1b18ef755dfe930a5 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:18.430495 858 leader_election.cc:336] T c95798f0daf4437db18a291c4464522e P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:18.430905 992 leader_election.cc:336] T 4410da9e95454ecf95faf80221b0a71c P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:18.431211 992 consensus_peers.cc:489] T c27bb23c150f4565a1d0ba8f231cc386 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250901 01:52:18.433672 992 leader_election.cc:336] T 2f28451a3ecc481a958588466f7fec77 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:18.433842 992 leader_election.cc:336] T a5f4b0a2d72a4b8095ae4af1fe7c0c5d P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:18.433980 992 leader_election.cc:304] T a5f4b0a2d72a4b8095ae4af1fe7c0c5d P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
W20250901 01:52:18.434244 992 raft_consensus.cc:2684] T a5f4b0a2d72a4b8095ae4af1fe7c0c5d P fd27ab6665304aaea53adbd102dd4f58: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250901 01:52:18.437935 1727 tablet.cc:1722] T 85322a864d154f299f2d670fdd2a7bfb P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250901 01:52:18.438481 441 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 1f18420261ae41459e1f25b927bd84a9 (table default.Table_a2bf411424c74109a7c2baa28a4db8b9 [id=7c904bfb6cc94518873d6806d1fda600]) successfully deleted
I20250901 01:52:18.439009 455 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_b7b2788a85494a56a4f6e87de2130cae" } new_table_name: "default.Table_fbc57d539a2b41e59dfc5369ac8c92e1" modify_external_catalogs: true
I20250901 01:52:18.441725 1712 raft_consensus.cc:491] T fcb7b5b7d63d4f0b906fea97cb548de2 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:18.441923 1712 raft_consensus.cc:513] T fcb7b5b7d63d4f0b906fea97cb548de2 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:18.442040 450 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 8e49863da5814e43b442e6081ee911b9
I20250901 01:52:18.438151 920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4410da9e95454ecf95faf80221b0a71c" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:18.442607 1712 leader_election.cc:290] T fcb7b5b7d63d4f0b906fea97cb548de2 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:18.442615 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_9734ae6c72e240c698b66ea83c536249"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:18.442751 920 raft_consensus.cc:3058] T 4410da9e95454ecf95faf80221b0a71c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
W20250901 01:52:18.442878 992 leader_election.cc:343] T 2f28451a3ecc481a958588466f7fec77 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 2f28451a3ecc481a958588466f7fec77
I20250901 01:52:18.442992 992 leader_election.cc:304] T 2f28451a3ecc481a958588466f7fec77 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
W20250901 01:52:18.443193 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9734ae6c72e240c698b66ea83c536249 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:18.443670 1712 raft_consensus.cc:2747] T 2f28451a3ecc481a958588466f7fec77 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:18.444077 992 leader_election.cc:304] T 7442c6ff699141e1aff208ebdc9d192a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
W20250901 01:52:18.444634 449 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 81ce448b148447ddbc41875d863e0518 has no consensus state
I20250901 01:52:18.444765 1712 raft_consensus.cc:2802] T 7442c6ff699141e1aff208ebdc9d192a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:18.444870 1712 raft_consensus.cc:491] T 7442c6ff699141e1aff208ebdc9d192a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:18.444970 1712 raft_consensus.cc:3058] T 7442c6ff699141e1aff208ebdc9d192a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:18.445675 920 raft_consensus.cc:2466] T 4410da9e95454ecf95faf80221b0a71c P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
W20250901 01:52:18.439646 859 leader_election.cc:343] T 66a48a8ebe154b2cac89c234eaa378ff P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:18.445885 859 leader_election.cc:304] T 66a48a8ebe154b2cac89c234eaa378ff P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:18.446355 859 leader_election.cc:343] T e6c10426f60f43e8bde69d9b1b9c8eb6 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:18.446501 859 leader_election.cc:304] T e6c10426f60f43e8bde69d9b1b9c8eb6 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:18.446560 449 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_8393a17fa28a481888acfc550afdb6a1" } new_table_name: "default.Table_d9e28b97a2e94ca8bf0ee8a01acd76e8" modify_external_catalogs: true
W20250901 01:52:18.446877 859 leader_election.cc:343] T abe27e749fa0485195e0064a49d5431e P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:18.446995 859 leader_election.cc:304] T abe27e749fa0485195e0064a49d5431e P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:18.447161 859 raft_consensus.cc:2684] T abe27e749fa0485195e0064a49d5431e P 00c282f779c1477d920f24dc0dc16628: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250901 01:52:18.447546 446 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet e25923b2daf9429683b2d44a75a56189
I20250901 01:52:18.447777 1293 raft_consensus.cc:2747] T 66a48a8ebe154b2cac89c234eaa378ff P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:18.447953 1293 raft_consensus.cc:2747] T e6c10426f60f43e8bde69d9b1b9c8eb6 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:18.448736 446 catalog_manager.cc:6652] ReplaceTablet: tablet e25923b2daf9429683b2d44a75a56189 deleted and replaced by tablet 2973b59bb1df41d18d9a5b6a29ae336f
I20250901 01:52:18.448666 1712 raft_consensus.cc:513] T 7442c6ff699141e1aff208ebdc9d192a P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:18.449090 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_09ffc2a7b4b14b25adbc5471363c02af"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:18.449738 1712 leader_election.cc:290] T 7442c6ff699141e1aff208ebdc9d192a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:18.450261 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_09ffc2a7b4b14b25adbc5471363c02af in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250901 01:52:18.451274 453 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7f46e9ddadaf4ff986d8c991178daa0c has no consensus state
I20250901 01:52:18.453806 992 leader_election.cc:304] T 34042ea5c1b645969c1de41fe587c343 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:18.454288 447 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet f65ec84958eb488d8b0f812b59b5b3d5 (table default.Table_a3ff88e352884abdba91e6d03acbae22 [id=d1cd5288671540c0a95f7a037363014e]). Expected version 2 got 0
I20250901 01:52:18.454483 1712 raft_consensus.cc:2802] T 34042ea5c1b645969c1de41fe587c343 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:18.454674 447 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 85322a864d154f299f2d670fdd2a7bfb (table default.Table_5a72e7206f604795915f3a577d37f874 [id=1d8f34cc14cb4e90a562ff4424b8577a]). Expected version 5 got 4
I20250901 01:52:18.455135 447 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 7cc1efcb6bff4aba8015d2d3b7f4c923 (table default.Table_5610feff8a344dc9be4bc1f0ad8ddd51 [id=78b00856a9b747c19f390b4fdae8ddcf]). Expected version 3 got 0
I20250901 01:52:18.455333 1712 raft_consensus.cc:695] T 34042ea5c1b645969c1de41fe587c343 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:18.455705 1712 consensus_queue.cc:237] T 34042ea5c1b645969c1de41fe587c343 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:18.457801 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 50e3da53433f4c14b57db8a733fa53bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:18 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:18.458277 889 tablet_service.cc:1468] Processing CreateTablet for tablet 1495de70f0844f82a0eea99c059f076b (DEFAULT_TABLE table=default.Table_aac69ecb4bd042ad83e7ea0fc72f6ef9 [id=70a5ce71d16b47d4bc6dae97be4548e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:18.458426 894 tablet_service.cc:1468] Processing CreateTablet for tablet 48a211e174054866b030cdfb0cf947f4 (DEFAULT_TABLE table=default.Table_7c23fc28bbae4efe982c240b87f40956 [id=65d035b79f004c3d9ac3b0cd6a515d45]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:18.458741 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1495de70f0844f82a0eea99c059f076b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:18.458992 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48a211e174054866b030cdfb0cf947f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:18.462386 898 tablet_service.cc:1468] Processing CreateTablet for tablet b81a12b38ac64efabd44aeb192d181a0 (DEFAULT_TABLE table=default.Table_e21873578fc14cc3b67e08478ec42192 [id=a404c32b3e8146f288ce7ea3a017ab96]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:18.462790 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b81a12b38ac64efabd44aeb192d181a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:18.470876 891 tablet_service.cc:1468] Processing CreateTablet for tablet ff013ea889d44cba8797834e4611b489 (DEFAULT_TABLE table=default.Table_0f91070385df441b96af4d5406a94724 [id=4747cdd47a954e0ab19609c9526b688d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:18.471401 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff013ea889d44cba8797834e4611b489. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:18.472656 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_ac189f6a9fcf457e9fa0d77badc81594"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:18.473449 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ac189f6a9fcf457e9fa0d77badc81594 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250901 01:52:18.478241 992 leader_election.cc:343] T 58c0e7a2e90e48528bc2bc7ec889f608 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 58c0e7a2e90e48528bc2bc7ec889f608
I20250901 01:52:18.478521 992 leader_election.cc:304] T 4410da9e95454ecf95faf80221b0a71c P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:18.484807 452 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_ee242c34410e4c6887405d04a4b26124" } modify_external_catalogs: true
I20250901 01:52:18.484990 452 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_ee242c34410e4c6887405d04a4b26124" } modify_external_catalogs: true
I20250901 01:52:18.492676 1669 raft_consensus.cc:2802] T 4410da9e95454ecf95faf80221b0a71c P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:18.492872 1669 raft_consensus.cc:695] T 4410da9e95454ecf95faf80221b0a71c P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:18.493228 1669 consensus_queue.cc:237] T 4410da9e95454ecf95faf80221b0a71c P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:18.499130 1710 raft_consensus.cc:491] T 3b18a7084727490380d26b3485e4e799 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:18.499212 448 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_e066e678c3a8406583dbe851d3424bac" } modify_external_catalogs: true
I20250901 01:52:18.499321 1710 raft_consensus.cc:513] T 3b18a7084727490380d26b3485e4e799 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:18.499606 448 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_e066e678c3a8406583dbe851d3424bac" } modify_external_catalogs: true
I20250901 01:52:18.500094 1710 leader_election.cc:290] T 3b18a7084727490380d26b3485e4e799 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:18.501696 859 leader_election.cc:343] T b009cc95c819431ea9827a3e453b8da3 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:18.501852 859 leader_election.cc:304] T b009cc95c819431ea9827a3e453b8da3 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:18.502322 859 leader_election.cc:343] T 2ff914352c7c43de800f3d54ecf63072 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:18.502419 921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "186ec810d02e4748aad15228b5866ac1" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:18.502526 441 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet ceb724d859d24d569598e18389dae3b3 (table default.Table_063de0c73851415196ba2e77742edee4 [id=696debd71f87437a880d4ad17fdf5ad8]) successfully deleted
W20250901 01:52:18.502560 992 leader_election.cc:336] T 7442c6ff699141e1aff208ebdc9d192a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:18.502651 921 raft_consensus.cc:2466] T 186ec810d02e4748aad15228b5866ac1 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
W20250901 01:52:18.502782 992 leader_election.cc:336] T 34042ea5c1b645969c1de41fe587c343 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:18.502892 1711 raft_consensus.cc:2747] T b009cc95c819431ea9827a3e453b8da3 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:18.502444 859 leader_election.cc:304] T 2ff914352c7c43de800f3d54ecf63072 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:18.503134 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet f3962b471bdb49aea6e9fa95f1f9dfe2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f3962b471bdb49aea6e9fa95f1f9dfe2 already in progress: creating tablet
W20250901 01:52:18.503335 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 98cca86164db4252b1b49de88747822a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 98cca86164db4252b1b49de88747822a already in progress: creating tablet
I20250901 01:52:18.503420 1711 raft_consensus.cc:2747] T 2ff914352c7c43de800f3d54ecf63072 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:18.503522 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet afc002ae2486476b9208e1cf8215157c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet afc002ae2486476b9208e1cf8215157c already in progress: creating tablet
W20250901 01:52:18.503738 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet e249474dcbf642189560c91ec3a99fdf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e249474dcbf642189560c91ec3a99fdf already in progress: creating tablet
W20250901 01:52:18.503961 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 20f7bfab18134e079e91c52c89468cc3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 20f7bfab18134e079e91c52c89468cc3 already in progress: creating tablet
W20250901 01:52:18.504189 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet c49d1143d54b4c319739488906b0e271 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c49d1143d54b4c319739488906b0e271 already in progress: creating tablet
W20250901 01:52:18.504565 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 0dbdcb76eeba4fefa56e7835c23c0316 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0dbdcb76eeba4fefa56e7835c23c0316 already in progress: creating tablet
W20250901 01:52:18.510689 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 9473f0ef978c4bfaa41fb7b0729e8863 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9473f0ef978c4bfaa41fb7b0729e8863 already in progress: creating tablet
I20250901 01:52:18.510728 919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcb7b5b7d63d4f0b906fea97cb548de2" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
W20250901 01:52:18.510911 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 50e3da53433f4c14b57db8a733fa53bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 50e3da53433f4c14b57db8a733fa53bc already in progress: creating tablet
I20250901 01:52:18.510948 919 raft_consensus.cc:2466] T fcb7b5b7d63d4f0b906fea97cb548de2 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
W20250901 01:52:18.511150 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet b81a12b38ac64efabd44aeb192d181a0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b81a12b38ac64efabd44aeb192d181a0 already in progress: creating tablet
W20250901 01:52:18.511384 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 7a607fac57994001b2304e5d81c9decc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7a607fac57994001b2304e5d81c9decc already in progress: creating tablet
W20250901 01:52:18.511617 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet af2c4bcbb00741f789edee109252d8cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet af2c4bcbb00741f789edee109252d8cb already in progress: creating tablet
W20250901 01:52:18.511929 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 6cf09e8f299548dba2cce16d7b565bad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6cf09e8f299548dba2cce16d7b565bad already in progress: creating tablet
I20250901 01:52:18.511981 919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7442c6ff699141e1aff208ebdc9d192a" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
W20250901 01:52:18.512187 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 7a2f6e2c7ee2402ea310c058e046416c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7a2f6e2c7ee2402ea310c058e046416c already in progress: creating tablet
I20250901 01:52:18.512204 919 raft_consensus.cc:3058] T 7442c6ff699141e1aff208ebdc9d192a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:18.514643 900 tablet_service.cc:1515] Processing DeleteTablet for tablet 7a607fac57994001b2304e5d81c9decc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:17 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:18.515210 904 tablet_service.cc:1515] Processing DeleteTablet for tablet af2c4bcbb00741f789edee109252d8cb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:18 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:18.515167 900 tablet_service.cc:1468] Processing CreateTablet for tablet a9b7f2e6548043a7865e25387cdd06c9 (DEFAULT_TABLE table=default.Table_d28d7742171c4465a8b438d2e337b5f7 [id=5bb33f9da2dc4a9cbf17b31cd14fdd23]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:18.515450 919 raft_consensus.cc:2466] T 7442c6ff699141e1aff208ebdc9d192a P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:18.515605 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9b7f2e6548043a7865e25387cdd06c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:18.515691 904 tablet_service.cc:1515] Processing DeleteTablet for tablet 8381305e4c4148808803e255b8a43a90 with delete_type TABLET_DATA_DELETED (replaced by tablet 8e49863da5814e43b442e6081ee911b9) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:18.515936 894 tablet_service.cc:1468] Processing CreateTablet for tablet 4cde5f4f02044b7bbd7d43b49aac4ae8 (DEFAULT_TABLE table=default.Table_c931e9dfbef04fd3b49aa4f43db3d981 [id=77ae3376281349e087a3b3cee41f2a09]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:18.516364 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cde5f4f02044b7bbd7d43b49aac4ae8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:18.516571 904 tablet_service.cc:1515] Processing DeleteTablet for tablet 6cf09e8f299548dba2cce16d7b565bad with delete_type TABLET_DATA_DELETED (replaced by tablet 0e4009963cfe42d6ba164fc828897f09) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:18.517036 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c95798f0daf4437db18a291c4464522e" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:18.517234 1711 raft_consensus.cc:491] T ef58f477ebeb49639ed7d071e1a62c10 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:18.517402 1711 raft_consensus.cc:513] T ef58f477ebeb49639ed7d071e1a62c10 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:18.518174 904 tablet_service.cc:1515] Processing DeleteTablet for tablet 7a2f6e2c7ee2402ea310c058e046416c with delete_type TABLET_DATA_DELETED (replaced by tablet 5f5396045979450193d461ba1334a5c9) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:18.518240 1711 leader_election.cc:290] T ef58f477ebeb49639ed7d071e1a62c10 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:18.519042 904 tablet_service.cc:1515] Processing DeleteTablet for tablet 66a48a8ebe154b2cac89c234eaa378ff with delete_type TABLET_DATA_DELETED (replaced by tablet 15e4d972e8254cf899c3e0b71b6e2bd9) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:18.520114 1035 tablet_service.cc:1468] Processing CreateTablet for tablet df01d5df9fc2424eaf9de4e68e781803 (DEFAULT_TABLE table=default.Table_d56ff20d129b440cbab7b15c4af6fa46 [id=c6acca4c20ba4dfaa385fc3c124507a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:18.520561 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df01d5df9fc2424eaf9de4e68e781803. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:18.520545 889 tablet_service.cc:1515] Processing DeleteTablet for tablet b81a12b38ac64efabd44aeb192d181a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:18 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:18.520864 1306 tablet_replica.cc:331] T 8381305e4c4148808803e255b8a43a90 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:18.520979 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 879fa66e55544118aafe504587393c66 (DEFAULT_TABLE table=default.Table_b362441111a241e19cc1a09bdeefea49 [id=f1d75be759db40ebb72ff32560f1d038]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:18.521085 1306 raft_consensus.cc:2241] T 8381305e4c4148808803e255b8a43a90 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:18.521293 1306 raft_consensus.cc:2270] T 8381305e4c4148808803e255b8a43a90 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:18.521528 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 6c0abbfc8d904dce989ce51961e4237d (DEFAULT_TABLE table=default.Table_8a88621295cf403da5cf0fc50afbb345 [id=f0dfb3811e744bd4ba6d59195cebde6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:18.521878 1306 ts_tablet_manager.cc:1905] T 8381305e4c4148808803e255b8a43a90 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:18.521991 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 5f52698fb56d472d9e91fb10cb10c69a (DEFAULT_TABLE table=default.Table_83cf2d86a1284544ba2e7aa9ebb66bb7 [id=264d1a223e7d43c8a7a661db6722cb17]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:18.522455 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 3239e58531d445fa80165cf678bc277c (DEFAULT_TABLE table=default.Table_9163155cb611406a8ce0d5ae600a15c8 [id=aa9c7768ab244d60bedd5ea94cc5e94e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:18.523092 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f52698fb56d472d9e91fb10cb10c69a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:18.523145 1033 tablet_service.cc:1468] Processing CreateTablet for tablet 31614db4c53a4f3c96fd190d326ce601 (DEFAULT_TABLE table=default.Table_e59cac5d3f0a4dd6b9846e7449b60005 [id=fb3be765c8e84e8cb4823153f82b374c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:18.523535 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31614db4c53a4f3c96fd190d326ce601. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:18.528673 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3239e58531d445fa80165cf678bc277c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:18.531566 1306 ts_tablet_manager.cc:1918] T 8381305e4c4148808803e255b8a43a90 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:18.531694 1306 log.cc:1199] T 8381305e4c4148808803e255b8a43a90 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/8381305e4c4148808803e255b8a43a90
I20250901 01:52:18.532099 1306 ts_tablet_manager.cc:1939] T 8381305e4c4148808803e255b8a43a90 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:18.532727 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c0abbfc8d904dce989ce51961e4237d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:18.533883 1033 tablet_service.cc:1468] Processing CreateTablet for tablet 5afae316ff164413be0d25d4bc0f58dc (DEFAULT_TABLE table=default.Table_1da54cdf1dfd4c0fb5f2538e26f17ad9 [id=8a1a77329ab740dda2d6a5d78bbff906]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:18.534327 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5afae316ff164413be0d25d4bc0f58dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:18.536917 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 879fa66e55544118aafe504587393c66. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:18.539340 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 9fbd44641d9e43c3b592e388e4adf350 (DEFAULT_TABLE table=default.Table_cbeebbc633854c008e07b7d39d2024ee [id=6fdd3d817dc74551923021bd716ba58b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:18.539785 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fbd44641d9e43c3b592e388e4adf350. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:18.540102 1306 tablet_replica.cc:331] T 66a48a8ebe154b2cac89c234eaa378ff P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:18.540335 1306 raft_consensus.cc:2241] T 66a48a8ebe154b2cac89c234eaa378ff P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:18.540549 1306 raft_consensus.cc:2270] T 66a48a8ebe154b2cac89c234eaa378ff P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:18.549365 1306 ts_tablet_manager.cc:1905] T 66a48a8ebe154b2cac89c234eaa378ff P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:18.550221 858 leader_election.cc:336] T ef58f477ebeb49639ed7d071e1a62c10 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:18.550887 1036 tablet_service.cc:1468] Processing CreateTablet for tablet abf6e3be60f74872b2308dc95bf9cd37 (DEFAULT_TABLE table=default.Table_fea398de1f1948809aebf95c387f4b17 [id=6975ac6754db4664a168e25ad61db164]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:18.551317 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abf6e3be60f74872b2308dc95bf9cd37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:18.557857 898 tablet_service.cc:1468] Processing CreateTablet for tablet 4902062977ea42149e3466e999117322 (DEFAULT_TABLE table=default.Table_fb5ee5a5a9514ddaac740c194a820fd2 [id=48b7b856dca841eba100ba8acf02b7e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:18.558413 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4902062977ea42149e3466e999117322. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:18.561264 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 6ba88af177874544b764cc32420e7ac9 (DEFAULT_TABLE table=default.Table_97c44ea002094df8a6af3dccd0099c39 [id=305a9d97cb394d61821c13807ebebc17]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:18.561627 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet a59ac57aebe24860aa02f53ce0485910 with delete_type TABLET_DATA_DELETED (replaced by tablet 93bcac1fd95d44dc927abfca81692921) from {username='slave'} at 127.0.0.1:44246
W20250901 01:52:18.562826 992 leader_election.cc:336] T 58c0e7a2e90e48528bc2bc7ec889f608 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:18.562992 992 leader_election.cc:304] T 58c0e7a2e90e48528bc2bc7ec889f608 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:18.563261 1025 tablet_service.cc:1468] Processing CreateTablet for tablet d1f2d98712ff4ea6aec665512de068bd (DEFAULT_TABLE table=default.Table_82d91a1ba2eb4265896824028dec8d3a [id=3f7624d2901445e39155c12fa68e34e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:18.563675 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet e25923b2daf9429683b2d44a75a56189 with delete_type TABLET_DATA_DELETED (replaced by tablet 2973b59bb1df41d18d9a5b6a29ae336f) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:18.564127 1026 tablet_service.cc:1468] Processing CreateTablet for tablet cb7d7e1796c0456f8de98fb812b9e374 (DEFAULT_TABLE table=default.Table_c4148158ed7b48a3946559ba978e72fd [id=327c5a63904540508142ab600c308757]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:18.568964 1712 raft_consensus.cc:2747] T 58c0e7a2e90e48528bc2bc7ec889f608 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:18.569445 992 leader_election.cc:336] T 4410da9e95454ecf95faf80221b0a71c P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:18.569702 992 leader_election.cc:336] T 186ec810d02e4748aad15228b5866ac1 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:18.571607 454 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_7c23fc28bbae4efe982c240b87f40956" } modify_external_catalogs: true
I20250901 01:52:18.571875 454 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_7c23fc28bbae4efe982c240b87f40956" } modify_external_catalogs: true
W20250901 01:52:18.574782 859 leader_election.cc:343] T c95798f0daf4437db18a291c4464522e P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:18.574970 859 leader_election.cc:304] T c95798f0daf4437db18a291c4464522e P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:18.575601 1711 raft_consensus.cc:2747] T c95798f0daf4437db18a291c4464522e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:18.576368 1306 ts_tablet_manager.cc:1918] T 66a48a8ebe154b2cac89c234eaa378ff P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:18.577625 992 leader_election.cc:304] T 186ec810d02e4748aad15228b5866ac1 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:18.582600 1306 log.cc:1199] T 66a48a8ebe154b2cac89c234eaa378ff P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/66a48a8ebe154b2cac89c234eaa378ff
I20250901 01:52:18.583344 1306 ts_tablet_manager.cc:1939] T 66a48a8ebe154b2cac89c234eaa378ff P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:18.587112 1712 raft_consensus.cc:2802] T 186ec810d02e4748aad15228b5866ac1 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:18.587255 1712 raft_consensus.cc:491] T 186ec810d02e4748aad15228b5866ac1 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:18.587419 1712 raft_consensus.cc:3058] T 186ec810d02e4748aad15228b5866ac1 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:18.591849 1712 raft_consensus.cc:513] T 186ec810d02e4748aad15228b5866ac1 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:18.561909 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ba88af177874544b764cc32420e7ac9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:18.592916 1712 leader_election.cc:290] T 186ec810d02e4748aad15228b5866ac1 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:18.594014 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1f2d98712ff4ea6aec665512de068bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:18.595839 1712 consensus_queue.cc:1035] T f65ec84958eb488d8b0f812b59b5b3d5 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:18.596553 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb7d7e1796c0456f8de98fb812b9e374. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:18.600436 992 leader_election.cc:304] T fcb7b5b7d63d4f0b906fea97cb548de2 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:18.601503 1712 raft_consensus.cc:2802] T fcb7b5b7d63d4f0b906fea97cb548de2 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:18.601606 1712 raft_consensus.cc:491] T fcb7b5b7d63d4f0b906fea97cb548de2 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:18.601941 1711 raft_consensus.cc:491] T 629db80a8ab444e582628578155638fd P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:18.602136 1711 raft_consensus.cc:513] T 629db80a8ab444e582628578155638fd P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:18.602955 1711 leader_election.cc:290] T 629db80a8ab444e582628578155638fd P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:18.601733 1712 raft_consensus.cc:3058] T fcb7b5b7d63d4f0b906fea97cb548de2 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
W20250901 01:52:18.606699 858 leader_election.cc:336] T 629db80a8ab444e582628578155638fd P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:18.608333 1712 raft_consensus.cc:513] T fcb7b5b7d63d4f0b906fea97cb548de2 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:18.609163 1712 leader_election.cc:290] T fcb7b5b7d63d4f0b906fea97cb548de2 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:18.609515 992 leader_election.cc:304] T 7442c6ff699141e1aff208ebdc9d192a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
W20250901 01:52:18.609827 992 leader_election.cc:343] T 3b18a7084727490380d26b3485e4e799 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 3b18a7084727490380d26b3485e4e799
I20250901 01:52:18.610723 894 tablet_service.cc:1515] Processing DeleteTablet for tablet a59ac57aebe24860aa02f53ce0485910 with delete_type TABLET_DATA_DELETED (replaced by tablet 93bcac1fd95d44dc927abfca81692921) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:18.611274 900 tablet_service.cc:1468] Processing CreateTablet for tablet df01d5df9fc2424eaf9de4e68e781803 (DEFAULT_TABLE table=default.Table_d56ff20d129b440cbab7b15c4af6fa46 [id=c6acca4c20ba4dfaa385fc3c124507a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:18.611765 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df01d5df9fc2424eaf9de4e68e781803. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:18.611783 1710 raft_consensus.cc:2802] T 7442c6ff699141e1aff208ebdc9d192a P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:18.611938 1710 raft_consensus.cc:695] T 7442c6ff699141e1aff208ebdc9d192a P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:18.611918 894 tablet_service.cc:1468] Processing CreateTablet for tablet abf6e3be60f74872b2308dc95bf9cd37 (DEFAULT_TABLE table=default.Table_fea398de1f1948809aebf95c387f4b17 [id=6975ac6754db4664a168e25ad61db164]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:18.612264 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abf6e3be60f74872b2308dc95bf9cd37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:18.612260 1710 consensus_queue.cc:237] T 7442c6ff699141e1aff208ebdc9d192a P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:18.619673 1306 tablet_replica.cc:331] T a59ac57aebe24860aa02f53ce0485910 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:18.619942 1306 raft_consensus.cc:2241] T a59ac57aebe24860aa02f53ce0485910 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:18.620193 1306 raft_consensus.cc:2270] T a59ac57aebe24860aa02f53ce0485910 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:18.620800 1306 ts_tablet_manager.cc:1905] T a59ac57aebe24860aa02f53ce0485910 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:18.628186 1058 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef58f477ebeb49639ed7d071e1a62c10" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:18.628618 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "629db80a8ab444e582628578155638fd" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:18.628931 1151 tablet_bootstrap.cc:492] T 3df908c3096a4df7bddd8ac88b49cfe0 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:18.629081 1151 ts_tablet_manager.cc:1397] T 3df908c3096a4df7bddd8ac88b49cfe0 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.217s user 0.013s sys 0.029s
W20250901 01:52:18.629614 992 leader_election.cc:336] T 7442c6ff699141e1aff208ebdc9d192a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:18.629967 1151 raft_consensus.cc:357] T 3df908c3096a4df7bddd8ac88b49cfe0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:18.630167 1151 raft_consensus.cc:383] T 3df908c3096a4df7bddd8ac88b49cfe0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250901 01:52:18.630054 992 leader_election.cc:336] T fcb7b5b7d63d4f0b906fea97cb548de2 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:18.630254 1151 raft_consensus.cc:738] T 3df908c3096a4df7bddd8ac88b49cfe0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:18.630618 1151 consensus_queue.cc:260] T 3df908c3096a4df7bddd8ac88b49cfe0 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:18.631181 1306 ts_tablet_manager.cc:1918] T a59ac57aebe24860aa02f53ce0485910 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:18.631412 1151 ts_tablet_manager.cc:1428] T 3df908c3096a4df7bddd8ac88b49cfe0 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20250901 01:52:18.642381 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet a59ac57aebe24860aa02f53ce0485910 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a59ac57aebe24860aa02f53ce0485910 already in progress: creating tablet
W20250901 01:52:18.642831 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet e25923b2daf9429683b2d44a75a56189 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e25923b2daf9429683b2d44a75a56189 already in progress: creating tablet
I20250901 01:52:18.639410 1151 tablet_bootstrap.cc:492] T d60d571b2c2840a2b20629bb832d1309 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:18.645148 1151 tablet_bootstrap.cc:654] T d60d571b2c2840a2b20629bb832d1309 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:18.649751 992 leader_election.cc:336] T 3b18a7084727490380d26b3485e4e799 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:18.649914 992 leader_election.cc:304] T 3b18a7084727490380d26b3485e4e799 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:18.650792 1712 raft_consensus.cc:2747] T 3b18a7084727490380d26b3485e4e799 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:18.651702 1151 tablet_bootstrap.cc:492] T d60d571b2c2840a2b20629bb832d1309 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:18.651908 1151 ts_tablet_manager.cc:1397] T d60d571b2c2840a2b20629bb832d1309 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.000s
I20250901 01:52:18.652907 1151 raft_consensus.cc:357] T d60d571b2c2840a2b20629bb832d1309 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:18.653116 1151 raft_consensus.cc:383] T d60d571b2c2840a2b20629bb832d1309 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:18.653218 1151 raft_consensus.cc:738] T d60d571b2c2840a2b20629bb832d1309 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:18.653498 1151 consensus_queue.cc:260] T d60d571b2c2840a2b20629bb832d1309 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:18.657528 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_dfb6e260190a40df8db77152b962cb8e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:18.657749 1033 tablet_service.cc:1468] Processing CreateTablet for tablet fa7fb88ded21432a8757f37ec3db5e71 (DEFAULT_TABLE table=default.Table_22d8d38d0b624912a0d8968fb764420e [id=808a8849c99e485b8be7cd17397874ec]), partition=RANGE (key) PARTITION UNBOUNDED
W20250901 01:52:18.658093 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dfb6e260190a40df8db77152b962cb8e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:18.658931 1038 tablet_service.cc:1468] Processing CreateTablet for tablet b85d64e02cc646e380581ac9f59dee9e (DEFAULT_TABLE table=default.Table_7be95f26c04746828f38a19ddb67afc0 [id=2ec1e340c6924cefa540916968475e1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:18.659618 1023 tablet_service.cc:1468] Processing CreateTablet for tablet becbaae4fd7e44faa679e9479465a7c4 (DEFAULT_TABLE table=default.Table_0e73f80424cd48408f18c2eff1411706 [id=47c13af47c944d0fa79dee115033265d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:18.660225 1021 tablet_service.cc:1468] Processing CreateTablet for tablet b164b4a67a9143ceaff69b0ce49d63fe (DEFAULT_TABLE table=default.Table_da2c70a8f2fe49a0a8184e88c0a9e0ea [id=c1b1e9e3876244f68d31c63ce7bdebf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:18.661063 1030 tablet_service.cc:1468] Processing CreateTablet for tablet eb19e3a424324a24824f6caaf5848952 (DEFAULT_TABLE table=default.Table_3b46a0a1f77a4cea842e19b3c89de475 [id=fac15e527f974e228cf2e53026fba40e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:18.661500 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 74b709cb5ec64006b2f12bead8217d80 (DEFAULT_TABLE table=default.Table_95884295301d476fb3e03054727f9cf1 [id=1aa18a2513db4e99a72572846527663c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:18.661933 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 06d5eeb70e664f73bf643922d445bc7a (DEFAULT_TABLE table=default.Table_2f266fe78a3c41c2ab134faad48a377d [id=86e2bc0201e04503a1f60452605eaf27]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:18.654804 922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "186ec810d02e4748aad15228b5866ac1" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:18.663024 1306 log.cc:1199] T a59ac57aebe24860aa02f53ce0485910 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/a59ac57aebe24860aa02f53ce0485910
I20250901 01:52:18.663074 922 raft_consensus.cc:3058] T 186ec810d02e4748aad15228b5866ac1 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:18.663481 1306 ts_tablet_manager.cc:1939] T a59ac57aebe24860aa02f53ce0485910 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:18.656797 921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fcb7b5b7d63d4f0b906fea97cb548de2" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
W20250901 01:52:18.656844 859 leader_election.cc:343] T ef58f477ebeb49639ed7d071e1a62c10 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
W20250901 01:52:18.675029 992 consensus_peers.cc:489] T f65ec84958eb488d8b0f812b59b5b3d5 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250901 01:52:18.683454 440 catalog_manager.cc:4671] TS 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): CreateTablet RPC failed for tablet 3c1fd0820d8a4995abe027f685da7f7b: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:18.687700 1151 ts_tablet_manager.cc:1428] T d60d571b2c2840a2b20629bb832d1309 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.036s user 0.003s sys 0.000s
W20250901 01:52:18.689508 449 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet eac50cbae691416e903cb2cbc907f96d consensus state has no leader
I20250901 01:52:18.691045 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_da6f34d486984dedbc970080cf14482b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:18.691589 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_da6f34d486984dedbc970080cf14482b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250901 01:52:18.693445 992 leader_election.cc:336] T fcb7b5b7d63d4f0b906fea97cb548de2 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:18.693687 992 leader_election.cc:336] T 186ec810d02e4748aad15228b5866ac1 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:18.696175 1031 tablet_service.cc:1468] Processing CreateTablet for tablet b13afacd37474781877b9f2cd20b313a (DEFAULT_TABLE table=default.Table_36f5f8854ed14d1ea48eb18865d00659 [id=bdf8a102ae7d43228173d7b82507d80d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:18.702543 904 tablet_service.cc:1468] Processing CreateTablet for tablet 3239e58531d445fa80165cf678bc277c (DEFAULT_TABLE table=default.Table_9163155cb611406a8ce0d5ae600a15c8 [id=aa9c7768ab244d60bedd5ea94cc5e94e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:18.703125 904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3239e58531d445fa80165cf678bc277c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:18.703778 921 raft_consensus.cc:3058] T fcb7b5b7d63d4f0b906fea97cb548de2 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:18.704202 859 leader_election.cc:304] T ef58f477ebeb49639ed7d071e1a62c10 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:18.705554 446 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_ca246ea8f871412d949125b3714d8b67" } modify_external_catalogs: true
I20250901 01:52:18.705802 446 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_ca246ea8f871412d949125b3714d8b67" } modify_external_catalogs: true
I20250901 01:52:18.709949 897 tablet_service.cc:1468] Processing CreateTablet for tablet 5afae316ff164413be0d25d4bc0f58dc (DEFAULT_TABLE table=default.Table_1da54cdf1dfd4c0fb5f2538e26f17ad9 [id=8a1a77329ab740dda2d6a5d78bbff906]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:18.711709 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5afae316ff164413be0d25d4bc0f58dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:18.713303 900 tablet_service.cc:1468] Processing CreateTablet for tablet 879fa66e55544118aafe504587393c66 (DEFAULT_TABLE table=default.Table_b362441111a241e19cc1a09bdeefea49 [id=f1d75be759db40ebb72ff32560f1d038]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:18.713784 1711 raft_consensus.cc:2747] T ef58f477ebeb49639ed7d071e1a62c10 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:18.716560 889 tablet_service.cc:1468] Processing CreateTablet for tablet 5f52698fb56d472d9e91fb10cb10c69a (DEFAULT_TABLE table=default.Table_83cf2d86a1284544ba2e7aa9ebb66bb7 [id=264d1a223e7d43c8a7a661db6722cb17]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:18.720096 903 tablet_service.cc:1468] Processing CreateTablet for tablet 31614db4c53a4f3c96fd190d326ce601 (DEFAULT_TABLE table=default.Table_e59cac5d3f0a4dd6b9846e7449b60005 [id=fb3be765c8e84e8cb4823153f82b374c]), partition=RANGE (key) PARTITION UNBOUNDED
W20250901 01:52:18.721158 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 50e3da53433f4c14b57db8a733fa53bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 50e3da53433f4c14b57db8a733fa53bc already in progress: creating tablet
I20250901 01:52:18.718701 894 tablet_service.cc:1468] Processing CreateTablet for tablet 6c0abbfc8d904dce989ce51961e4237d (DEFAULT_TABLE table=default.Table_8a88621295cf403da5cf0fc50afbb345 [id=f0dfb3811e744bd4ba6d59195cebde6f]), partition=RANGE (key) PARTITION UNBOUNDED
W20250901 01:52:18.721829 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 7a607fac57994001b2304e5d81c9decc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7a607fac57994001b2304e5d81c9decc already in progress: creating tablet
W20250901 01:52:18.722288 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet af2c4bcbb00741f789edee109252d8cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet af2c4bcbb00741f789edee109252d8cb already in progress: creating tablet
W20250901 01:52:18.722330 455 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4fca1435d82447f289fda4035aa2ed89 has no consensus state
W20250901 01:52:18.722820 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 6cf09e8f299548dba2cce16d7b565bad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6cf09e8f299548dba2cce16d7b565bad already in progress: creating tablet
W20250901 01:52:18.723122 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 7a2f6e2c7ee2402ea310c058e046416c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7a2f6e2c7ee2402ea310c058e046416c already in progress: creating tablet
I20250901 01:52:18.723466 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 66a48a8ebe154b2cac89c234eaa378ff (table default.Table_ce16a1b94f714c2699ee6cd916bb0215 [id=551e2f9616d84026b96b6c22507f29de]) successfully deleted
W20250901 01:52:18.723737 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet b81a12b38ac64efabd44aeb192d181a0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b81a12b38ac64efabd44aeb192d181a0 already in progress: creating tablet
I20250901 01:52:18.723821 922 raft_consensus.cc:2466] T 186ec810d02e4748aad15228b5866ac1 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:18.724327 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet a59ac57aebe24860aa02f53ce0485910 (table default.Table_c2829420d635461ea141ba803c71bdf9 [id=b8f69313439543be9c11eea88dfa69c5]) successfully deleted
I20250901 01:52:18.730495 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_a96f01d4ce3c4694a635575ffaa4f02d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:18.731071 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a96f01d4ce3c4694a635575ffaa4f02d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:18.731272 890 tablet_service.cc:1515] Processing DeleteTablet for tablet e25923b2daf9429683b2d44a75a56189 with delete_type TABLET_DATA_DELETED (replaced by tablet 2973b59bb1df41d18d9a5b6a29ae336f) from {username='slave'} at 127.0.0.1:52846
W20250901 01:52:18.731503 859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:39447, user_credentials={real_user=slave}} blocked reactor thread for 50444.9us
W20250901 01:52:18.731949 859 leader_election.cc:343] T 629db80a8ab444e582628578155638fd P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:18.732851 455 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_28ee2277a7d442ec97b05865b2661076" } new_table_name: "default.Table_1cd3f3fb6b244b3f8796bfca9f7622c8" modify_external_catalogs: true
I20250901 01:52:18.734073 899 tablet_service.cc:1468] Processing CreateTablet for tablet cb7d7e1796c0456f8de98fb812b9e374 (DEFAULT_TABLE table=default.Table_c4148158ed7b48a3946559ba978e72fd [id=327c5a63904540508142ab600c308757]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:18.734527 892 tablet_service.cc:1468] Processing CreateTablet for tablet fa7fb88ded21432a8757f37ec3db5e71 (DEFAULT_TABLE table=default.Table_22d8d38d0b624912a0d8968fb764420e [id=808a8849c99e485b8be7cd17397874ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:18.734807 452 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 05630a72ddae4de0b890dece2feacc31
I20250901 01:52:18.734970 901 tablet_service.cc:1468] Processing CreateTablet for tablet d1f2d98712ff4ea6aec665512de068bd (DEFAULT_TABLE table=default.Table_82d91a1ba2eb4265896824028dec8d3a [id=3f7624d2901445e39155c12fa68e34e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:18.735160 896 tablet_service.cc:1468] Processing CreateTablet for tablet 6ba88af177874544b764cc32420e7ac9 (DEFAULT_TABLE table=default.Table_97c44ea002094df8a6af3dccd0099c39 [id=305a9d97cb394d61821c13807ebebc17]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:18.736377 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 879fa66e55544118aafe504587393c66. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:18.737313 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 48a211e174054866b030cdfb0cf947f4
I20250901 01:52:18.737396 895 tablet_service.cc:1468] Processing CreateTablet for tablet 06d5eeb70e664f73bf643922d445bc7a (DEFAULT_TABLE table=default.Table_2f266fe78a3c41c2ab134faad48a377d [id=86e2bc0201e04503a1f60452605eaf27]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:18.737911 454 master_service.cc:948] ReplaceTablet: received request to replace tablet 670d90a37b934579b353bb4e4820127b from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:18.732977 902 tablet_service.cc:1468] Processing CreateTablet for tablet 9fbd44641d9e43c3b592e388e4adf350 (DEFAULT_TABLE table=default.Table_cbeebbc633854c008e07b7d39d2024ee [id=6fdd3d817dc74551923021bd716ba58b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:18.732688 859 leader_election.cc:304] T 629db80a8ab444e582628578155638fd P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:18.740849 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa7fb88ded21432a8757f37ec3db5e71. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:18.741009 452 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_fb5ee5a5a9514ddaac740c194a820fd2" } modify_external_catalogs: true
I20250901 01:52:18.741199 452 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_fb5ee5a5a9514ddaac740c194a820fd2" } modify_external_catalogs: true
I20250901 01:52:18.742063 446 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 75e7a90815e449c39f386901a504143c
I20250901 01:52:18.742784 446 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_123e5dc85bca4350b541ce0f0e9e7d3e" } new_table_name: "default.Table_74a88a4668244f6294b0fd18f24dc677" modify_external_catalogs: true
I20250901 01:52:18.744431 449 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_c2829420d635461ea141ba803c71bdf9" } new_table_name: "default.Table_b8f8ee74648c406eaf589d3ab80934db" modify_external_catalogs: true
I20250901 01:52:18.745074 890 tablet_service.cc:1468] Processing CreateTablet for tablet 74b709cb5ec64006b2f12bead8217d80 (DEFAULT_TABLE table=default.Table_95884295301d476fb3e03054727f9cf1 [id=1aa18a2513db4e99a72572846527663c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:18.745432 450 master_service.cc:948] ReplaceTablet: received request to replace tablet 1a60c10e42104502ac2d3b80b8126ceb from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:18.746915 893 tablet_service.cc:1468] Processing CreateTablet for tablet eb19e3a424324a24824f6caaf5848952 (DEFAULT_TABLE table=default.Table_3b46a0a1f77a4cea842e19b3c89de475 [id=fac15e527f974e228cf2e53026fba40e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:18.769157 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 670d90a37b934579b353bb4e4820127b
I20250901 01:52:18.769662 454 catalog_manager.cc:6652] ReplaceTablet: tablet 670d90a37b934579b353bb4e4820127b deleted and replaced by tablet 75a703c8e20a4a2fa0ab6ec58d41ef92
I20250901 01:52:18.770006 454 master_service.cc:948] ReplaceTablet: received request to replace tablet 824229183e024feaa975c7f0387c9c1a from {username='slave'} at 127.0.0.1:57810
W20250901 01:52:18.771378 449 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 93bcac1fd95d44dc927abfca81692921 has no consensus state
I20250901 01:52:18.771665 449 master_service.cc:948] ReplaceTablet: received request to replace tablet 75e7a90815e449c39f386901a504143c from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:18.772310 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_622c258d8abf4619a3919206fe82e6e7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:18.772923 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_622c258d8abf4619a3919206fe82e6e7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:18.775871 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_9e04f976a93f4d30baccf90fa3622dfe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:18.781862 1151 tablet_bootstrap.cc:492] T 8381305e4c4148808803e255b8a43a90 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:18.782375 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet becbaae4fd7e44faa679e9479465a7c4. 1 dirs total, 0 dirs full, 0 dirs failed
W20250901 01:52:18.783116 455 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ad45c90ba0da43c9860745e534664a98 consensus state has no leader
I20250901 01:52:18.783746 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b164b4a67a9143ceaff69b0ce49d63fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:18.784659 451 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 629db80a8ab444e582628578155638fd (table default.Table_f0e56dba1e8a4ed49a760e204c1ff96e [id=c3865e058daa4cea9a89f5046fca712a]). Expected version 1 got 0
I20250901 01:52:18.785117 1036 tablet_service.cc:1515] Processing DeleteTablet for tablet 48a211e174054866b030cdfb0cf947f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:18 UTC) from {username='slave'} at 127.0.0.1:44246
W20250901 01:52:18.786943 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9e04f976a93f4d30baccf90fa3622dfe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:18.786957 451 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 7cc1efcb6bff4aba8015d2d3b7f4c923 (table default.Table_5610feff8a344dc9be4bc1f0ad8ddd51 [id=78b00856a9b747c19f390b4fdae8ddcf]). Expected version 3 got 0
I20250901 01:52:18.788213 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet 05630a72ddae4de0b890dece2feacc31 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:18 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:18.789070 992 leader_election.cc:304] T 186ec810d02e4748aad15228b5866ac1 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:18.789659 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06d5eeb70e664f73bf643922d445bc7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:18.791195 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet c27bb23c150f4565a1d0ba8f231cc386
W20250901 01:52:18.793053 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 48a211e174054866b030cdfb0cf947f4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 48a211e174054866b030cdfb0cf947f4 already in progress: creating tablet
W20250901 01:52:18.793335 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 05630a72ddae4de0b890dece2feacc31 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 05630a72ddae4de0b890dece2feacc31 already in progress: creating tablet
W20250901 01:52:18.797006 446 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 61ddcf86c9f34726939cc116434f0822 has no consensus state
I20250901 01:52:18.798373 1669 raft_consensus.cc:2802] T 186ec810d02e4748aad15228b5866ac1 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:18.800499 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b13afacd37474781877b9f2cd20b313a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:18.803088 1711 raft_consensus.cc:2747] T 629db80a8ab444e582628578155638fd P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:18.797778 886 tablet_service.cc:1468] Processing CreateTablet for tablet b164b4a67a9143ceaff69b0ce49d63fe (DEFAULT_TABLE table=default.Table_da2c70a8f2fe49a0a8184e88c0a9e0ea [id=c1b1e9e3876244f68d31c63ce7bdebf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:18.804559 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 75e7a90815e449c39f386901a504143c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:18 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:18.804245 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet 670d90a37b934579b353bb4e4820127b with delete_type TABLET_DATA_DELETED (replaced by tablet 75a703c8e20a4a2fa0ab6ec58d41ef92) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:18.807914 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06d5eeb70e664f73bf643922d445bc7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:18.796166 887 tablet_service.cc:1468] Processing CreateTablet for tablet becbaae4fd7e44faa679e9479465a7c4 (DEFAULT_TABLE table=default.Table_0e73f80424cd48408f18c2eff1411706 [id=47c13af47c944d0fa79dee115033265d]), partition=RANGE (key) PARTITION UNBOUNDED
W20250901 01:52:18.809690 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 75e7a90815e449c39f386901a504143c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 75e7a90815e449c39f386901a504143c already in progress: creating tablet
W20250901 01:52:18.810038 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 670d90a37b934579b353bb4e4820127b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 670d90a37b934579b353bb4e4820127b already in progress: creating tablet
I20250901 01:52:18.811733 1714 tablet.cc:1722] T 7cc1efcb6bff4aba8015d2d3b7f4c923 P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20250901 01:52:18.813509 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet e25923b2daf9429683b2d44a75a56189 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e25923b2daf9429683b2d44a75a56189 already in progress: creating tablet
I20250901 01:52:18.818607 1727 tablet.cc:1722] T 7cc1efcb6bff4aba8015d2d3b7f4c923 P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250901 01:52:18.818634 885 tablet_service.cc:1468] Processing CreateTablet for tablet b13afacd37474781877b9f2cd20b313a (DEFAULT_TABLE table=default.Table_36f5f8854ed14d1ea48eb18865d00659 [id=bdf8a102ae7d43228173d7b82507d80d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:18.819828 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb19e3a424324a24824f6caaf5848952. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:18.820636 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74b709cb5ec64006b2f12bead8217d80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:18.820753 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet becbaae4fd7e44faa679e9479465a7c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:18.820709 1669 raft_consensus.cc:695] T 186ec810d02e4748aad15228b5866ac1 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:18.821063 1669 consensus_queue.cc:237] T 186ec810d02e4748aad15228b5866ac1 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:18.822824 1035 tablet_service.cc:1515] Processing DeleteTablet for tablet c27bb23c150f4565a1d0ba8f231cc386 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:18 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:18.824543 1332 tablet_replica.cc:331] T c27bb23c150f4565a1d0ba8f231cc386 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
W20250901 01:52:18.824806 1736 log.cc:927] Time spent T f65ec84958eb488d8b0f812b59b5b3d5 P 00c282f779c1477d920f24dc0dc16628: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20250901 01:52:18.825026 1332 raft_consensus.cc:2241] T c27bb23c150f4565a1d0ba8f231cc386 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:18.825416 1332 raft_consensus.cc:2270] T c27bb23c150f4565a1d0ba8f231cc386 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:18.829278 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb19e3a424324a24824f6caaf5848952. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:18.831506 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b85d64e02cc646e380581ac9f59dee9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:18.833719 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa7fb88ded21432a8757f37ec3db5e71. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:18.840641 1332 ts_tablet_manager.cc:1905] T c27bb23c150f4565a1d0ba8f231cc386 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:18.844784 888 tablet_service.cc:1468] Processing CreateTablet for tablet b85d64e02cc646e380581ac9f59dee9e (DEFAULT_TABLE table=default.Table_7be95f26c04746828f38a19ddb67afc0 [id=2ec1e340c6924cefa540916968475e1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:18.852217 1151 tablet_bootstrap.cc:654] T 8381305e4c4148808803e255b8a43a90 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:18.866127 1332 ts_tablet_manager.cc:1918] T c27bb23c150f4565a1d0ba8f231cc386 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:18.866297 1332 log.cc:1199] T c27bb23c150f4565a1d0ba8f231cc386 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/c27bb23c150f4565a1d0ba8f231cc386
I20250901 01:52:18.867086 1332 ts_tablet_manager.cc:1939] T c27bb23c150f4565a1d0ba8f231cc386 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:18.867517 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f52698fb56d472d9e91fb10cb10c69a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:18.868067 441 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet c27bb23c150f4565a1d0ba8f231cc386 (table default.Table_e066e678c3a8406583dbe851d3424bac [id=f70c2bc07ea245bea6293c3742dfcf94]) successfully deleted
I20250901 01:52:18.869372 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c0abbfc8d904dce989ce51961e4237d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:18.874531 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_f05ff31088d34354b72333af6d6c7235"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:18.875155 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f05ff31088d34354b72333af6d6c7235 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:18.875638 446 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 7cc1efcb6bff4aba8015d2d3b7f4c923 (table default.Table_5610feff8a344dc9be4bc1f0ad8ddd51 [id=78b00856a9b747c19f390b4fdae8ddcf]). Expected version 3 got 0
I20250901 01:52:18.876009 446 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet f65ec84958eb488d8b0f812b59b5b3d5 (table default.Table_a3ff88e352884abdba91e6d03acbae22 [id=d1cd5288671540c0a95f7a037363014e]). Expected version 2 got 0
I20250901 01:52:18.876703 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_d82e47b1bd8743649c6f3d31da751ed7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:18.877218 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d82e47b1bd8743649c6f3d31da751ed7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:18.883455 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_21b93b5fe2ce49c6bc637d8cab41fc06"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:18.889596 1151 tablet_bootstrap.cc:492] T 8381305e4c4148808803e255b8a43a90 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:18.889762 1151 ts_tablet_manager.cc:1397] T 8381305e4c4148808803e255b8a43a90 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.139s user 0.011s sys 0.002s
I20250901 01:52:18.891126 1151 raft_consensus.cc:357] T 8381305e4c4148808803e255b8a43a90 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:18.891312 1151 raft_consensus.cc:383] T 8381305e4c4148808803e255b8a43a90 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:18.891409 1151 raft_consensus.cc:738] T 8381305e4c4148808803e255b8a43a90 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:18.891590 1151 consensus_queue.cc:260] T 8381305e4c4148808803e255b8a43a90 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:18.892249 1033 tablet_service.cc:1515] Processing DeleteTablet for tablet 58c0e7a2e90e48528bc2bc7ec889f608 with delete_type TABLET_DATA_DELETED (replaced by tablet a6445b29591548be9f9577db1ba0bfa5) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:18.892513 1033 tablet_service.cc:1515] Processing DeleteTablet for tablet 3b18a7084727490380d26b3485e4e799 with delete_type TABLET_DATA_DELETED (replaced by tablet b164b4a67a9143ceaff69b0ce49d63fe) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:18.892635 1151 ts_tablet_manager.cc:1428] T 8381305e4c4148808803e255b8a43a90 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250901 01:52:18.892895 1151 tablet_bootstrap.cc:492] T a59ac57aebe24860aa02f53ce0485910 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:18.894531 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b13afacd37474781877b9f2cd20b313a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:18.894563 1151 tablet_bootstrap.cc:654] T a59ac57aebe24860aa02f53ce0485910 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:18.897153 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 2f28451a3ecc481a958588466f7fec77 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:12 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:18.899080 1332 tablet_replica.cc:331] T 58c0e7a2e90e48528bc2bc7ec889f608 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:18.899271 1332 raft_consensus.cc:2241] T 58c0e7a2e90e48528bc2bc7ec889f608 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:18.899431 1332 raft_consensus.cc:2270] T 58c0e7a2e90e48528bc2bc7ec889f608 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:18.899927 1332 ts_tablet_manager.cc:1905] T 58c0e7a2e90e48528bc2bc7ec889f608 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:18.910669 1332 ts_tablet_manager.cc:1918] T 58c0e7a2e90e48528bc2bc7ec889f608 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:18.912546 1332 log.cc:1199] T 58c0e7a2e90e48528bc2bc7ec889f608 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/58c0e7a2e90e48528bc2bc7ec889f608
I20250901 01:52:18.913497 1332 ts_tablet_manager.cc:1939] T 58c0e7a2e90e48528bc2bc7ec889f608 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:18.916705 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b164b4a67a9143ceaff69b0ce49d63fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:18.917040 1332 tablet_replica.cc:331] T 3b18a7084727490380d26b3485e4e799 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:18.917423 1332 raft_consensus.cc:2241] T 3b18a7084727490380d26b3485e4e799 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:18.917675 1332 raft_consensus.cc:2270] T 3b18a7084727490380d26b3485e4e799 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:18.916651 897 tablet_service.cc:1515] Processing DeleteTablet for tablet 05630a72ddae4de0b890dece2feacc31 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:18 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:18.918373 1332 ts_tablet_manager.cc:1905] T 3b18a7084727490380d26b3485e4e799 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:18.920069 1150 tablet_bootstrap.cc:492] T b73e362321284df1b18ef755dfe930a5 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:18.920240 1150 ts_tablet_manager.cc:1397] T b73e362321284df1b18ef755dfe930a5 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.492s user 0.075s sys 0.192s
I20250901 01:52:18.922178 1150 raft_consensus.cc:357] T b73e362321284df1b18ef755dfe930a5 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:18.922397 1150 raft_consensus.cc:383] T b73e362321284df1b18ef755dfe930a5 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:18.924191 897 tablet_service.cc:1515] Processing DeleteTablet for tablet 48a211e174054866b030cdfb0cf947f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:18 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:18.924782 1151 tablet_bootstrap.cc:492] T a59ac57aebe24860aa02f53ce0485910 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:18.924947 1151 ts_tablet_manager.cc:1397] T a59ac57aebe24860aa02f53ce0485910 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.008s
I20250901 01:52:18.925705 1151 raft_consensus.cc:357] T a59ac57aebe24860aa02f53ce0485910 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:18.925709 1150 raft_consensus.cc:738] T b73e362321284df1b18ef755dfe930a5 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:18.925892 1151 raft_consensus.cc:383] T a59ac57aebe24860aa02f53ce0485910 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:18.926703 1151 raft_consensus.cc:738] T a59ac57aebe24860aa02f53ce0485910 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:18.926929 1151 consensus_queue.cc:260] T a59ac57aebe24860aa02f53ce0485910 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:18.927436 1151 ts_tablet_manager.cc:1428] T a59ac57aebe24860aa02f53ce0485910 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250901 01:52:18.927891 1151 tablet_bootstrap.cc:492] T 5adf6b492640417ca071c94702b48cb9 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:18.929416 904 tablet_service.cc:1515] Processing DeleteTablet for tablet 75e7a90815e449c39f386901a504143c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:18 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:18.929749 1151 tablet_bootstrap.cc:654] T 5adf6b492640417ca071c94702b48cb9 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:18.929733 904 tablet_service.cc:1515] Processing DeleteTablet for tablet c27bb23c150f4565a1d0ba8f231cc386 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:18 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:18.929937 904 tablet_service.cc:1515] Processing DeleteTablet for tablet 66a48a8ebe154b2cac89c234eaa378ff with delete_type TABLET_DATA_DELETED (replaced by tablet 15e4d972e8254cf899c3e0b71b6e2bd9) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:18.930097 904 tablet_service.cc:1515] Processing DeleteTablet for tablet c95798f0daf4437db18a291c4464522e with delete_type TABLET_DATA_DELETED (replaced by tablet 7a2f6e2c7ee2402ea310c058e046416c) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:18.930246 1150 consensus_queue.cc:260] T b73e362321284df1b18ef755dfe930a5 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:18.933887 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b85d64e02cc646e380581ac9f59dee9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:18.941290 1332 ts_tablet_manager.cc:1918] T 3b18a7084727490380d26b3485e4e799 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:18.941457 1332 log.cc:1199] T 3b18a7084727490380d26b3485e4e799 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/3b18a7084727490380d26b3485e4e799
I20250901 01:52:18.941877 1332 ts_tablet_manager.cc:1939] T 3b18a7084727490380d26b3485e4e799 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:18.945456 1151 tablet_bootstrap.cc:492] T 5adf6b492640417ca071c94702b48cb9 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:18.945609 1151 ts_tablet_manager.cc:1397] T 5adf6b492640417ca071c94702b48cb9 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20250901 01:52:18.946458 1151 raft_consensus.cc:357] T 5adf6b492640417ca071c94702b48cb9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:18.947678 1151 raft_consensus.cc:383] T 5adf6b492640417ca071c94702b48cb9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:18.947793 1151 raft_consensus.cc:738] T 5adf6b492640417ca071c94702b48cb9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:18.948033 1151 consensus_queue.cc:260] T 5adf6b492640417ca071c94702b48cb9 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:18.948531 1151 ts_tablet_manager.cc:1428] T 5adf6b492640417ca071c94702b48cb9 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250901 01:52:18.948829 1151 tablet_bootstrap.cc:492] T 6c05eff658924ca4a9a1e7a37f917d67 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:18.950696 1151 tablet_bootstrap.cc:654] T 6c05eff658924ca4a9a1e7a37f917d67 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:18.950716 1150 ts_tablet_manager.cc:1428] T b73e362321284df1b18ef755dfe930a5 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20250901 01:52:18.951076 1150 tablet_bootstrap.cc:492] T 23af154a6f1c4e2e9352fb17c9024f10 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:18.952639 1332 tablet_replica.cc:331] T 2f28451a3ecc481a958588466f7fec77 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:18.952844 1332 raft_consensus.cc:2241] T 2f28451a3ecc481a958588466f7fec77 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:18.953032 1332 raft_consensus.cc:2270] T 2f28451a3ecc481a958588466f7fec77 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:18.953579 1332 ts_tablet_manager.cc:1905] T 2f28451a3ecc481a958588466f7fec77 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:18.955698 897 tablet_service.cc:1515] Processing DeleteTablet for tablet 670d90a37b934579b353bb4e4820127b with delete_type TABLET_DATA_DELETED (replaced by tablet 75a703c8e20a4a2fa0ab6ec58d41ef92) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:18.956111 921 raft_consensus.cc:2466] T fcb7b5b7d63d4f0b906fea97cb548de2 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:18.956828 992 leader_election.cc:304] T fcb7b5b7d63d4f0b906fea97cb548de2 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:18.957173 1669 raft_consensus.cc:2802] T fcb7b5b7d63d4f0b906fea97cb548de2 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:18.957312 1669 raft_consensus.cc:695] T fcb7b5b7d63d4f0b906fea97cb548de2 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:18.957577 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74b709cb5ec64006b2f12bead8217d80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:18.957615 1669 consensus_queue.cc:237] T fcb7b5b7d63d4f0b906fea97cb548de2 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:18.959064 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb7d7e1796c0456f8de98fb812b9e374. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:18.959937 901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1f2d98712ff4ea6aec665512de068bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:18.960832 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ba88af177874544b764cc32420e7ac9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:18.965567 1306 tablet_replica.cc:331] T c27bb23c150f4565a1d0ba8f231cc386 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:18.968933 1150 tablet_bootstrap.cc:654] T 23af154a6f1c4e2e9352fb17c9024f10 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:18.969970 1332 ts_tablet_manager.cc:1918] T 2f28451a3ecc481a958588466f7fec77 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:18.970782 1151 tablet_bootstrap.cc:492] T 6c05eff658924ca4a9a1e7a37f917d67 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:18.970963 1151 ts_tablet_manager.cc:1397] T 6c05eff658924ca4a9a1e7a37f917d67 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.001s
I20250901 01:52:18.971729 1151 raft_consensus.cc:357] T 6c05eff658924ca4a9a1e7a37f917d67 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:18.971947 1151 raft_consensus.cc:383] T 6c05eff658924ca4a9a1e7a37f917d67 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:18.972028 1151 raft_consensus.cc:738] T 6c05eff658924ca4a9a1e7a37f917d67 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:18.972299 1151 consensus_queue.cc:260] T 6c05eff658924ca4a9a1e7a37f917d67 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:18.973965 1151 ts_tablet_manager.cc:1428] T 6c05eff658924ca4a9a1e7a37f917d67 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20250901 01:52:18.974586 1151 tablet_bootstrap.cc:492] T dd694cfa09564d3da5d279c0861a4c32 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:18.978883 1332 log.cc:1199] T 2f28451a3ecc481a958588466f7fec77 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/2f28451a3ecc481a958588466f7fec77
I20250901 01:52:18.979854 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fbd44641d9e43c3b592e388e4adf350. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:18.981302 1332 ts_tablet_manager.cc:1939] T 2f28451a3ecc481a958588466f7fec77 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:18.982805 1151 tablet_bootstrap.cc:654] T dd694cfa09564d3da5d279c0861a4c32 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:18.988888 903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31614db4c53a4f3c96fd190d326ce601. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:18.989590 1306 raft_consensus.cc:2241] T c27bb23c150f4565a1d0ba8f231cc386 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:18.996683 1306 raft_consensus.cc:2270] T c27bb23c150f4565a1d0ba8f231cc386 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:18.998883 1306 ts_tablet_manager.cc:1905] T c27bb23c150f4565a1d0ba8f231cc386 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:19.008581 441 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 58c0e7a2e90e48528bc2bc7ec889f608 (table default.Table_409dad98d7eb455a8e16edfb4acedb34 [id=6f29cdc9974049368af2ae10beadb772]) successfully deleted
W20250901 01:52:19.008764 441 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.3:39447, user_credentials={real_user=slave}} blocked reactor thread for 94204.5us
I20250901 01:52:19.009539 441 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 3b18a7084727490380d26b3485e4e799 (table default.Table_da2c70a8f2fe49a0a8184e88c0a9e0ea [id=c1b1e9e3876244f68d31c63ce7bdebf1]) successfully deleted
I20250901 01:52:19.009771 441 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 2f28451a3ecc481a958588466f7fec77 (table default.Table_12450bd55edc4f7d96797422cfae2c50 [id=a919133c60784a4ca1bec96b7f410029]) successfully deleted
I20250901 01:52:19.009956 447 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_3fc2615b8f444281a2c3d5a748d58499" } new_table_name: "default.Table_db61e055dbd7450898e82c4c0c4763be" modify_external_catalogs: true
I20250901 01:52:19.019512 1151 tablet_bootstrap.cc:492] T dd694cfa09564d3da5d279c0861a4c32 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:19.019685 1151 ts_tablet_manager.cc:1397] T dd694cfa09564d3da5d279c0861a4c32 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.045s user 0.007s sys 0.015s
I20250901 01:52:19.022310 1151 raft_consensus.cc:357] T dd694cfa09564d3da5d279c0861a4c32 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:19.022502 1151 raft_consensus.cc:383] T dd694cfa09564d3da5d279c0861a4c32 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:19.022585 1151 raft_consensus.cc:738] T dd694cfa09564d3da5d279c0861a4c32 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:19.022828 1151 consensus_queue.cc:260] T dd694cfa09564d3da5d279c0861a4c32 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:19.023334 1151 ts_tablet_manager.cc:1428] T dd694cfa09564d3da5d279c0861a4c32 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250901 01:52:19.024107 1151 tablet_bootstrap.cc:492] T a7ff996482164776b468cf305989866b P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:19.026068 1151 tablet_bootstrap.cc:654] T a7ff996482164776b468cf305989866b P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:19.029594 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_c0977985d6ee4b21a086dc892c860d6a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:19.030732 1696 catalog_manager.cc:1261] Loaded cluster ID: d2561e56f09e40169f2e331183eec812
I20250901 01:52:19.030853 1696 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: loading cluster ID for follower catalog manager: success
W20250901 01:52:19.032491 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c0977985d6ee4b21a086dc892c860d6a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:19.034257 451 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_d228a4fe971b454ebe99e712ed80abdb" } new_table_name: "default.Table_bd8094cd20b6427f85ed4c7121ce549c" modify_external_catalogs: true
I20250901 01:52:19.032464 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 824229183e024feaa975c7f0387c9c1a
I20250901 01:52:19.038901 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 824229183e024feaa975c7f0387c9c1a with delete_type TABLET_DATA_DELETED (replaced by tablet d4c9aa723aee4dc4b6237cb5c3722b52) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:19.041188 454 catalog_manager.cc:6652] ReplaceTablet: tablet 824229183e024feaa975c7f0387c9c1a deleted and replaced by tablet d4c9aa723aee4dc4b6237cb5c3722b52
I20250901 01:52:19.042282 1696 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: acquiring CA information for follower catalog manager: success
I20250901 01:52:19.047251 1151 tablet_bootstrap.cc:492] T a7ff996482164776b468cf305989866b P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:19.047439 1151 ts_tablet_manager.cc:1397] T a7ff996482164776b468cf305989866b P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.002s
I20250901 01:52:19.048566 1151 raft_consensus.cc:357] T a7ff996482164776b468cf305989866b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:19.048758 1151 raft_consensus.cc:383] T a7ff996482164776b468cf305989866b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:19.048827 1151 raft_consensus.cc:738] T a7ff996482164776b468cf305989866b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:19.049077 1151 consensus_queue.cc:260] T a7ff996482164776b468cf305989866b P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
W20250901 01:52:19.053506 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_21b93b5fe2ce49c6bc637d8cab41fc06 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:19.054303 1696 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250901 01:52:19.054838 454 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_eb9eaa97a468420bb672b720834c6ddd" } modify_external_catalogs: true
I20250901 01:52:19.055145 1151 ts_tablet_manager.cc:1428] T a7ff996482164776b468cf305989866b P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20250901 01:52:19.055522 1151 tablet_bootstrap.cc:492] T 154574e17f214cc0ad56e6e6be3af899 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:19.055806 454 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_eb9eaa97a468420bb672b720834c6ddd" } modify_external_catalogs: true
I20250901 01:52:19.057315 1151 tablet_bootstrap.cc:654] T 154574e17f214cc0ad56e6e6be3af899 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:19.059787 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_d51ef24ec0f846c7ad212360eb478cd7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:19.061076 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d51ef24ec0f846c7ad212360eb478cd7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:19.062489 446 catalog_manager.cc:6127] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: default.Table_5a72e7206f604795915f3a577d37f874 [id=1d8f34cc14cb4e90a562ff4424b8577a] alter complete (version 5)
I20250901 01:52:19.063360 446 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_c96393e11f3a4cca8202ccc2df6635e0" } modify_external_catalogs: true
W20250901 01:52:19.054762 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 824229183e024feaa975c7f0387c9c1a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 824229183e024feaa975c7f0387c9c1a already in progress: creating tablet
I20250901 01:52:19.068894 446 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_c96393e11f3a4cca8202ccc2df6635e0" } modify_external_catalogs: true
W20250901 01:52:19.086475 447 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c6f7c525f59b49938b697de98d4b5835 has no consensus state
I20250901 01:52:19.088020 453 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_9bf84b7a07be4f57a58101480fe6d2c0" } modify_external_catalogs: true
I20250901 01:52:19.088205 453 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_9bf84b7a07be4f57a58101480fe6d2c0" } modify_external_catalogs: true
I20250901 01:52:19.088822 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_aa1bfa4e8b59486f96c587bc9cf84d73"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:19.090592 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aa1bfa4e8b59486f96c587bc9cf84d73 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250901 01:52:19.093196 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 05630a72ddae4de0b890dece2feacc31 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 05630a72ddae4de0b890dece2feacc31 already in progress: creating tablet
W20250901 01:52:19.093662 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 48a211e174054866b030cdfb0cf947f4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 48a211e174054866b030cdfb0cf947f4 already in progress: creating tablet
W20250901 01:52:19.093883 440 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 75e7a90815e449c39f386901a504143c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 75e7a90815e449c39f386901a504143c already in progress: creating tablet
I20250901 01:52:19.099016 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 824229183e024feaa975c7f0387c9c1a with delete_type TABLET_DATA_DELETED (replaced by tablet d4c9aa723aee4dc4b6237cb5c3722b52) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:19.099273 892 tablet_service.cc:1515] Processing DeleteTablet for tablet ef58f477ebeb49639ed7d071e1a62c10 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:13 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:19.104647 1669 raft_consensus.cc:491] T 3df908c3096a4df7bddd8ac88b49cfe0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:19.104918 1669 raft_consensus.cc:513] T 3df908c3096a4df7bddd8ac88b49cfe0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:19.105792 1669 leader_election.cc:290] T 3df908c3096a4df7bddd8ac88b49cfe0 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:19.108969 992 leader_election.cc:336] T 3df908c3096a4df7bddd8ac88b49cfe0 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:19.109639 992 leader_election.cc:343] T 3df908c3096a4df7bddd8ac88b49cfe0 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 3df908c3096a4df7bddd8ac88b49cfe0
I20250901 01:52:19.109812 992 leader_election.cc:304] T 3df908c3096a4df7bddd8ac88b49cfe0 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:19.110450 1669 raft_consensus.cc:2747] T 3df908c3096a4df7bddd8ac88b49cfe0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:19.110826 451 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 410d44e5f1974ed69e5dee65e98856a4 has no consensus state
I20250901 01:52:19.111451 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_820e1c5a64cd4128b7ae1b70bd951964"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:19.118247 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_820e1c5a64cd4128b7ae1b70bd951964 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:19.121598 447 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_74a88a4668244f6294b0fd18f24dc677" } new_table_name: "default.Table_277b91103c4d41d2a9f9424f99082b29" modify_external_catalogs: true
I20250901 01:52:19.130591 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_df96f1aa51644aed967bdfd4f75588f6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:19.131597 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_df96f1aa51644aed967bdfd4f75588f6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250901 01:52:19.137900 447 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 61ddcf86c9f34726939cc116434f0822 has no consensus state
I20250901 01:52:19.137982 448 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_aac69ecb4bd042ad83e7ea0fc72f6ef9" } new_table_name: "default.Table_02e911ce389e4ad581c0a2f976f99c4e" modify_external_catalogs: true
I20250901 01:52:19.138672 447 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_95884295301d476fb3e03054727f9cf1" } modify_external_catalogs: true
I20250901 01:52:19.139004 447 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_95884295301d476fb3e03054727f9cf1" } modify_external_catalogs: true
I20250901 01:52:19.140017 1150 tablet_bootstrap.cc:492] T 23af154a6f1c4e2e9352fb17c9024f10 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:19.140928 1150 ts_tablet_manager.cc:1397] T 23af154a6f1c4e2e9352fb17c9024f10 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.189s user 0.015s sys 0.003s
I20250901 01:52:19.145442 451 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_fe014107d0ee4fb8bdfca5356093854b" } new_table_name: "default.Table_fd17a2ad225546aa821489ce94a788a2" modify_external_catalogs: true
I20250901 01:52:19.148054 1150 raft_consensus.cc:357] T 23af154a6f1c4e2e9352fb17c9024f10 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:19.148770 1150 raft_consensus.cc:383] T 23af154a6f1c4e2e9352fb17c9024f10 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:19.148900 1150 raft_consensus.cc:738] T 23af154a6f1c4e2e9352fb17c9024f10 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:19.149547 1150 consensus_queue.cc:260] T 23af154a6f1c4e2e9352fb17c9024f10 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:19.150068 1150 ts_tablet_manager.cc:1428] T 23af154a6f1c4e2e9352fb17c9024f10 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20250901 01:52:19.153354 1150 tablet_bootstrap.cc:492] T 9473f0ef978c4bfaa41fb7b0729e8863 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
W20250901 01:52:19.155480 448 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1495de70f0844f82a0eea99c059f076b consensus state has no leader
I20250901 01:52:19.156662 448 master_service.cc:948] ReplaceTablet: received request to replace tablet 670d90a37b934579b353bb4e4820127b from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:19.160198 1150 tablet_bootstrap.cc:654] T 9473f0ef978c4bfaa41fb7b0729e8863 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:19.163756 451 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e1d18e999b5e42b1873631e0f1445081 consensus state has no leader
I20250901 01:52:19.171577 1306 ts_tablet_manager.cc:1918] T c27bb23c150f4565a1d0ba8f231cc386 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:19.176254 451 master_service.cc:948] ReplaceTablet: received request to replace tablet 0cb3262973934c8689b851adad60afe0 from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:19.181849 1306 log.cc:1199] T c27bb23c150f4565a1d0ba8f231cc386 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/c27bb23c150f4565a1d0ba8f231cc386
I20250901 01:52:19.183207 1306 ts_tablet_manager.cc:1939] T c27bb23c150f4565a1d0ba8f231cc386 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:19.185424 1306 tablet_replica.cc:331] T c95798f0daf4437db18a291c4464522e P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:19.185750 1306 raft_consensus.cc:2241] T c95798f0daf4437db18a291c4464522e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:19.187011 1306 raft_consensus.cc:2270] T c95798f0daf4437db18a291c4464522e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:19.187774 1306 ts_tablet_manager.cc:1905] T c95798f0daf4437db18a291c4464522e P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:19.187734 440 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet c27bb23c150f4565a1d0ba8f231cc386 (table default.Table_e066e678c3a8406583dbe851d3424bac [id=f70c2bc07ea245bea6293c3742dfcf94]) successfully deleted
W20250901 01:52:19.188015 440 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 66a48a8ebe154b2cac89c234eaa378ff because the tablet was not found. No further retry: Not found: Tablet not found: 66a48a8ebe154b2cac89c234eaa378ff
I20250901 01:52:19.198102 1150 tablet_bootstrap.cc:492] T 9473f0ef978c4bfaa41fb7b0729e8863 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:19.198307 1150 ts_tablet_manager.cc:1397] T 9473f0ef978c4bfaa41fb7b0729e8863 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.048s user 0.002s sys 0.012s
I20250901 01:52:19.199538 1150 raft_consensus.cc:357] T 9473f0ef978c4bfaa41fb7b0729e8863 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:19.199803 1150 raft_consensus.cc:383] T 9473f0ef978c4bfaa41fb7b0729e8863 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:19.199932 1150 raft_consensus.cc:738] T 9473f0ef978c4bfaa41fb7b0729e8863 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:19.200250 1150 consensus_queue.cc:260] T 9473f0ef978c4bfaa41fb7b0729e8863 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:19.201062 1150 ts_tablet_manager.cc:1428] T 9473f0ef978c4bfaa41fb7b0729e8863 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250901 01:52:19.201786 1150 tablet_bootstrap.cc:492] T 438cd35ff3f74c49b908d072560077db P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:19.202222 1669 raft_consensus.cc:491] T a59ac57aebe24860aa02f53ce0485910 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:19.202425 1669 raft_consensus.cc:513] T a59ac57aebe24860aa02f53ce0485910 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:19.203135 1669 leader_election.cc:290] T a59ac57aebe24860aa02f53ce0485910 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:19.204284 992 leader_election.cc:343] T a59ac57aebe24860aa02f53ce0485910 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: a59ac57aebe24860aa02f53ce0485910
W20250901 01:52:19.206001 992 leader_election.cc:336] T a59ac57aebe24860aa02f53ce0485910 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:19.206214 992 leader_election.cc:304] T a59ac57aebe24860aa02f53ce0485910 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:19.207190 1669 raft_consensus.cc:2747] T a59ac57aebe24860aa02f53ce0485910 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:19.208252 1150 tablet_bootstrap.cc:654] T 438cd35ff3f74c49b908d072560077db P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:19.209255 451 master_service.cc:948] ReplaceTablet: received request to replace tablet a54af99bc10045e8b0b6868e5aed98b4 from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:19.214243 1150 tablet_bootstrap.cc:492] T 438cd35ff3f74c49b908d072560077db P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:19.214411 1150 ts_tablet_manager.cc:1397] T 438cd35ff3f74c49b908d072560077db P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.003s
I20250901 01:52:19.215219 1150 raft_consensus.cc:357] T 438cd35ff3f74c49b908d072560077db P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:19.215425 1150 raft_consensus.cc:383] T 438cd35ff3f74c49b908d072560077db P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:19.215497 1150 raft_consensus.cc:738] T 438cd35ff3f74c49b908d072560077db P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:19.215724 1150 consensus_queue.cc:260] T 438cd35ff3f74c49b908d072560077db P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:19.216217 1150 ts_tablet_manager.cc:1428] T 438cd35ff3f74c49b908d072560077db P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250901 01:52:19.216710 1150 tablet_bootstrap.cc:492] T a6445b29591548be9f9577db1ba0bfa5 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:19.218611 1150 tablet_bootstrap.cc:654] T a6445b29591548be9f9577db1ba0bfa5 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:19.221485 1711 raft_consensus.cc:491] T b73e362321284df1b18ef755dfe930a5 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:19.221684 1711 raft_consensus.cc:513] T b73e362321284df1b18ef755dfe930a5 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:19.222465 1711 leader_election.cc:290] T b73e362321284df1b18ef755dfe930a5 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:19.224761 858 leader_election.cc:336] T b73e362321284df1b18ef755dfe930a5 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:19.225440 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b73e362321284df1b18ef755dfe930a5" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:19.227516 859 leader_election.cc:343] T b73e362321284df1b18ef755dfe930a5 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:19.227609 1150 tablet_bootstrap.cc:492] T a6445b29591548be9f9577db1ba0bfa5 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:19.227659 859 leader_election.cc:304] T b73e362321284df1b18ef755dfe930a5 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:19.227758 1150 ts_tablet_manager.cc:1397] T a6445b29591548be9f9577db1ba0bfa5 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.007s
I20250901 01:52:19.228546 1711 raft_consensus.cc:2747] T b73e362321284df1b18ef755dfe930a5 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:19.230330 1150 raft_consensus.cc:357] T a6445b29591548be9f9577db1ba0bfa5 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:19.230518 1150 raft_consensus.cc:383] T a6445b29591548be9f9577db1ba0bfa5 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:19.230621 1150 raft_consensus.cc:738] T a6445b29591548be9f9577db1ba0bfa5 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:19.230881 1150 consensus_queue.cc:260] T a6445b29591548be9f9577db1ba0bfa5 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:19.231400 1150 ts_tablet_manager.cc:1428] T a6445b29591548be9f9577db1ba0bfa5 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250901 01:52:19.231683 1150 tablet_bootstrap.cc:492] T 3fda93767e494a4684978c8522b829dd P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:19.233711 902 tablet_service.cc:1468] Processing CreateTablet for tablet 75345ea2a3dc40fc8c9f8090d8839d2d (DEFAULT_TABLE table=default.Table_9734ae6c72e240c698b66ea83c536249 [id=f6d2701cc218463a9d1847e617bfec46]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.234347 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75345ea2a3dc40fc8c9f8090d8839d2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.234812 455 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet f65ec84958eb488d8b0f812b59b5b3d5 (table default.Table_a3ff88e352884abdba91e6d03acbae22 [id=d1cd5288671540c0a95f7a037363014e]). Expected version 2 got 0
I20250901 01:52:19.236833 896 tablet_service.cc:1468] Processing CreateTablet for tablet 3eba54e5ddd14c518f4cbd539268ae37 (DEFAULT_TABLE table=default.Table_df46d9604f6a4a19b4d7c3424f48d90a [id=d28ebc8fa6d745919bf025a5a9f6ebca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.237226 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3eba54e5ddd14c518f4cbd539268ae37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.240715 1150 tablet_bootstrap.cc:654] T 3fda93767e494a4684978c8522b829dd P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:19.240695 1033 tablet_service.cc:1468] Processing CreateTablet for tablet 3eba54e5ddd14c518f4cbd539268ae37 (DEFAULT_TABLE table=default.Table_df46d9604f6a4a19b4d7c3424f48d90a [id=d28ebc8fa6d745919bf025a5a9f6ebca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.241137 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3eba54e5ddd14c518f4cbd539268ae37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.242846 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_59039012009944fdbe0179002ea44633"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:19.243412 449 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet f65ec84958eb488d8b0f812b59b5b3d5 (table default.Table_a3ff88e352884abdba91e6d03acbae22 [id=d1cd5288671540c0a95f7a037363014e]). Expected version 2 got 0
W20250901 01:52:19.243626 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_59039012009944fdbe0179002ea44633 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:19.243883 449 catalog_manager.cc:5582] T 34042ea5c1b645969c1de41fe587c343 P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: UNKNOWN } } }
I20250901 01:52:19.240695 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 75345ea2a3dc40fc8c9f8090d8839d2d (DEFAULT_TABLE table=default.Table_9734ae6c72e240c698b66ea83c536249 [id=f6d2701cc218463a9d1847e617bfec46]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.244298 449 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 34042ea5c1b645969c1de41fe587c343 (table default.Table_e37cc89ad7ee4d96852b121d67b06248 [id=edbc04078c904f29a16a1bbb354dad92]). Expected version 1 got 0
I20250901 01:52:19.244552 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75345ea2a3dc40fc8c9f8090d8839d2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.244589 449 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet fcb7b5b7d63d4f0b906fea97cb548de2 (table default.Table_dd8e6f9348c440bca1569cf68eec5beb [id=4d9ea39016ca4bf4859f4f80ea4e9ed6]). Expected version 2 got 0
I20250901 01:52:19.244846 449 catalog_manager.cc:5582] T 4410da9e95454ecf95faf80221b0a71c P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: UNKNOWN } } }
I20250901 01:52:19.245273 449 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 4410da9e95454ecf95faf80221b0a71c (table default.Table_420b0979d6fd42da9b320456f4ccc9ce [id=d2004563c4374790954327f05c305cc6]). Expected version 1 got 0
I20250901 01:52:19.246058 449 catalog_manager.cc:5582] T 7442c6ff699141e1aff208ebdc9d192a P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: UNKNOWN } } }
I20250901 01:52:19.247783 449 catalog_manager.cc:5582] T 186ec810d02e4748aad15228b5866ac1 P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } }
I20250901 01:52:19.253206 452 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 4902062977ea42149e3466e999117322
I20250901 01:52:19.253997 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_122585be3bdc45fb9498d02564fb8567"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:19.254551 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_122585be3bdc45fb9498d02564fb8567 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:19.255039 1038 tablet_service.cc:1468] Processing CreateTablet for tablet e2aa32450a584395ae07b49f76a7b26d (DEFAULT_TABLE table=default.Table_7dd48000e9b24337b75ceb4c53b6be25 [id=8051adf4041f4a2686a0b8e3850ef29c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.255456 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2aa32450a584395ae07b49f76a7b26d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.256454 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 1a60c10e42104502ac2d3b80b8126ceb
I20250901 01:52:19.256937 450 catalog_manager.cc:6652] ReplaceTablet: tablet 1a60c10e42104502ac2d3b80b8126ceb deleted and replaced by tablet 88ceba7f48b04d7885cc0164668700a5
I20250901 01:52:19.257247 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_bb44f1b7c92140309a212038f8bf1286"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:19.257756 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bb44f1b7c92140309a212038f8bf1286 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:19.257767 896 tablet_service.cc:1468] Processing CreateTablet for tablet e2aa32450a584395ae07b49f76a7b26d (DEFAULT_TABLE table=default.Table_7dd48000e9b24337b75ceb4c53b6be25 [id=8051adf4041f4a2686a0b8e3850ef29c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.258167 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2aa32450a584395ae07b49f76a7b26d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.261806 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 617cc33d0d23439cb5f1e0c31bf11281 (DEFAULT_TABLE table=default.Table_f4482a2fa4cc4cfd9658b47dda711f2d [id=fb088d31bba64f04871fbfd355d6263b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.262181 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 617cc33d0d23439cb5f1e0c31bf11281. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.262243 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet a54af99bc10045e8b0b6868e5aed98b4
I20250901 01:52:19.262934 451 catalog_manager.cc:6652] ReplaceTablet: tablet a54af99bc10045e8b0b6868e5aed98b4 deleted and replaced by tablet 50472a7aa2f04e0fbd65ac6c86c34b3e
I20250901 01:52:19.263726 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_c7bfa05d564246a6be270afb7a3f0bde"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:19.264163 447 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 74b709cb5ec64006b2f12bead8217d80
W20250901 01:52:19.264394 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c7bfa05d564246a6be270afb7a3f0bde in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:19.264878 1669 raft_consensus.cc:491] T d60d571b2c2840a2b20629bb832d1309 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:19.265053 1669 raft_consensus.cc:513] T d60d571b2c2840a2b20629bb832d1309 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:19.265183 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 246b33338de647689d7404c70676e76f (DEFAULT_TABLE table=default.Table_8deed8083e3f4f4f8533e42022a8633d [id=a7b1d2a8ace14ef8bcce0a3bed41e85d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.265565 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 246b33338de647689d7404c70676e76f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.265776 1669 leader_election.cc:290] T d60d571b2c2840a2b20629bb832d1309 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:19.264941 453 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 64fb46b942ff4e4c8749986ecc3dfe65
I20250901 01:52:19.265172 446 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 49d30da95b14400b845270122487a2a0
I20250901 01:52:19.265395 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 6741c56499b44f648fb032c8cf50205b
I20250901 01:52:19.265957 455 catalog_manager.cc:6127] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: default.Table_5610feff8a344dc9be4bc1f0ad8ddd51 [id=78b00856a9b747c19f390b4fdae8ddcf] alter complete (version 3)
I20250901 01:52:19.268873 1029 tablet_service.cc:1468] Processing CreateTablet for tablet e8e7d2a72f704f52a8f8d46911a8a9e4 (DEFAULT_TABLE table=default.Table_0ece0b3619db43da9d3bb2b0a72a7ef9 [id=623e11cd89c94142beb5c1a46e1adcd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.269263 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8e7d2a72f704f52a8f8d46911a8a9e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.272876 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet 4902062977ea42149e3466e999117322 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:18 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:19.273500 1031 tablet_service.cc:1468] Processing CreateTablet for tablet 410d44e5f1974ed69e5dee65e98856a4 (DEFAULT_TABLE table=default.Table_bd8094cd20b6427f85ed4c7121ce549c [id=ae79f2c40fd7483ea400a36cf3c48888]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.273828 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 410d44e5f1974ed69e5dee65e98856a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.274618 455 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 438cd35ff3f74c49b908d072560077db (table default.Table_62ececf8293e4e22bad9e9568f496bbe [id=5a33cfdf60684867bb5f36d84ee80468]). Expected version 4 got 0
I20250901 01:52:19.288236 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 1a60c10e42104502ac2d3b80b8126ceb with delete_type TABLET_DATA_DELETED (replaced by tablet 88ceba7f48b04d7885cc0164668700a5) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:19.288285 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 268439ff8d3a4f7a89f3d6395f2079c5 (DEFAULT_TABLE table=default.Table_b4fbc34da806475c912052d0fab3fc53 [id=3f1b378fb3444fd08dc947740ac0fa01]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.289125 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 268439ff8d3a4f7a89f3d6395f2079c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.291750 1150 tablet_bootstrap.cc:492] T 3fda93767e494a4684978c8522b829dd P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:19.291929 1150 ts_tablet_manager.cc:1397] T 3fda93767e494a4684978c8522b829dd P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.060s user 0.003s sys 0.006s
I20250901 01:52:19.292656 455 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_622c258d8abf4619a3919206fe82e6e7" } new_table_name: "default.Table_8825dd28afbe464193eb1b668cc9326a" modify_external_catalogs: true
I20250901 01:52:19.293056 1150 raft_consensus.cc:357] T 3fda93767e494a4684978c8522b829dd P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:19.293344 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_a5c783cbdb694af69047b31d99ff8def"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:19.293478 1150 raft_consensus.cc:383] T 3fda93767e494a4684978c8522b829dd P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:19.293546 1150 raft_consensus.cc:738] T 3fda93767e494a4684978c8522b829dd P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:19.293797 1150 consensus_queue.cc:260] T 3fda93767e494a4684978c8522b829dd P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:19.293903 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a5c783cbdb694af69047b31d99ff8def in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:19.295781 1150 ts_tablet_manager.cc:1428] T 3fda93767e494a4684978c8522b829dd P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250901 01:52:19.295708 1024 tablet_service.cc:1468] Processing CreateTablet for tablet c6f7c525f59b49938b697de98d4b5835 (DEFAULT_TABLE table=default.Table_db61e055dbd7450898e82c4c0c4763be [id=aabca67f6f754820977ac9c4c20f3e4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.296190 1150 tablet_bootstrap.cc:492] T 9d886fa63f8f4996b087997b542e8bb1 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:19.296823 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_a133336a993b4c9aa51562985c3ce4f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:19.297375 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a133336a993b4c9aa51562985c3ce4f9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:19.298723 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6f7c525f59b49938b697de98d4b5835. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.300915 1150 tablet_bootstrap.cc:654] T 9d886fa63f8f4996b087997b542e8bb1 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:19.305055 1669 raft_consensus.cc:491] T dd694cfa09564d3da5d279c0861a4c32 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:19.305214 1669 raft_consensus.cc:513] T dd694cfa09564d3da5d279c0861a4c32 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:19.306313 1035 tablet_service.cc:1468] Processing CreateTablet for tablet 26c8ed083cea4dc0acca80d00b11b29b (DEFAULT_TABLE table=default.Table_5a32379f33204b9e9b2dfe82995134dc [id=5e18d8cdb4fd4399b1a9f8518676f08f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.307736 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 764bfeb2a7d14c40815b5ec1b07bdd62 (DEFAULT_TABLE table=default.Table_4401d83f75f8464da49d5b811d7dd89d [id=16143c8e6f484f9a873437e709d8466d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.308372 1669 leader_election.cc:290] T dd694cfa09564d3da5d279c0861a4c32 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:19.309300 1037 tablet_service.cc:1468] Processing CreateTablet for tablet dabebc77f72e4bb19e264be2f496202a (DEFAULT_TABLE table=default.Table_364cf1adeb2c4102aad338f2e228fb0b [id=a4d0af97df5949cca940a81a41b8e454]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.310024 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 764bfeb2a7d14c40815b5ec1b07bdd62. 1 dirs total, 0 dirs full, 0 dirs failed
W20250901 01:52:19.310400 455 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e9f1f892658b4736859ff507536920a1 has no consensus state
I20250901 01:52:19.315662 448 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_853b0fae8015415b91325d10a633ccf6" } new_table_name: "default.Table_5ea919f20c6b4a4e91e599357a17a53a" modify_external_catalogs: true
I20250901 01:52:19.315997 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26c8ed083cea4dc0acca80d00b11b29b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.323861 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dabebc77f72e4bb19e264be2f496202a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.324537 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_f611c1794e564a0496a761da46171d0e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:19.325259 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f611c1794e564a0496a761da46171d0e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250901 01:52:19.325865 448 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7954393c8c1c4457ab8bb723c03a9c8b consensus state has no leader
I20250901 01:52:19.328035 1150 tablet_bootstrap.cc:492] T 9d886fa63f8f4996b087997b542e8bb1 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:19.328236 1150 ts_tablet_manager.cc:1397] T 9d886fa63f8f4996b087997b542e8bb1 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.032s user 0.012s sys 0.004s
I20250901 01:52:19.329197 1150 raft_consensus.cc:357] T 9d886fa63f8f4996b087997b542e8bb1 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:19.329452 1150 raft_consensus.cc:383] T 9d886fa63f8f4996b087997b542e8bb1 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:19.329612 1150 raft_consensus.cc:738] T 9d886fa63f8f4996b087997b542e8bb1 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:19.329921 1150 consensus_queue.cc:260] T 9d886fa63f8f4996b087997b542e8bb1 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:19.329977 1036 tablet_service.cc:1468] Processing CreateTablet for tablet 858218c4427c41e980b2b5c969a5af40 (DEFAULT_TABLE table=default.Table_4185b4f73a2743e08f24a26878cf73ca [id=264cc469a73841e79538843479a165ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.330942 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 858218c4427c41e980b2b5c969a5af40. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.331198 1150 ts_tablet_manager.cc:1428] T 9d886fa63f8f4996b087997b542e8bb1 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250901 01:52:19.331601 1150 tablet_bootstrap.cc:492] T f62d96c37cca4143a4b098b7c69979e6 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:19.331616 1025 tablet_service.cc:1468] Processing CreateTablet for tablet da5f69c5b0ca4257ad2ff7918ee2af2e (DEFAULT_TABLE table=default.Table_8f7db2ae75a24dfab71c2a95c6a0a404 [id=61f1e8a842844ed899e5913c76f362ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.331962 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da5f69c5b0ca4257ad2ff7918ee2af2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.332455 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_99afdac28a4e4ac9ae93b4fbbe704a4d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:19.332995 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_99afdac28a4e4ac9ae93b4fbbe704a4d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:19.329372 902 tablet_service.cc:1468] Processing CreateTablet for tablet 8b9d79a25452410891c09fd3f7c4be6d (DEFAULT_TABLE table=default.Table_442e11b9211c4b209c68f653d7bfc313 [id=af349c3f534e4e8e99fdae0344187c44]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.333787 1150 tablet_bootstrap.cc:654] T f62d96c37cca4143a4b098b7c69979e6 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:19.333832 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b9d79a25452410891c09fd3f7c4be6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.341307 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet a54af99bc10045e8b0b6868e5aed98b4 with delete_type TABLET_DATA_DELETED (replaced by tablet 50472a7aa2f04e0fbd65ac6c86c34b3e) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:19.341832 1022 tablet_service.cc:1468] Processing CreateTablet for tablet f195b040a6ae4b7b83a8308ab0a12b37 (DEFAULT_TABLE table=default.Table_8b1086c9a4c44239851c586ea7267e3c [id=523de440fbab4bf687c0f6168e48513c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.342355 454 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_add9fa2d6db444b6896dd970425259ae" } new_table_name: "default.Table_f840789a58984799bd8057128f6ea439" modify_external_catalogs: true
I20250901 01:52:19.343513 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f195b040a6ae4b7b83a8308ab0a12b37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.346462 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_259a716264db43c49671b6fef6665f0e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:19.347262 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_efc44c841f0b4ae2a168be904751b65c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:19.347846 1019 tablet_service.cc:1468] Processing CreateTablet for tablet d0b7c84ec9394a20ad77add3808e555a (DEFAULT_TABLE table=default.Table_85b9e2f3f8f7400aa833201daae93dc3 [id=cac94430e07342dfa9b266df6472b6e3]), partition=RANGE (key) PARTITION UNBOUNDED
W20250901 01:52:19.347630 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_259a716264db43c49671b6fef6665f0e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:19.348650 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0b7c84ec9394a20ad77add3808e555a. 1 dirs total, 0 dirs full, 0 dirs failed
W20250901 01:52:19.349450 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_efc44c841f0b4ae2a168be904751b65c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:19.354135 902 tablet_service.cc:1468] Processing CreateTablet for tablet 617cc33d0d23439cb5f1e0c31bf11281 (DEFAULT_TABLE table=default.Table_f4482a2fa4cc4cfd9658b47dda711f2d [id=fb088d31bba64f04871fbfd355d6263b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.354616 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 617cc33d0d23439cb5f1e0c31bf11281. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.356976 449 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_bd8094cd20b6427f85ed4c7121ce549c" } new_table_name: "default.Table_78ccd6cda2e540bd8eadf559a5c0327b" modify_external_catalogs: true
W20250901 01:52:19.363732 454 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e87d1281f35648b0b115b67c95e244e7 consensus state has no leader
I20250901 01:52:19.368367 896 tablet_service.cc:1468] Processing CreateTablet for tablet 246b33338de647689d7404c70676e76f (DEFAULT_TABLE table=default.Table_8deed8083e3f4f4f8533e42022a8633d [id=a7b1d2a8ace14ef8bcce0a3bed41e85d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.368391 1028 tablet_service.cc:1468] Processing CreateTablet for tablet f98d1d8d660443cb8066a547e5d9b77f (DEFAULT_TABLE table=default.Table_583918346a424cb293d2c35bfa4a5b5f [id=1f677c66803b4cb1bf585cb621a4df6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.368548 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 4902062977ea42149e3466e999117322 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:18 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:19.368898 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 246b33338de647689d7404c70676e76f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.368832 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 4fca1435d82447f289fda4035aa2ed89 (DEFAULT_TABLE table=default.Table_fbc57d539a2b41e59dfc5369ac8c92e1 [id=8e14f99474a1467baed19f5268f09c5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.369230 886 tablet_service.cc:1468] Processing CreateTablet for tablet e8e7d2a72f704f52a8f8d46911a8a9e4 (DEFAULT_TABLE table=default.Table_0ece0b3619db43da9d3bb2b0a72a7ef9 [id=623e11cd89c94142beb5c1a46e1adcd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.369611 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8e7d2a72f704f52a8f8d46911a8a9e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.370481 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fca1435d82447f289fda4035aa2ed89. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.371187 1032 tablet_service.cc:1468] Processing CreateTablet for tablet 93bcac1fd95d44dc927abfca81692921 (DEFAULT_TABLE table=default.Table_b8f8ee74648c406eaf589d3ab80934db [id=b8f69313439543be9c11eea88dfa69c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.371977 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93bcac1fd95d44dc927abfca81692921. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.376797 899 tablet_service.cc:1468] Processing CreateTablet for tablet 81ce448b148447ddbc41875d863e0518 (DEFAULT_TABLE table=default.Table_6e461d9cb0ae4dd095975bbc21533794 [id=49673fa73f8c436e9b8026efc4f3cbbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.376955 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 1a60c10e42104502ac2d3b80b8126ceb with delete_type TABLET_DATA_DELETED (replaced by tablet 88ceba7f48b04d7885cc0164668700a5) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:19.377161 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81ce448b148447ddbc41875d863e0518. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.377302 886 tablet_service.cc:1468] Processing CreateTablet for tablet 410d44e5f1974ed69e5dee65e98856a4 (DEFAULT_TABLE table=default.Table_bd8094cd20b6427f85ed4c7121ce549c [id=ae79f2c40fd7483ea400a36cf3c48888]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.377663 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 410d44e5f1974ed69e5dee65e98856a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.378437 455 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_8a88621295cf403da5cf0fc50afbb345" } new_table_name: "default.Table_3f9422b08b0349b9a5466218f0fe1b06" modify_external_catalogs: true
I20250901 01:52:19.380678 893 tablet_service.cc:1468] Processing CreateTablet for tablet 268439ff8d3a4f7a89f3d6395f2079c5 (DEFAULT_TABLE table=default.Table_b4fbc34da806475c912052d0fab3fc53 [id=3f1b378fb3444fd08dc947740ac0fa01]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.381034 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 268439ff8d3a4f7a89f3d6395f2079c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.353003 1038 tablet_service.cc:1468] Processing CreateTablet for tablet 8b9d79a25452410891c09fd3f7c4be6d (DEFAULT_TABLE table=default.Table_442e11b9211c4b209c68f653d7bfc313 [id=af349c3f534e4e8e99fdae0344187c44]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.383916 1034 tablet_service.cc:1468] Processing CreateTablet for tablet 61ddcf86c9f34726939cc116434f0822 (DEFAULT_TABLE table=default.Table_277b91103c4d41d2a9f9424f99082b29 [id=f9dcf4937b1143248bfa9d88053fce17]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.384238 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f98d1d8d660443cb8066a547e5d9b77f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.384224 903 tablet_service.cc:1468] Processing CreateTablet for tablet c6f7c525f59b49938b697de98d4b5835 (DEFAULT_TABLE table=default.Table_db61e055dbd7450898e82c4c0c4763be [id=aabca67f6f754820977ac9c4c20f3e4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.384567 903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6f7c525f59b49938b697de98d4b5835. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.393556 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61ddcf86c9f34726939cc116434f0822. 1 dirs total, 0 dirs full, 0 dirs failed
W20250901 01:52:19.398298 449 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 410d44e5f1974ed69e5dee65e98856a4 consensus state has no leader
I20250901 01:52:19.404945 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 81ce448b148447ddbc41875d863e0518 (DEFAULT_TABLE table=default.Table_6e461d9cb0ae4dd095975bbc21533794 [id=49673fa73f8c436e9b8026efc4f3cbbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.406543 903 tablet_service.cc:1468] Processing CreateTablet for tablet 26c8ed083cea4dc0acca80d00b11b29b (DEFAULT_TABLE table=default.Table_5a32379f33204b9e9b2dfe82995134dc [id=5e18d8cdb4fd4399b1a9f8518676f08f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.406984 903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26c8ed083cea4dc0acca80d00b11b29b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250901 01:52:19.407305 455 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6c0abbfc8d904dce989ce51961e4237d consensus state has no leader
I20250901 01:52:19.413178 899 tablet_service.cc:1468] Processing CreateTablet for tablet dabebc77f72e4bb19e264be2f496202a (DEFAULT_TABLE table=default.Table_364cf1adeb2c4102aad338f2e228fb0b [id=a4d0af97df5949cca940a81a41b8e454]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.413431 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 64fb46b942ff4e4c8749986ecc3dfe65 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:19.414220 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dabebc77f72e4bb19e264be2f496202a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.419523 1033 tablet_service.cc:1515] Processing DeleteTablet for tablet 74b709cb5ec64006b2f12bead8217d80 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:19.419946 893 tablet_service.cc:1468] Processing CreateTablet for tablet 858218c4427c41e980b2b5c969a5af40 (DEFAULT_TABLE table=default.Table_4185b4f73a2743e08f24a26878cf73ca [id=264cc469a73841e79538843479a165ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.418805 886 tablet_service.cc:1468] Processing CreateTablet for tablet 764bfeb2a7d14c40815b5ec1b07bdd62 (DEFAULT_TABLE table=default.Table_4401d83f75f8464da49d5b811d7dd89d [id=16143c8e6f484f9a873437e709d8466d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.420346 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 858218c4427c41e980b2b5c969a5af40. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.421006 896 tablet_service.cc:1468] Processing CreateTablet for tablet da5f69c5b0ca4257ad2ff7918ee2af2e (DEFAULT_TABLE table=default.Table_8f7db2ae75a24dfab71c2a95c6a0a404 [id=61f1e8a842844ed899e5913c76f362ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.421396 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da5f69c5b0ca4257ad2ff7918ee2af2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.430789 896 tablet_service.cc:1468] Processing CreateTablet for tablet f195b040a6ae4b7b83a8308ab0a12b37 (DEFAULT_TABLE table=default.Table_8b1086c9a4c44239851c586ea7267e3c [id=523de440fbab4bf687c0f6168e48513c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.431514 903 tablet_service.cc:1515] Processing DeleteTablet for tablet a54af99bc10045e8b0b6868e5aed98b4 with delete_type TABLET_DATA_DELETED (replaced by tablet 50472a7aa2f04e0fbd65ac6c86c34b3e) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:19.431818 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 764bfeb2a7d14c40815b5ec1b07bdd62. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.432336 903 tablet_service.cc:1468] Processing CreateTablet for tablet d0b7c84ec9394a20ad77add3808e555a (DEFAULT_TABLE table=default.Table_85b9e2f3f8f7400aa833201daae93dc3 [id=cac94430e07342dfa9b266df6472b6e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.433797 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 74b709cb5ec64006b2f12bead8217d80 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:19.434604 887 tablet_service.cc:1468] Processing CreateTablet for tablet 4fca1435d82447f289fda4035aa2ed89 (DEFAULT_TABLE table=default.Table_fbc57d539a2b41e59dfc5369ac8c92e1 [id=8e14f99474a1467baed19f5268f09c5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.435616 888 tablet_service.cc:1468] Processing CreateTablet for tablet 61ddcf86c9f34726939cc116434f0822 (DEFAULT_TABLE table=default.Table_277b91103c4d41d2a9f9424f99082b29 [id=f9dcf4937b1143248bfa9d88053fce17]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.435773 893 tablet_service.cc:1468] Processing CreateTablet for tablet 93bcac1fd95d44dc927abfca81692921 (DEFAULT_TABLE table=default.Table_b8f8ee74648c406eaf589d3ab80934db [id=b8f69313439543be9c11eea88dfa69c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.436252 899 tablet_service.cc:1468] Processing CreateTablet for tablet f98d1d8d660443cb8066a547e5d9b77f (DEFAULT_TABLE table=default.Table_583918346a424cb293d2c35bfa4a5b5f [id=1f677c66803b4cb1bf585cb621a4df6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.436630 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f195b040a6ae4b7b83a8308ab0a12b37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.436892 885 tablet_service.cc:1468] Processing CreateTablet for tablet 4b4aa063ffe948e69e47f03102581899 (DEFAULT_TABLE table=default.Table_14e7f1e7dd314cdcaee09ac2fdf28993 [id=76e521ff309947a58840a7135e783477]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.437251 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b4aa063ffe948e69e47f03102581899. 1 dirs total, 0 dirs full, 0 dirs failed
W20250901 01:52:19.437731 441 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet ad45c90ba0da43c9860745e534664a98 (table default.Table_1cd3f3fb6b244b3f8796bfca9f7622c8 [id=6fd739b0e725424ca844dbb652ea8028], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet ad45c90ba0da43c9860745e534664a98 consensus state has no leader
I20250901 01:52:19.440551 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93bcac1fd95d44dc927abfca81692921. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.440996 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f98d1d8d660443cb8066a547e5d9b77f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.441305 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61ddcf86c9f34726939cc116434f0822. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.441788 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fca1435d82447f289fda4035aa2ed89. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.442276 903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0b7c84ec9394a20ad77add3808e555a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.443375 900 tablet_service.cc:1468] Processing CreateTablet for tablet fecf7d538b4c4c619182d5eccb44dd8b (DEFAULT_TABLE table=default.Table_27b1a96341864be385ff5523f4f7cf2b [id=02d0056abe2146ac9af81ebf4447b99d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.444541 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 64fb46b942ff4e4c8749986ecc3dfe65 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:19.444355 894 tablet_service.cc:1468] Processing CreateTablet for tablet 0e4009963cfe42d6ba164fc828897f09 (DEFAULT_TABLE table=default.Table_749fcd4bf97d439c9fd49b138a52a3fd [id=24fffc7ebf5a47bdb37211910301acf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.445166 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 49d30da95b14400b845270122487a2a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:19.445644 890 tablet_service.cc:1468] Processing CreateTablet for tablet 2dacb608897d47d28512df87dc320927 (DEFAULT_TABLE table=default.Table_91214af783ff4aa3b6c3cbdb0358307e [id=94d314a62cf24de19d6e1fff9b3794ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.446105 901 tablet_service.cc:1468] Processing CreateTablet for tablet 5f5396045979450193d461ba1334a5c9 (DEFAULT_TABLE table=default.Table_a89cc8e48563460baabc34c821ceae35 [id=9c3930e9fcfe4b49ba4bea67a2b0ad07]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.446262 889 tablet_service.cc:1468] Processing CreateTablet for tablet 45f6e8ba99b74898a2922b36768c6c36 (DEFAULT_TABLE table=default.Table_e533a2595cc9420f9ab7a36bda6dc4a3 [id=9ce26dcf307149b595a43b6ac4562fba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.446519 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 6741c56499b44f648fb032c8cf50205b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:19.446888 895 tablet_service.cc:1515] Processing DeleteTablet for tablet b73e362321284df1b18ef755dfe930a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:15 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:19.447321 895 tablet_service.cc:1468] Processing CreateTablet for tablet 7f46e9ddadaf4ff986d8c991178daa0c (DEFAULT_TABLE table=default.Table_15c9f4f02eb24474936a43d705f90f7b [id=ea928501fe954cd0b81d38add4ef7128]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.447438 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 23af154a6f1c4e2e9352fb17c9024f10 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:15 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:19.447937 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 9473f0ef978c4bfaa41fb7b0729e8863 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:17 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:19.450220 892 tablet_service.cc:1468] Processing CreateTablet for tablet 2973b59bb1df41d18d9a5b6a29ae336f (DEFAULT_TABLE table=default.Table_202b4c5ccc56474a8b2e7b9d7c4a28d6 [id=746845bd84c6444b8a1e5683eaba4a68]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.450639 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2973b59bb1df41d18d9a5b6a29ae336f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.454232 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b9d79a25452410891c09fd3f7c4be6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.455483 1711 raft_consensus.cc:491] T 23af154a6f1c4e2e9352fb17c9024f10 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:19.455693 1711 raft_consensus.cc:513] T 23af154a6f1c4e2e9352fb17c9024f10 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:19.457152 1711 leader_election.cc:290] T 23af154a6f1c4e2e9352fb17c9024f10 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:19.460363 455 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_b23a4778615d4597950ce1709ab00d37" } modify_external_catalogs: true
I20250901 01:52:19.460594 455 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_b23a4778615d4597950ce1709ab00d37" } modify_external_catalogs: true
I20250901 01:52:19.461679 1033 tablet_service.cc:1468] Processing CreateTablet for tablet 5f5396045979450193d461ba1334a5c9 (DEFAULT_TABLE table=default.Table_a89cc8e48563460baabc34c821ceae35 [id=9c3930e9fcfe4b49ba4bea67a2b0ad07]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.467020 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet 6741c56499b44f648fb032c8cf50205b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:19.467339 449 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_c0977985d6ee4b21a086dc892c860d6a" } modify_external_catalogs: true
I20250901 01:52:19.467543 449 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_c0977985d6ee4b21a086dc892c860d6a" } modify_external_catalogs: true
I20250901 01:52:19.468369 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81ce448b148447ddbc41875d863e0518. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.468490 450 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_a210833682434f928e7da36d2fb4e09c" } modify_external_catalogs: true
I20250901 01:52:19.468683 450 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_a210833682434f928e7da36d2fb4e09c" } modify_external_catalogs: true
I20250901 01:52:19.469785 448 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_0ece0b3619db43da9d3bb2b0a72a7ef9" } modify_external_catalogs: true
I20250901 01:52:19.469765 454 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_14cf61059c5145b0a02159aae2a3e457" } modify_external_catalogs: true
I20250901 01:52:19.469969 448 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_0ece0b3619db43da9d3bb2b0a72a7ef9" } modify_external_catalogs: true
I20250901 01:52:19.470098 454 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_14cf61059c5145b0a02159aae2a3e457" } modify_external_catalogs: true
W20250901 01:52:19.472258 858 leader_election.cc:336] T 23af154a6f1c4e2e9352fb17c9024f10 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:19.473453 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2dacb608897d47d28512df87dc320927. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.473706 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fecf7d538b4c4c619182d5eccb44dd8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.473897 901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f5396045979450193d461ba1334a5c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.474223 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e4009963cfe42d6ba164fc828897f09. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.477950 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f46e9ddadaf4ff986d8c991178daa0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.477239 1036 tablet_service.cc:1468] Processing CreateTablet for tablet 45f6e8ba99b74898a2922b36768c6c36 (DEFAULT_TABLE table=default.Table_e533a2595cc9420f9ab7a36bda6dc4a3 [id=9ce26dcf307149b595a43b6ac4562fba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.480592 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45f6e8ba99b74898a2922b36768c6c36. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.482555 1031 tablet_service.cc:1468] Processing CreateTablet for tablet fecf7d538b4c4c619182d5eccb44dd8b (DEFAULT_TABLE table=default.Table_27b1a96341864be385ff5523f4f7cf2b [id=02d0056abe2146ac9af81ebf4447b99d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.482758 1037 tablet_service.cc:1468] Processing CreateTablet for tablet 7f46e9ddadaf4ff986d8c991178daa0c (DEFAULT_TABLE table=default.Table_15c9f4f02eb24474936a43d705f90f7b [id=ea928501fe954cd0b81d38add4ef7128]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.493561 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_0e2cd82b00f1466eaef7f287402e644d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:19.494233 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0e2cd82b00f1466eaef7f287402e644d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:19.496287 451 master_service.cc:948] ReplaceTablet: received request to replace tablet e2aa32450a584395ae07b49f76a7b26d from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:19.502832 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 435b1cca5a7d4d49959a310cb250b2a0 (DEFAULT_TABLE table=default.Table_dc9c5d18c7ec478592a76dbd76424b88 [id=126b298fef6c4653952d672adff1db7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.502882 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_cb4904990e284a1e8137d11272036154"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:19.503428 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cb4904990e284a1e8137d11272036154 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:19.505910 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 0e4009963cfe42d6ba164fc828897f09 (DEFAULT_TABLE table=default.Table_749fcd4bf97d439c9fd49b138a52a3fd [id=24fffc7ebf5a47bdb37211910301acf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.506886 901 tablet_service.cc:1515] Processing DeleteTablet for tablet 3fda93767e494a4684978c8522b829dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:15 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:19.506990 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_95515169b000454aa95025573560278f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:19.507371 893 tablet_service.cc:1468] Processing CreateTablet for tablet 435b1cca5a7d4d49959a310cb250b2a0 (DEFAULT_TABLE table=default.Table_dc9c5d18c7ec478592a76dbd76424b88 [id=126b298fef6c4653952d672adff1db7e]), partition=RANGE (key) PARTITION UNBOUNDED
W20250901 01:52:19.507557 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_95515169b000454aa95025573560278f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:19.507591 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 2dacb608897d47d28512df87dc320927 (DEFAULT_TABLE table=default.Table_91214af783ff4aa3b6c3cbdb0358307e [id=94d314a62cf24de19d6e1fff9b3794ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.507759 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 435b1cca5a7d4d49959a310cb250b2a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.508824 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_1d63a9f86fe4488b92d5d607b102328c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:19.509248 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1d63a9f86fe4488b92d5d607b102328c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:19.510119 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 4b4aa063ffe948e69e47f03102581899 (DEFAULT_TABLE table=default.Table_14e7f1e7dd314cdcaee09ac2fdf28993 [id=76e521ff309947a58840a7135e783477]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.510648 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 49d30da95b14400b845270122487a2a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:19.512138 447 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_8deed8083e3f4f4f8533e42022a8633d" } new_table_name: "default.Table_219f0092f43743039ae1a61fedb3e4d1" modify_external_catalogs: true
I20250901 01:52:19.516083 899 tablet_service.cc:1468] Processing CreateTablet for tablet 69f61ec283484f9d8b3eebcea27f741b (DEFAULT_TABLE table=default.Table_583e53c2e86544e9be68eb27ab6068a3 [id=f376eaf65bb549429aa71b03fca81647]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.516492 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69f61ec283484f9d8b3eebcea27f741b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.517469 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_0c9bfebabc22494489b1c1251a412c81"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:19.517943 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0c9bfebabc22494489b1c1251a412c81 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:19.520033 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 8381305e4c4148808803e255b8a43a90 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:17 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:19.520787 1035 tablet_service.cc:1515] Processing DeleteTablet for tablet d60d571b2c2840a2b20629bb832d1309 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:13 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:19.521613 1332 tablet_replica.cc:331] T 8381305e4c4148808803e255b8a43a90 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:19.524362 1669 raft_consensus.cc:491] T 5adf6b492640417ca071c94702b48cb9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:19.524945 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f5396045979450193d461ba1334a5c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.525352 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_d6b752d82c2b4be88b762e67bac0d5a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:19.525883 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d6b752d82c2b4be88b762e67bac0d5a1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250901 01:52:19.528187 447 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 246b33338de647689d7404c70676e76f consensus state has no leader
I20250901 01:52:19.528272 452 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_d6119d1604674319914fa8cb698f16c5" } new_table_name: "default.Table_b5d681f51fc84395bbabe1e5b4cfb358" modify_external_catalogs: true
I20250901 01:52:19.530839 901 tablet_service.cc:1515] Processing DeleteTablet for tablet 9d886fa63f8f4996b087997b542e8bb1 with delete_type TABLET_DATA_DELETED (replaced by tablet eaa521c52a2f4408b59770bd2a05907c) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:19.531669 901 tablet_service.cc:1468] Processing CreateTablet for tablet b2b172757e50404aae78e818921a7d9d (DEFAULT_TABLE table=default.Table_21b93b5fe2ce49c6bc637d8cab41fc06 [id=5419e36671ec470a98dff53acdbd4760]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.532187 901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2b172757e50404aae78e818921a7d9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.534838 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 3df908c3096a4df7bddd8ac88b49cfe0 with delete_type TABLET_DATA_DELETED (replaced by tablet b009cc95c819431ea9827a3e453b8da3) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:19.536391 892 tablet_service.cc:1468] Processing CreateTablet for tablet 44af4764d4f64e44aec84ccb84be6f5f (DEFAULT_TABLE table=default.Table_ac189f6a9fcf457e9fa0d77badc81594 [id=a541b55be0dc420a8353ef4793b925b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.536866 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44af4764d4f64e44aec84ccb84be6f5f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250901 01:52:19.540022 452 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 42c1156e8dd84c6fa1396b46de13eb6f consensus state has no leader
I20250901 01:52:19.541154 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 69f61ec283484f9d8b3eebcea27f741b (DEFAULT_TABLE table=default.Table_583e53c2e86544e9be68eb27ab6068a3 [id=f376eaf65bb549429aa71b03fca81647]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.542096 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b4aa063ffe948e69e47f03102581899. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.542146 890 tablet_service.cc:1468] Processing CreateTablet for tablet f23df6f253a943fa8e146bb8c811ef51 (DEFAULT_TABLE table=default.Table_59039012009944fdbe0179002ea44633 [id=218947d08fb24732b5d0375d4af973e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.542512 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f23df6f253a943fa8e146bb8c811ef51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.544457 900 tablet_service.cc:1468] Processing CreateTablet for tablet 36e10328c34242ebb4dbd8f294162696 (DEFAULT_TABLE table=default.Table_d51ef24ec0f846c7ad212360eb478cd7 [id=958e51a6bbee47638e504e1a44094737]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.544871 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36e10328c34242ebb4dbd8f294162696. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.547179 895 tablet_service.cc:1468] Processing CreateTablet for tablet d000ba05e12843b4925e622ab55471c0 (DEFAULT_TABLE table=default.Table_820e1c5a64cd4128b7ae1b70bd951964 [id=b43a92ed46164d199b445287ef039bc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.547540 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d000ba05e12843b4925e622ab55471c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.547498 889 tablet_service.cc:1468] Processing CreateTablet for tablet e1c8fe96a6ba4edba4f09c00f7add75e (DEFAULT_TABLE table=default.Table_f05ff31088d34354b72333af6d6c7235 [id=1c1a988d4e434fa88e7b5abded4419c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.547634 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 2973b59bb1df41d18d9a5b6a29ae336f (DEFAULT_TABLE table=default.Table_202b4c5ccc56474a8b2e7b9d7c4a28d6 [id=746845bd84c6444b8a1e5683eaba4a68]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.547818 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1c8fe96a6ba4edba4f09c00f7add75e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.548488 894 tablet_service.cc:1468] Processing CreateTablet for tablet 56f2c4d2e6ab4bb4bdd6d025843589b6 (DEFAULT_TABLE table=default.Table_c0977985d6ee4b21a086dc892c860d6a [id=f2ff184777494b45a70f9bc4f66a448b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.548818 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56f2c4d2e6ab4bb4bdd6d025843589b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.549530 1332 raft_consensus.cc:2241] T 8381305e4c4148808803e255b8a43a90 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:19.553738 887 tablet_service.cc:1468] Processing CreateTablet for tablet ae21b9ae2e3b472eb67fb5f96388a50f (DEFAULT_TABLE table=default.Table_d82e47b1bd8743649c6f3d31da751ed7 [id=db57b18a9ce24e068b362f5fbe940531]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.554113 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae21b9ae2e3b472eb67fb5f96388a50f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.554361 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_3dde393d3a564e84b8ef08c0d25265ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:19.554953 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3dde393d3a564e84b8ef08c0d25265ce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:19.557723 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_37e451883f5642f094a03b5ca48a0b25"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:19.558252 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_37e451883f5642f094a03b5ca48a0b25 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:19.560302 449 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 56f2c4d2e6ab4bb4bdd6d025843589b6
I20250901 01:52:19.560302 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 46bfbdff3f904715a8c24907b20fcab0
I20250901 01:52:19.565518 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_6295dbf840ff483a91edb67301e9b439"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:19.566151 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6295dbf840ff483a91edb67301e9b439 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:19.566637 449 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_983ec443070b4ffc8508ea825e30471a" } new_table_name: "default.Table_2ece91972d494680a8ad00273c10dee4" modify_external_catalogs: true
I20250901 01:52:19.576541 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 44af4764d4f64e44aec84ccb84be6f5f (DEFAULT_TABLE table=default.Table_ac189f6a9fcf457e9fa0d77badc81594 [id=a541b55be0dc420a8353ef4793b925b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.578941 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e4009963cfe42d6ba164fc828897f09. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.573230 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_0b769136b0664331945b30cfe0482fc6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:19.579790 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0b769136b0664331945b30cfe0482fc6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:19.580719 1332 raft_consensus.cc:2270] T 8381305e4c4148808803e255b8a43a90 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:19.581046 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 56f2c4d2e6ab4bb4bdd6d025843589b6 (DEFAULT_TABLE table=default.Table_c0977985d6ee4b21a086dc892c860d6a [id=f2ff184777494b45a70f9bc4f66a448b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.583935 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet e8e7d2a72f704f52a8f8d46911a8a9e4
I20250901 01:52:19.585842 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet bb0cf5a6a27045a08df6845d7473184d
I20250901 01:52:19.587949 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2dacb608897d47d28512df87dc320927. 1 dirs total, 0 dirs full, 0 dirs failed
W20250901 01:52:19.587970 449 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 33f637f6a8134215bb1995ec38440073 consensus state has no leader
I20250901 01:52:19.588618 1332 ts_tablet_manager.cc:1905] T 8381305e4c4148808803e255b8a43a90 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:19.585389 1669 raft_consensus.cc:513] T 5adf6b492640417ca071c94702b48cb9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:19.593274 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet e2aa32450a584395ae07b49f76a7b26d
I20250901 01:52:19.593724 451 catalog_manager.cc:6652] ReplaceTablet: tablet e2aa32450a584395ae07b49f76a7b26d deleted and replaced by tablet 7304937048ca48a7bd153f3cdcfaeff0
I20250901 01:52:19.599350 1669 leader_election.cc:290] T 5adf6b492640417ca071c94702b48cb9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:19.601469 1027 tablet_service.cc:1468] Processing CreateTablet for tablet f23df6f253a943fa8e146bb8c811ef51 (DEFAULT_TABLE table=default.Table_59039012009944fdbe0179002ea44633 [id=218947d08fb24732b5d0375d4af973e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.601990 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56f2c4d2e6ab4bb4bdd6d025843589b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.603475 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet eaa521c52a2f4408b59770bd2a05907c
I20250901 01:52:19.598232 1035 tablet_service.cc:1468] Processing CreateTablet for tablet b2b172757e50404aae78e818921a7d9d (DEFAULT_TABLE table=default.Table_21b93b5fe2ce49c6bc637d8cab41fc06 [id=5419e36671ec470a98dff53acdbd4760]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.605669 446 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_a133336a993b4c9aa51562985c3ce4f9" } new_table_name: "default.Table_faf7ef12324f454396e5ae64d64ebe51" modify_external_catalogs: true
I20250901 01:52:19.611207 454 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_f4482a2fa4cc4cfd9658b47dda711f2d" } new_table_name: "default.Table_ce1fd26ac65a4ed085d023e10a849e5c" modify_external_catalogs: true
I20250901 01:52:19.612056 903 tablet_service.cc:1468] Processing CreateTablet for tablet b4278a013ae84082bc7a8497b04ac147 (DEFAULT_TABLE table=default.Table_aa1bfa4e8b59486f96c587bc9cf84d73 [id=6b36921a6764455aa19284d36c3f8d32]), partition=RANGE (key) PARTITION UNBOUNDED
W20250901 01:52:19.615031 446 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3dae00aa38204ee3ac77325eb334be42 consensus state has no leader
I20250901 01:52:19.616545 886 tablet_service.cc:1468] Processing CreateTablet for tablet 3363fd67b1ad44849ecb7b4090b61a72 (DEFAULT_TABLE table=default.Table_09ffc2a7b4b14b25adbc5471363c02af [id=eaa8142db4844b908cf36991898aca3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.616951 891 tablet_service.cc:1468] Processing CreateTablet for tablet e9f1f892658b4736859ff507536920a1 (DEFAULT_TABLE table=default.Table_8825dd28afbe464193eb1b668cc9326a [id=9aa62cb634314a0a849612c54c1220d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.617158 885 tablet_service.cc:1468] Processing CreateTablet for tablet 34ebb441c27747ec860ff2e25f9a0adc (DEFAULT_TABLE table=default.Table_da6f34d486984dedbc970080cf14482b [id=b572fb895e3248978514cf759bc531d4]), partition=RANGE (key) PARTITION UNBOUNDED
W20250901 01:52:19.617666 454 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 617cc33d0d23439cb5f1e0c31bf11281 consensus state has no leader
I20250901 01:52:19.621320 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f23df6f253a943fa8e146bb8c811ef51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.623402 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fecf7d538b4c4c619182d5eccb44dd8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.629434 897 tablet_service.cc:1468] Processing CreateTablet for tablet f8d9bbf0c5be462a8ddc539238cbdf97 (DEFAULT_TABLE table=default.Table_dfb6e260190a40df8db77152b962cb8e [id=0e10c4b90e884aaeb0a75dbcf3dfd1bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.629516 898 tablet_service.cc:1468] Processing CreateTablet for tablet 75a703c8e20a4a2fa0ab6ec58d41ef92 (DEFAULT_TABLE table=default.Table_2f09538b6fbe401f909105decb56e1cb [id=a90d1f10531042c3bb7a196b8be61c5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.630576 896 tablet_service.cc:1468] Processing CreateTablet for tablet 6ba8241644984394ada95b561e76467e (DEFAULT_TABLE table=default.Table_a96f01d4ce3c4694a635575ffaa4f02d [id=5dc1ab36f2974f07866b14fedbdabd57]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.629540 888 tablet_service.cc:1468] Processing CreateTablet for tablet d4c9aa723aee4dc4b6237cb5c3722b52 (DEFAULT_TABLE table=default.Table_5e65c5f068b84995a2d7a76fa65dda0f [id=b9d81ddd38fe47838877b010be2b40d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.635087 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2b172757e50404aae78e818921a7d9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.631541 1032 tablet_service.cc:1468] Processing CreateTablet for tablet 36e10328c34242ebb4dbd8f294162696 (DEFAULT_TABLE table=default.Table_d51ef24ec0f846c7ad212360eb478cd7 [id=958e51a6bbee47638e504e1a44094737]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.636438 1022 tablet_service.cc:1468] Processing CreateTablet for tablet e1c8fe96a6ba4edba4f09c00f7add75e (DEFAULT_TABLE table=default.Table_f05ff31088d34354b72333af6d6c7235 [id=1c1a988d4e434fa88e7b5abded4419c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.635490 902 tablet_service.cc:1468] Processing CreateTablet for tablet 50472a7aa2f04e0fbd65ac6c86c34b3e (DEFAULT_TABLE table=default.Table_489b87a28dac48d093c241f301d5cccc [id=5fdd352d5e5e47a3a2bd8fd608a263dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.637037 904 tablet_service.cc:1468] Processing CreateTablet for tablet 508406601c994e42a5c5b1f665e74957 (DEFAULT_TABLE table=default.Table_df96f1aa51644aed967bdfd4f75588f6 [id=4aabc328f6374ec5b1dea26f778208b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.635684 895 tablet_service.cc:1468] Processing CreateTablet for tablet 088ef4861dc14f81ae470c3aaee5b4e1 (DEFAULT_TABLE table=default.Table_9e04f976a93f4d30baccf90fa3622dfe [id=d68960dbe1154bd8a8016f9a5718e5ad]), partition=RANGE (key) PARTITION UNBOUNDED
W20250901 01:52:19.642197 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:40561, user_credentials={real_user=slave}} blocked reactor thread for 154488us
I20250901 01:52:19.645558 903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4278a013ae84082bc7a8497b04ac147. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.653534 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2973b59bb1df41d18d9a5b6a29ae336f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.663618 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36e10328c34242ebb4dbd8f294162696. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.664079 1019 tablet_service.cc:1468] Processing CreateTablet for tablet d000ba05e12843b4925e622ab55471c0 (DEFAULT_TABLE table=default.Table_820e1c5a64cd4128b7ae1b70bd951964 [id=b43a92ed46164d199b445287ef039bc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.664935 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 435b1cca5a7d4d49959a310cb250b2a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.667219 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1c8fe96a6ba4edba4f09c00f7add75e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.669595 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d000ba05e12843b4925e622ab55471c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.672713 893 tablet_service.cc:1468] Processing CreateTablet for tablet 88ceba7f48b04d7885cc0164668700a5 (DEFAULT_TABLE table=default.Table_53e30205c5fd488eb3b48c685a159d04 [id=60837b865ee7415ea42a85a953d837a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.675980 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9f1f892658b4736859ff507536920a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.678476 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69f61ec283484f9d8b3eebcea27f741b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.682121 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 088ef4861dc14f81ae470c3aaee5b4e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.683104 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88ceba7f48b04d7885cc0164668700a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.683349 904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 508406601c994e42a5c5b1f665e74957. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.688032 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75a703c8e20a4a2fa0ab6ec58d41ef92. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.689953 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8d9bbf0c5be462a8ddc539238cbdf97. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.692234 454 master_service.cc:948] ReplaceTablet: received request to replace tablet 822382a1781b4304bce29355ce83730e from {username='slave'} at 127.0.0.1:57810
I20250901 01:52:19.699360 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 822382a1781b4304bce29355ce83730e
I20250901 01:52:19.699889 454 catalog_manager.cc:6652] ReplaceTablet: tablet 822382a1781b4304bce29355ce83730e deleted and replaced by tablet 0958b0a4068344abb0f9db601d7dd2ad
W20250901 01:52:19.704846 440 catalog_manager.cc:4671] TS 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a7ff996482164776b468cf305989866b: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:19.705768 899 tablet_service.cc:1515] Processing DeleteTablet for tablet 56f2c4d2e6ab4bb4bdd6d025843589b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:19.707574 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_c714bf77d32b42aeb225bbc5b9a6f43b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:19.708256 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c714bf77d32b42aeb225bbc5b9a6f43b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:19.714669 1306 ts_tablet_manager.cc:1918] T c95798f0daf4437db18a291c4464522e P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:19.714857 1306 log.cc:1199] T c95798f0daf4437db18a291c4464522e P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/c95798f0daf4437db18a291c4464522e
I20250901 01:52:19.717082 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50472a7aa2f04e0fbd65ac6c86c34b3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.720971 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_5e8920128d214deb96ef83f369a8b3f1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:19.721545 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5e8920128d214deb96ef83f369a8b3f1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:19.722858 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3363fd67b1ad44849ecb7b4090b61a72. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.722909 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_2227d0e5eae04a08853aa24bfe533ee5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:19.723423 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2227d0e5eae04a08853aa24bfe533ee5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:19.730700 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_7f273a7efc7c496d8d8f9a49871026f4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:19.731309 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7f273a7efc7c496d8d8f9a49871026f4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:19.732667 450 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_dd8e6f9348c440bca1569cf68eec5beb" } new_table_name: "default.Table_1c8baaaa506548bd85b9f6a7483e5c7a" modify_external_catalogs: true
I20250901 01:52:19.736199 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_37dedd79eacd4c56ac546fe8b1902259"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:19.736756 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_37dedd79eacd4c56ac546fe8b1902259 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:19.741305 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_16ca44aa872f426f858aa71d8491b750"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:19.741808 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_16ca44aa872f426f858aa71d8491b750 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:19.743575 451 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_78ccd6cda2e540bd8eadf559a5c0327b" } new_table_name: "default.Table_5f4ab26cf337468fa2ac8e5a6713bf27" modify_external_catalogs: true
W20250901 01:52:19.751031 450 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fcb7b5b7d63d4f0b906fea97cb548de2 consensus state has no leader
W20250901 01:52:19.759501 451 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 410d44e5f1974ed69e5dee65e98856a4 consensus state has no leader
I20250901 01:52:19.762465 454 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_9734ae6c72e240c698b66ea83c536249" } modify_external_catalogs: true
I20250901 01:52:19.762763 451 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_62ececf8293e4e22bad9e9568f496bbe" } modify_external_catalogs: true
I20250901 01:52:19.762944 451 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_62ececf8293e4e22bad9e9568f496bbe" } modify_external_catalogs: true
I20250901 01:52:19.762792 454 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_9734ae6c72e240c698b66ea83c536249" } modify_external_catalogs: true
I20250901 01:52:19.774394 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_ca5bada2f51949d790e9a38e25fa1bac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:19.775161 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ca5bada2f51949d790e9a38e25fa1bac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:19.779525 901 tablet_service.cc:1515] Processing DeleteTablet for tablet bb0cf5a6a27045a08df6845d7473184d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:19.779855 889 tablet_service.cc:1515] Processing DeleteTablet for tablet e2aa32450a584395ae07b49f76a7b26d with delete_type TABLET_DATA_DELETED (replaced by tablet 7304937048ca48a7bd153f3cdcfaeff0) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:19.781005 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f46e9ddadaf4ff986d8c991178daa0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.780973 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_ea8e4576448a476b8bf399421a378e2c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:19.781515 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ea8e4576448a476b8bf399421a378e2c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:19.782789 899 tablet_service.cc:1515] Processing DeleteTablet for tablet e8e7d2a72f704f52a8f8d46911a8a9e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:19.788017 455 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_f05ff31088d34354b72333af6d6c7235" } modify_external_catalogs: true
I20250901 01:52:19.788216 455 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_f05ff31088d34354b72333af6d6c7235" } modify_external_catalogs: true
I20250901 01:52:19.790647 450 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_7be95f26c04746828f38a19ddb67afc0" } modify_external_catalogs: true
I20250901 01:52:19.790850 450 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_7be95f26c04746828f38a19ddb67afc0" } modify_external_catalogs: true
I20250901 01:52:19.793516 453 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_531b3137a19f4061878ea96aae227517" } new_table_name: "default.Table_0b4ac0a1caa5472d909448e41318e679" modify_external_catalogs: true
I20250901 01:52:19.795540 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_da76b13a22bc47eea908959d5c1720df"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:19.796176 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_da76b13a22bc47eea908959d5c1720df in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:19.796873 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45f6e8ba99b74898a2922b36768c6c36. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.797057 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34ebb441c27747ec860ff2e25f9a0adc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.802716 1029 tablet_service.cc:1468] Processing CreateTablet for tablet ae21b9ae2e3b472eb67fb5f96388a50f (DEFAULT_TABLE table=default.Table_d82e47b1bd8743649c6f3d31da751ed7 [id=db57b18a9ce24e068b362f5fbe940531]), partition=RANGE (key) PARTITION UNBOUNDED
W20250901 01:52:19.805996 1151 thread.cc:641] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.735s user 0.156s sys 0.274s
W20250901 01:52:19.806180 1151 thread.cc:608] log-alloc [worker] (thread pool) Time spent starting thread: real 0.748s user 0.156s sys 0.274s
I20250901 01:52:19.806841 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_2d7b072b55354009b9cce1aef9408e35"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:19.807355 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2d7b072b55354009b9cce1aef9408e35 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:19.808480 452 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_e59cac5d3f0a4dd6b9846e7449b60005" } modify_external_catalogs: true
I20250901 01:52:19.808634 452 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_e59cac5d3f0a4dd6b9846e7449b60005" } modify_external_catalogs: true
I20250901 01:52:19.817201 1034 tablet_service.cc:1468] Processing CreateTablet for tablet 6ba8241644984394ada95b561e76467e (DEFAULT_TABLE table=default.Table_a96f01d4ce3c4694a635575ffaa4f02d [id=5dc1ab36f2974f07866b14fedbdabd57]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.817245 447 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_5f4ab26cf337468fa2ac8e5a6713bf27" } new_table_name: "default.Table_bec114a616504798bede9a9066e57192" modify_external_catalogs: true
I20250901 01:52:19.823009 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44af4764d4f64e44aec84ccb84be6f5f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250901 01:52:19.825881 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 4902062977ea42149e3466e999117322 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4902062977ea42149e3466e999117322 already in progress: creating tablet
I20250901 01:52:19.828230 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet e1c8fe96a6ba4edba4f09c00f7add75e
I20250901 01:52:19.828641 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet b85d64e02cc646e380581ac9f59dee9e
W20250901 01:52:19.829588 453 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d0c3c363dda444519a17dca63ad9caa7 consensus state has no leader
W20250901 01:52:19.832899 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 1a60c10e42104502ac2d3b80b8126ceb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1a60c10e42104502ac2d3b80b8126ceb already in progress: creating tablet
I20250901 01:52:19.833340 895 tablet_service.cc:1468] Processing CreateTablet for tablet 3dae00aa38204ee3ac77325eb334be42 (DEFAULT_TABLE table=default.Table_a133336a993b4c9aa51562985c3ce4f9 [id=5812902df9ea406ba59ff0ff23f130b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.834348 890 tablet_service.cc:1468] Processing CreateTablet for tablet eee34044c39d4013baf78d3705489626 (DEFAULT_TABLE table=default.Table_99afdac28a4e4ac9ae93b4fbbe704a4d [id=4639e91c14d8427d86c89d3c2b0918ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.836520 1711 raft_consensus.cc:491] T a6445b29591548be9f9577db1ba0bfa5 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:19.837383 449 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_537be4a82c0348f6952b911b167d02ad" } new_table_name: "default.Table_9e7ca61fcc324a8984bd1e470162c61f" modify_external_catalogs: true
W20250901 01:52:19.844252 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet a54af99bc10045e8b0b6868e5aed98b4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a54af99bc10045e8b0b6868e5aed98b4 already in progress: creating tablet
I20250901 01:52:19.844812 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 46bfbdff3f904715a8c24907b20fcab0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:52846
W20250901 01:52:19.846177 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 64fb46b942ff4e4c8749986ecc3dfe65 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64fb46b942ff4e4c8749986ecc3dfe65 already in progress: creating tablet
W20250901 01:52:19.846446 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 74b709cb5ec64006b2f12bead8217d80 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 74b709cb5ec64006b2f12bead8217d80 already in progress: creating tablet
W20250901 01:52:19.846613 447 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 410d44e5f1974ed69e5dee65e98856a4 consensus state has no leader
W20250901 01:52:19.846870 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 6741c56499b44f648fb032c8cf50205b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6741c56499b44f648fb032c8cf50205b already in progress: creating tablet
W20250901 01:52:19.850333 441 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 49d30da95b14400b845270122487a2a0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 49d30da95b14400b845270122487a2a0 already in progress: creating tablet
I20250901 01:52:19.854584 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae21b9ae2e3b472eb67fb5f96388a50f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250901 01:52:19.858328 449 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 081e2630e284498aa819f85bf8326315 consensus state has no leader
I20250901 01:52:19.869386 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ba8241644984394ada95b561e76467e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.869835 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3dae00aa38204ee3ac77325eb334be42. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.870703 1150 tablet_bootstrap.cc:492] T f62d96c37cca4143a4b098b7c69979e6 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:19.861326 1711 raft_consensus.cc:513] T a6445b29591548be9f9577db1ba0bfa5 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:19.876084 1150 ts_tablet_manager.cc:1397] T f62d96c37cca4143a4b098b7c69979e6 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.543s user 0.022s sys 0.076s
I20250901 01:52:19.876122 1711 leader_election.cc:290] T a6445b29591548be9f9577db1ba0bfa5 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:19.877084 1306 ts_tablet_manager.cc:1939] T c95798f0daf4437db18a291c4464522e P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:19.878927 1033 tablet_service.cc:1468] Processing CreateTablet for tablet 3363fd67b1ad44849ecb7b4090b61a72 (DEFAULT_TABLE table=default.Table_09ffc2a7b4b14b25adbc5471363c02af [id=eaa8142db4844b908cf36991898aca3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.879650 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ba8241644984394ada95b561e76467e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.888535 1150 raft_consensus.cc:357] T f62d96c37cca4143a4b098b7c69979e6 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:19.889104 1306 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52846 (request call id 431) took 1014 ms. Trace:
I20250901 01:52:19.890250 887 tablet_service.cc:1515] Processing DeleteTablet for tablet eaa521c52a2f4408b59770bd2a05907c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:19.890477 924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5adf6b492640417ca071c94702b48cb9" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:19.891299 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eee34044c39d4013baf78d3705489626. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.892737 452 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 31614db4c53a4f3c96fd190d326ce601
W20250901 01:52:19.896095 858 leader_election.cc:336] T a6445b29591548be9f9577db1ba0bfa5 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:19.896898 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4c9aa723aee4dc4b6237cb5c3722b52. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.904656 900 tablet_service.cc:1468] Processing CreateTablet for tablet 3a2bc5ad1cd243daa3c92fceca8ba414 (DEFAULT_TABLE table=default.Table_122585be3bdc45fb9498d02564fb8567 [id=e7b9f3937d0241b291db9ccbaa523da2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.906055 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 438cd35ff3f74c49b908d072560077db
I20250901 01:52:19.909880 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 75345ea2a3dc40fc8c9f8090d8839d2d
I20250901 01:52:19.911093 1306 rpcz_store.cc:276] 0901 01:52:18.869303 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:18.930044 (+ 60741us) service_pool.cc:225] Handling call
0901 01:52:19.184903 (+254859us) ts_tablet_manager.cc:1145] Deleting tablet c95798f0daf4437db18a291c4464522e
0901 01:52:19.184910 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:19.531776 (+346866us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:19.714473 (+182697us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:19.883292 (+168819us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":254722}
I20250901 01:52:19.911623 1306 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52846 (request call id 428) took 1090 ms. Trace:
I20250901 01:52:19.912401 924 raft_consensus.cc:2466] T 5adf6b492640417ca071c94702b48cb9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:19.914615 1150 raft_consensus.cc:383] T f62d96c37cca4143a4b098b7c69979e6 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:19.916275 1150 raft_consensus.cc:738] T f62d96c37cca4143a4b098b7c69979e6 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:19.912736 1306 rpcz_store.cc:276] 0901 01:52:18.821064 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:18.924613 (+103549us) service_pool.cc:225] Handling call
0901 01:52:19.911455 (+986842us) ts_tablet_manager.cc:1145] Deleting tablet 670d90a37b934579b353bb4e4820127b
0901 01:52:19.911464 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:19.911560 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":955461}
I20250901 01:52:19.921284 1306 tablet_replica.cc:331] T 824229183e024feaa975c7f0387c9c1a P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:19.923846 887 tablet_service.cc:1468] Processing CreateTablet for tablet 3697821cd29446bb908b020599df0db9 (DEFAULT_TABLE table=default.Table_efc44c841f0b4ae2a168be904751b65c [id=8106fbf3fbe244dabcc6f134e512a9d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.930269 891 tablet_service.cc:1468] Processing CreateTablet for tablet b8ca4e66c6a74f9d8d90a69be5d6a58f (DEFAULT_TABLE table=default.Table_259a716264db43c49671b6fef6665f0e [id=aebce875b7dd4718a3611eec4ca56917]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.930644 889 tablet_service.cc:1468] Processing CreateTablet for tablet 0cba5329ab31440999faec599d704633 (DEFAULT_TABLE table=default.Table_a5c783cbdb694af69047b31d99ff8def [id=3debd2adcd05421ea96dc426f739f8bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.940029 899 tablet_service.cc:1468] Processing CreateTablet for tablet c83ce95fd6fa41478de0e2168be201b1 (DEFAULT_TABLE table=default.Table_bb44f1b7c92140309a212038f8bf1286 [id=bfd87b9b708449d3b16e3e0639561a49]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.941030 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a2bc5ad1cd243daa3c92fceca8ba414. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.944521 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_3940656768d5406fb0e10a71f8f77003"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:19.945123 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3940656768d5406fb0e10a71f8f77003 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:19.950342 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_bfb1749ab3454806b139ff43f672583a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:19.950901 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bfb1749ab3454806b139ff43f672583a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:19.952570 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_54422fcb2b624e7fa8d42cba3558a327"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:19.953047 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_54422fcb2b624e7fa8d42cba3558a327 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:19.953459 1151 tablet_bootstrap.cc:492] T 154574e17f214cc0ad56e6e6be3af899 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:19.953644 1151 ts_tablet_manager.cc:1397] T 154574e17f214cc0ad56e6e6be3af899 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.898s user 0.157s sys 0.275s
I20250901 01:52:19.953759 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_1a9de416bd864f7ba7ed302579ebda5a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:19.954174 449 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_0ff9dd032f74401b967bb5e4ed86d3ec" } new_table_name: "default.Table_ce86974fd8974de9a5a737954fac2ad7" modify_external_catalogs: true
W20250901 01:52:19.954238 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1a9de416bd864f7ba7ed302579ebda5a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:19.957444 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0cba5329ab31440999faec599d704633. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.959242 1038 tablet_service.cc:1468] Processing CreateTablet for tablet b4278a013ae84082bc7a8497b04ac147 (DEFAULT_TABLE table=default.Table_aa1bfa4e8b59486f96c587bc9cf84d73 [id=6b36921a6764455aa19284d36c3f8d32]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.959990 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3363fd67b1ad44849ecb7b4090b61a72. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.961331 448 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_d82e47b1bd8743649c6f3d31da751ed7" } new_table_name: "default.Table_046289f393ff432aae0b033bdec1fb17" modify_external_catalogs: true
I20250901 01:52:19.961957 1332 ts_tablet_manager.cc:1918] T 8381305e4c4148808803e255b8a43a90 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:19.962393 1306 raft_consensus.cc:2241] T 824229183e024feaa975c7f0387c9c1a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:19.962600 1306 raft_consensus.cc:2270] T 824229183e024feaa975c7f0387c9c1a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:19.962810 892 tablet_service.cc:1468] Processing CreateTablet for tablet c422881b40f54971a68530abfbd1945b (DEFAULT_TABLE table=default.Table_f611c1794e564a0496a761da46171d0e [id=dc22dac261644596b2d65e804791462d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.963176 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_8ad4807386b64e4195236b6fe222ad8e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:19.962845 901 tablet_service.cc:1515] Processing DeleteTablet for tablet 822382a1781b4304bce29355ce83730e with delete_type TABLET_DATA_DELETED (replaced by tablet 0958b0a4068344abb0f9db601d7dd2ad) from {username='slave'} at 127.0.0.1:52846
W20250901 01:52:19.963654 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8ad4807386b64e4195236b6fe222ad8e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:19.963835 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_3552791ff8104f71a25c92ce0842e349"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:19.964232 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3552791ff8104f71a25c92ce0842e349 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:19.954583 1151 raft_consensus.cc:357] T 154574e17f214cc0ad56e6e6be3af899 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:19.969058 1306 ts_tablet_manager.cc:1905] T 824229183e024feaa975c7f0387c9c1a P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:19.970248 1332 log.cc:1199] T 8381305e4c4148808803e255b8a43a90 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/8381305e4c4148808803e255b8a43a90
I20250901 01:52:19.970656 1332 ts_tablet_manager.cc:1939] T 8381305e4c4148808803e255b8a43a90 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
W20250901 01:52:19.970961 449 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f80f13db501c42c7950499df8fcb6fc8 consensus state has no leader
I20250901 01:52:19.971654 1332 tablet_replica.cc:331] T d60d571b2c2840a2b20629bb832d1309 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:19.971886 1332 raft_consensus.cc:2241] T d60d571b2c2840a2b20629bb832d1309 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:19.972107 1332 raft_consensus.cc:2270] T d60d571b2c2840a2b20629bb832d1309 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:19.972671 1332 ts_tablet_manager.cc:1905] T d60d571b2c2840a2b20629bb832d1309 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:19.974331 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 088ef4861dc14f81ae470c3aaee5b4e1 (DEFAULT_TABLE table=default.Table_9e04f976a93f4d30baccf90fa3622dfe [id=d68960dbe1154bd8a8016f9a5718e5ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.974898 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4278a013ae84082bc7a8497b04ac147. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.957093 1150 consensus_queue.cc:260] T f62d96c37cca4143a4b098b7c69979e6 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:19.976315 448 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ae21b9ae2e3b472eb67fb5f96388a50f consensus state has no leader
I20250901 01:52:19.979422 1019 tablet_service.cc:1468] Processing CreateTablet for tablet d4c9aa723aee4dc4b6237cb5c3722b52 (DEFAULT_TABLE table=default.Table_5e65c5f068b84995a2d7a76fa65dda0f [id=b9d81ddd38fe47838877b010be2b40d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.979537 1151 raft_consensus.cc:383] T 154574e17f214cc0ad56e6e6be3af899 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:19.982986 886 tablet_service.cc:1515] Processing DeleteTablet for tablet b85d64e02cc646e380581ac9f59dee9e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:19.985232 904 tablet_service.cc:1515] Processing DeleteTablet for tablet e1c8fe96a6ba4edba4f09c00f7add75e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:19.985374 894 tablet_service.cc:1468] Processing CreateTablet for tablet ced4813805004bba9c4667a79ac201fc (DEFAULT_TABLE table=default.Table_c7bfa05d564246a6be270afb7a3f0bde [id=16030cb64ac64ff68762651bd095fed1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:19.987697 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c83ce95fd6fa41478de0e2168be201b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.988817 1150 ts_tablet_manager.cc:1428] T f62d96c37cca4143a4b098b7c69979e6 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.111s user 0.002s sys 0.000s
I20250901 01:52:19.989832 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 088ef4861dc14f81ae470c3aaee5b4e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:19.999769 1151 raft_consensus.cc:738] T 154574e17f214cc0ad56e6e6be3af899 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:20.001967 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_77f4810bc83c4d948531da2a45a3792e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:20.002621 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_77f4810bc83c4d948531da2a45a3792e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:20.003857 1150 tablet_bootstrap.cc:492] T 1286c0ab85704837a99b6e5a62184d77 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:20.015962 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 31614db4c53a4f3c96fd190d326ce601 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:20.017059 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c422881b40f54971a68530abfbd1945b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:20.023056 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ced4813805004bba9c4667a79ac201fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:20.025177 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3697821cd29446bb908b020599df0db9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:20.027325 901 tablet_service.cc:1515] Processing DeleteTablet for tablet 75345ea2a3dc40fc8c9f8090d8839d2d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:20.033309 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_97a5ec9ec34c4a89b0f917704b43e52b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:20.033986 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_97a5ec9ec34c4a89b0f917704b43e52b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:20.036762 447 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_fbc57d539a2b41e59dfc5369ac8c92e1" } modify_external_catalogs: true
I20250901 01:52:20.036937 447 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_fbc57d539a2b41e59dfc5369ac8c92e1" } modify_external_catalogs: true
I20250901 01:52:20.037427 904 tablet_service.cc:1515] Processing DeleteTablet for tablet 438cd35ff3f74c49b908d072560077db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:52846
I20250901 01:52:20.038147 448 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_27b1a96341864be385ff5523f4f7cf2b" } new_table_name: "default.Table_1b5729d9768546fe99eaafc4e2604f42" modify_external_catalogs: true
I20250901 01:52:20.039343 454 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_489b87a28dac48d093c241f301d5cccc" } modify_external_catalogs: true
I20250901 01:52:20.039526 454 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_489b87a28dac48d093c241f301d5cccc" } modify_external_catalogs: true
I20250901 01:52:20.042862 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4c9aa723aee4dc4b6237cb5c3722b52. 1 dirs total, 0 dirs full, 0 dirs failed
W20250901 01:52:20.046826 448 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fecf7d538b4c4c619182d5eccb44dd8b consensus state has no leader
I20250901 01:52:20.051314 1032 tablet_service.cc:1468] Processing CreateTablet for tablet 34ebb441c27747ec860ff2e25f9a0adc (DEFAULT_TABLE table=default.Table_da6f34d486984dedbc970080cf14482b [id=b572fb895e3248978514cf759bc531d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:20.053227 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34ebb441c27747ec860ff2e25f9a0adc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:20.054354 455 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_a3ff88e352884abdba91e6d03acbae22" } new_table_name: "default.Table_66ad08ce0b6840d4848bcddb304cf88b" modify_external_catalogs: true
I20250901 01:52:20.054404 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_75ba3ba70ca24a94920e7bce96190ff2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:20.054922 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_75ba3ba70ca24a94920e7bce96190ff2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:20.055646 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_23fb8cbcf1bd43fa8346f3c26c8b4c65"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:20.056140 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_23fb8cbcf1bd43fa8346f3c26c8b4c65 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:20.061654 452 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_2872910642aa48eca0350b2792008200" } new_table_name: "default.Table_03fb48eaab9b42858ab7cefb722e43a8" modify_external_catalogs: true
I20250901 01:52:20.053679 1151 consensus_queue.cc:260] T 154574e17f214cc0ad56e6e6be3af899 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
W20250901 01:52:20.071974 452 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d10c193452954c198bba686169b8025b consensus state has no leader
I20250901 01:52:20.081844 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "23af154a6f1c4e2e9352fb17c9024f10" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:20.084148 901 tablet_service.cc:1468] Processing CreateTablet for tablet 98c56eb62ad84f79b04f17a43d44c479 (DEFAULT_TABLE table=default.Table_cb4904990e284a1e8137d11272036154 [id=00e7f7fe6dc44f2787b9dd8e3cbf4a51]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:20.085079 886 tablet_service.cc:1468] Processing CreateTablet for tablet 2df2240631844defa433270a3829d65b (DEFAULT_TABLE table=default.Table_6295dbf840ff483a91edb67301e9b439 [id=597ca217b3f945d88f32297e970378a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:20.090238 1711 raft_consensus.cc:491] T 438cd35ff3f74c49b908d072560077db P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:20.091075 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8ca4e66c6a74f9d8d90a69be5d6a58f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:20.092981 1151 ts_tablet_manager.cc:1428] T 154574e17f214cc0ad56e6e6be3af899 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.111s user 0.002s sys 0.000s
I20250901 01:52:20.093950 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskddMNdm/build/asan/bin/kudu pid 424
I20250901 01:52:20.101076 904 tablet_service.cc:1468] Processing CreateTablet for tablet 7304937048ca48a7bd153f3cdcfaeff0 (DEFAULT_TABLE table=default.Table_7dd48000e9b24337b75ceb4c53b6be25 [id=8051adf4041f4a2686a0b8e3850ef29c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:20.102226 898 tablet_service.cc:1468] Processing CreateTablet for tablet 008d23f236c348b6be9aee718f8698af (DEFAULT_TABLE table=default.Table_0e2cd82b00f1466eaef7f287402e644d [id=0d9b514272444aa1914c4ff033b8f611]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:20.104377 485 generic_service.cc:196] Checking for leaks (request via RPC)
I20250901 01:52:20.103677 902 tablet_service.cc:1468] Processing CreateTablet for tablet 315c15cf0afc44d4a79cb29294281d60 (DEFAULT_TABLE table=default.Table_37e451883f5642f094a03b5ca48a0b25 [id=aed8d4b4fa6a4fbf850cf5718610108f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:20.106894 1030 tablet_service.cc:1468] Processing CreateTablet for tablet e9f1f892658b4736859ff507536920a1 (DEFAULT_TABLE table=default.Table_8825dd28afbe464193eb1b668cc9326a [id=9aa62cb634314a0a849612c54c1220d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:20.107929 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 50472a7aa2f04e0fbd65ac6c86c34b3e (DEFAULT_TABLE table=default.Table_489b87a28dac48d093c241f301d5cccc [id=5fdd352d5e5e47a3a2bd8fd608a263dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:20.108834 1151 tablet_bootstrap.cc:492] T 60285f2ba037433999f5455fb9cc7acd P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:20.111449 451 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_4401d83f75f8464da49d5b811d7dd89d" } modify_external_catalogs: true
I20250901 01:52:20.113359 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Sending DeleteTablet for 3 replicas of tablet 50472a7aa2f04e0fbd65ac6c86c34b3e
I20250901 01:52:20.127703 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a6445b29591548be9f9577db1ba0bfa5" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:20.132876 1711 raft_consensus.cc:513] T 438cd35ff3f74c49b908d072560077db P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:20.134538 897 tablet_service.cc:1468] Processing CreateTablet for tablet d0463a8d655d4db4800b4e1b221c8315 (DEFAULT_TABLE table=default.Table_0b769136b0664331945b30cfe0482fc6 [id=135c19f9795c4195bb211d0be5fba6a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:20.142756 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9f1f892658b4736859ff507536920a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:20.148732 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2df2240631844defa433270a3829d65b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:20.157133 1711 leader_election.cc:290] T 438cd35ff3f74c49b908d072560077db P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:20.164685 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50472a7aa2f04e0fbd65ac6c86c34b3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:20.168326 1031 tablet_service.cc:1468] Processing CreateTablet for tablet 75a703c8e20a4a2fa0ab6ec58d41ef92 (DEFAULT_TABLE table=default.Table_2f09538b6fbe401f909105decb56e1cb [id=a90d1f10531042c3bb7a196b8be61c5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:20.172070 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 508406601c994e42a5c5b1f665e74957 (DEFAULT_TABLE table=default.Table_df96f1aa51644aed967bdfd4f75588f6 [id=4aabc328f6374ec5b1dea26f778208b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:20.173614 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75a703c8e20a4a2fa0ab6ec58d41ef92. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:20.174476 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 008d23f236c348b6be9aee718f8698af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:20.170398 1027 tablet_service.cc:1468] Processing CreateTablet for tablet f8d9bbf0c5be462a8ddc539238cbdf97 (DEFAULT_TABLE table=default.Table_dfb6e260190a40df8db77152b962cb8e [id=0e10c4b90e884aaeb0a75dbcf3dfd1bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:20.176765 897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0463a8d655d4db4800b4e1b221c8315. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:20.177803 888 tablet_service.cc:1468] Processing CreateTablet for tablet d27c8d99528a42ed8dd1c5a7f373bbae (DEFAULT_TABLE table=default.Table_d6b752d82c2b4be88b762e67bac0d5a1 [id=ce82be9787f14aa6bcdcb070228abc51]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:20.182431 901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98c56eb62ad84f79b04f17a43d44c479. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:20.185019 887 tablet_service.cc:1468] Processing CreateTablet for tablet 438ae684af234c868a419fbab43b0b98 (DEFAULT_TABLE table=default.Table_95515169b000454aa95025573560278f [id=1442b92adbbe43a6b8580cce1bbfe389]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:20.186825 1150 tablet_bootstrap.cc:654] T 1286c0ab85704837a99b6e5a62184d77 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:20.187012 992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:40731, user_credentials={real_user=slave}} blocked reactor thread for 55274.6us
I20250901 01:52:20.189826 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 508406601c994e42a5c5b1f665e74957. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:20.196189 1711 raft_consensus.cc:491] T 3fda93767e494a4684978c8522b829dd P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:20.197992 902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 315c15cf0afc44d4a79cb29294281d60. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:20.198426 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 438ae684af234c868a419fbab43b0b98. 1 dirs total, 0 dirs full, 0 dirs failed
W20250901 01:52:20.200995 859 leader_election.cc:343] T 23af154a6f1c4e2e9352fb17c9024f10 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
W20250901 01:52:20.202088 858 leader_election.cc:336] T 438cd35ff3f74c49b908d072560077db P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:20.202509 859 leader_election.cc:304] T 23af154a6f1c4e2e9352fb17c9024f10 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:20.203922 903 tablet_service.cc:1468] Processing CreateTablet for tablet 5d68ea97c71942c7b55b967ba870e0d7 (DEFAULT_TABLE table=default.Table_1d63a9f86fe4488b92d5d607b102328c [id=36915a3e843340ba8ef3578ac35a9af4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:20.209659 893 tablet_service.cc:1468] Processing CreateTablet for tablet 5dd0c6b806574caf9126dad94833fcf7 (DEFAULT_TABLE table=default.Table_3dde393d3a564e84b8ef08c0d25265ce [id=649558231b1542b5bc2d90699a644b2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:20.210388 904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7304937048ca48a7bd153f3cdcfaeff0. 1 dirs total, 0 dirs full, 0 dirs failed
W20250901 01:52:20.213065 992 leader_election.cc:343] T dd694cfa09564d3da5d279c0861a4c32 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: dd694cfa09564d3da5d279c0861a4c32
I20250901 01:52:20.216887 1761 raft_consensus.cc:491] T a7ff996482164776b468cf305989866b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:20.217083 1761 raft_consensus.cc:513] T a7ff996482164776b468cf305989866b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:20.217945 1761 leader_election.cc:290] T a7ff996482164776b468cf305989866b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:20.217954 858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:40731, user_credentials={real_user=slave}} blocked reactor thread for 46700.4us
I20250901 01:52:20.218299 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8d9bbf0c5be462a8ddc539238cbdf97. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:20.218279 1711 raft_consensus.cc:513] T 3fda93767e494a4684978c8522b829dd P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:20.223402 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 88ceba7f48b04d7885cc0164668700a5 (DEFAULT_TABLE table=default.Table_53e30205c5fd488eb3b48c685a159d04 [id=60837b865ee7415ea42a85a953d837a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:20.223887 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88ceba7f48b04d7885cc0164668700a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:20.224049 1764 raft_consensus.cc:491] T 6c05eff658924ca4a9a1e7a37f917d67 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:20.224212 1764 raft_consensus.cc:513] T 6c05eff658924ca4a9a1e7a37f917d67 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:20.225112 1764 leader_election.cc:290] T 6c05eff658924ca4a9a1e7a37f917d67 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:20.227650 992 leader_election.cc:343] T d60d571b2c2840a2b20629bb832d1309 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: d60d571b2c2840a2b20629bb832d1309
I20250901 01:52:20.232455 1669 consensus_queue.cc:1035] T 7442c6ff699141e1aff208ebdc9d192a P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20250901 01:52:20.234272 885 tablet_service.cc:1468] Processing CreateTablet for tablet bd9b59f0864a497faf281edd6077668d (DEFAULT_TABLE table=default.Table_0c9bfebabc22494489b1c1251a412c81 [id=8b74044a78e0493dbaf1835f33334212]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:20.236331 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5dd0c6b806574caf9126dad94833fcf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:20.237891 903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d68ea97c71942c7b55b967ba870e0d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:20.240831 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d27c8d99528a42ed8dd1c5a7f373bbae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:20.241595 1711 leader_election.cc:290] T 3fda93767e494a4684978c8522b829dd P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:20.240262 1761 consensus_queue.cc:1035] T 4410da9e95454ecf95faf80221b0a71c P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.017s
I20250901 01:52:20.246315 1151 tablet_bootstrap.cc:654] T 60285f2ba037433999f5455fb9cc7acd P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:20.267377 1711 raft_consensus.cc:491] T 9d886fa63f8f4996b087997b542e8bb1 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:20.274065 992 leader_election.cc:304] T 5adf6b492640417ca071c94702b48cb9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:20.274238 1038 tablet_service.cc:1515] Processing DeleteTablet for tablet 56f2c4d2e6ab4bb4bdd6d025843589b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:20.284054 1033 tablet_service.cc:1468] Processing CreateTablet for tablet 3a2bc5ad1cd243daa3c92fceca8ba414 (DEFAULT_TABLE table=default.Table_122585be3bdc45fb9498d02564fb8567 [id=e7b9f3937d0241b291db9ccbaa523da2]), partition=RANGE (key) PARTITION UNBOUNDED
W20250901 01:52:20.273243 858 leader_election.cc:336] T 3fda93767e494a4684978c8522b829dd P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:20.284588 1765 raft_consensus.cc:2747] T 23af154a6f1c4e2e9352fb17c9024f10 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:20.275233 1711 raft_consensus.cc:513] T 9d886fa63f8f4996b087997b542e8bb1 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:20.284976 1038 tablet_service.cc:1468] Processing CreateTablet for tablet eee34044c39d4013baf78d3705489626 (DEFAULT_TABLE table=default.Table_99afdac28a4e4ac9ae93b4fbbe704a4d [id=4639e91c14d8427d86c89d3c2b0918ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:20.285423 1764 raft_consensus.cc:2802] T 5adf6b492640417ca071c94702b48cb9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:20.285889 1764 raft_consensus.cc:491] T 5adf6b492640417ca071c94702b48cb9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:20.286013 1764 raft_consensus.cc:3058] T 5adf6b492640417ca071c94702b48cb9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:20.286743 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a2bc5ad1cd243daa3c92fceca8ba414. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:20.287290 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eee34044c39d4013baf78d3705489626. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:20.288334 1035 tablet_service.cc:1468] Processing CreateTablet for tablet 3dae00aa38204ee3ac77325eb334be42 (DEFAULT_TABLE table=default.Table_a133336a993b4c9aa51562985c3ce4f9 [id=5812902df9ea406ba59ff0ff23f130b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:20.288920 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd9b59f0864a497faf281edd6077668d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:20.288988 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3dae00aa38204ee3ac77325eb334be42. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:20.299007 1711 leader_election.cc:290] T 9d886fa63f8f4996b087997b542e8bb1 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:20.298010 858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:40731, user_credentials={real_user=slave}} blocked reactor thread for 43955.7us
I20250901 01:52:20.300806 1711 raft_consensus.cc:491] T 9473f0ef978c4bfaa41fb7b0729e8863 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:20.300951 1711 raft_consensus.cc:513] T 9473f0ef978c4bfaa41fb7b0729e8863 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:20.302731 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet e8e7d2a72f704f52a8f8d46911a8a9e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:20.305931 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 46bfbdff3f904715a8c24907b20fcab0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:20.306433 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet e2aa32450a584395ae07b49f76a7b26d with delete_type TABLET_DATA_DELETED (replaced by tablet 7304937048ca48a7bd153f3cdcfaeff0) from {username='slave'} at 127.0.0.1:44246
W20250901 01:52:20.307550 858 leader_election.cc:336] T 9d886fa63f8f4996b087997b542e8bb1 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:20.307744 858 leader_election.cc:336] T 9473f0ef978c4bfaa41fb7b0729e8863 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:20.312579 1711 leader_election.cc:290] T 9473f0ef978c4bfaa41fb7b0729e8863 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:20.314745 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 3697821cd29446bb908b020599df0db9 (DEFAULT_TABLE table=default.Table_efc44c841f0b4ae2a168be904751b65c [id=8106fbf3fbe244dabcc6f134e512a9d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:20.315145 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3697821cd29446bb908b020599df0db9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:20.316021 1022 tablet_service.cc:1468] Processing CreateTablet for tablet ced4813805004bba9c4667a79ac201fc (DEFAULT_TABLE table=default.Table_c7bfa05d564246a6be270afb7a3f0bde [id=16030cb64ac64ff68762651bd095fed1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:20.318240 1037 tablet_service.cc:1468] Processing CreateTablet for tablet c83ce95fd6fa41478de0e2168be201b1 (DEFAULT_TABLE table=default.Table_bb44f1b7c92140309a212038f8bf1286 [id=bfd87b9b708449d3b16e3e0639561a49]), partition=RANGE (key) PARTITION UNBOUNDED
W20250901 01:52:20.319537 859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:39447, user_credentials={real_user=slave}} blocked reactor thread for 123353us
I20250901 01:52:20.319873 1024 tablet_service.cc:1468] Processing CreateTablet for tablet b8ca4e66c6a74f9d8d90a69be5d6a58f (DEFAULT_TABLE table=default.Table_259a716264db43c49671b6fef6665f0e [id=aebce875b7dd4718a3611eec4ca56917]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:20.327283 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 0cba5329ab31440999faec599d704633 (DEFAULT_TABLE table=default.Table_a5c783cbdb694af69047b31d99ff8def [id=3debd2adcd05421ea96dc426f739f8bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:20.327663 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c83ce95fd6fa41478de0e2168be201b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:20.327834 1028 tablet_service.cc:1468] Processing CreateTablet for tablet c422881b40f54971a68530abfbd1945b (DEFAULT_TABLE table=default.Table_f611c1794e564a0496a761da46171d0e [id=dc22dac261644596b2d65e804791462d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:20.328564 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet eaa521c52a2f4408b59770bd2a05907c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:20.336616 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 822382a1781b4304bce29355ce83730e with delete_type TABLET_DATA_DELETED (replaced by tablet 0958b0a4068344abb0f9db601d7dd2ad) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:20.341127 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet e1c8fe96a6ba4edba4f09c00f7add75e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:20.308272 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet bb0cf5a6a27045a08df6845d7473184d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:20.343290 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet b85d64e02cc646e380581ac9f59dee9e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:20.343793 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 315c15cf0afc44d4a79cb29294281d60 (DEFAULT_TABLE table=default.Table_37e451883f5642f094a03b5ca48a0b25 [id=aed8d4b4fa6a4fbf850cf5718610108f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:20.344503 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8ca4e66c6a74f9d8d90a69be5d6a58f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:20.344767 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0cba5329ab31440999faec599d704633. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:20.347332 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 438cd35ff3f74c49b908d072560077db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:20.347811 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c422881b40f54971a68530abfbd1945b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:20.350203 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 75345ea2a3dc40fc8c9f8090d8839d2d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:20.354022 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 315c15cf0afc44d4a79cb29294281d60. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:20.354453 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet 31614db4c53a4f3c96fd190d326ce601 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:20.361081 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ced4813805004bba9c4667a79ac201fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:20.361651 1032 tablet_service.cc:1468] Processing CreateTablet for tablet 98c56eb62ad84f79b04f17a43d44c479 (DEFAULT_TABLE table=default.Table_cb4904990e284a1e8137d11272036154 [id=00e7f7fe6dc44f2787b9dd8e3cbf4a51]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:20.358696 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 2df2240631844defa433270a3829d65b (DEFAULT_TABLE table=default.Table_6295dbf840ff483a91edb67301e9b439 [id=597ca217b3f945d88f32297e970378a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:20.365849 1332 ts_tablet_manager.cc:1918] T d60d571b2c2840a2b20629bb832d1309 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:20.369146 1764 raft_consensus.cc:513] T 5adf6b492640417ca071c94702b48cb9 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:20.382683 1332 log.cc:1199] T d60d571b2c2840a2b20629bb832d1309 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/d60d571b2c2840a2b20629bb832d1309
I20250901 01:52:20.387907 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2df2240631844defa433270a3829d65b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:20.391451 1151 tablet_bootstrap.cc:492] T 60285f2ba037433999f5455fb9cc7acd P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:20.391834 1151 ts_tablet_manager.cc:1397] T 60285f2ba037433999f5455fb9cc7acd P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.295s user 0.028s sys 0.020s
I20250901 01:52:20.395345 1764 leader_election.cc:290] T 5adf6b492640417ca071c94702b48cb9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:20.395735 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 008d23f236c348b6be9aee718f8698af (DEFAULT_TABLE table=default.Table_0e2cd82b00f1466eaef7f287402e644d [id=0d9b514272444aa1914c4ff033b8f611]), partition=RANGE (key) PARTITION UNBOUNDED
W20250901 01:52:20.413811 859 leader_election.cc:343] T a6445b29591548be9f9577db1ba0bfa5 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:20.418159 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98c56eb62ad84f79b04f17a43d44c479. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:20.419625 859 leader_election.cc:304] T a6445b29591548be9f9577db1ba0bfa5 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:20.419628 1034 tablet_service.cc:1468] Processing CreateTablet for tablet 7304937048ca48a7bd153f3cdcfaeff0 (DEFAULT_TABLE table=default.Table_7dd48000e9b24337b75ceb4c53b6be25 [id=8051adf4041f4a2686a0b8e3850ef29c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:20.421015 1306 ts_tablet_manager.cc:1918] T 824229183e024feaa975c7f0387c9c1a P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:20.421999 1769 raft_consensus.cc:2747] T a6445b29591548be9f9577db1ba0bfa5 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:20.427978 1306 log.cc:1199] T 824229183e024feaa975c7f0387c9c1a P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/824229183e024feaa975c7f0387c9c1a
I20250901 01:52:20.431623 923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c05eff658924ca4a9a1e7a37f917d67" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:20.431850 923 raft_consensus.cc:2466] T 6c05eff658924ca4a9a1e7a37f917d67 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:20.434257 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "438cd35ff3f74c49b908d072560077db" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:20.434522 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 008d23f236c348b6be9aee718f8698af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:20.393152 1151 raft_consensus.cc:357] T 60285f2ba037433999f5455fb9cc7acd P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:20.435412 1151 raft_consensus.cc:383] T 60285f2ba037433999f5455fb9cc7acd P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:20.435578 1151 raft_consensus.cc:738] T 60285f2ba037433999f5455fb9cc7acd P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:20.435649 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7304937048ca48a7bd153f3cdcfaeff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:20.436597 1768 raft_consensus.cc:491] T f62d96c37cca4143a4b098b7c69979e6 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:20.436604 1035 tablet_service.cc:1468] Processing CreateTablet for tablet 5d68ea97c71942c7b55b967ba870e0d7 (DEFAULT_TABLE table=default.Table_1d63a9f86fe4488b92d5d607b102328c [id=36915a3e843340ba8ef3578ac35a9af4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:20.436983 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d68ea97c71942c7b55b967ba870e0d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:20.437626 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 5dd0c6b806574caf9126dad94833fcf7 (DEFAULT_TABLE table=default.Table_3dde393d3a564e84b8ef08c0d25265ce [id=649558231b1542b5bc2d90699a644b2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:20.437665 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3fda93767e494a4684978c8522b829dd" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:20.438062 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5dd0c6b806574caf9126dad94833fcf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:20.439162 1768 raft_consensus.cc:513] T f62d96c37cca4143a4b098b7c69979e6 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:20.428162 1332 ts_tablet_manager.cc:1939] T d60d571b2c2840a2b20629bb832d1309 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:20.440331 1768 leader_election.cc:290] T f62d96c37cca4143a4b098b7c69979e6 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:20.450349 924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5adf6b492640417ca071c94702b48cb9" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:20.451100 1306 ts_tablet_manager.cc:1939] T 824229183e024feaa975c7f0387c9c1a P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
W20250901 01:52:20.451898 858 leader_election.cc:336] T f62d96c37cca4143a4b098b7c69979e6 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:20.455588 1306 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52846 (request call id 432) took 1352 ms. Trace:
I20250901 01:52:20.455730 1306 rpcz_store.cc:276] 0901 01:52:19.098708 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:19.098916 (+ 208us) service_pool.cc:225] Handling call
0901 01:52:19.921227 (+822311us) ts_tablet_manager.cc:1145] Deleting tablet 824229183e024feaa975c7f0387c9c1a
0901 01:52:19.921236 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:20.211596 (+290360us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:20.412859 (+201263us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:20.451431 (+ 38572us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":822052}
I20250901 01:52:20.456202 1306 tablet_replica.cc:331] T ef58f477ebeb49639ed7d071e1a62c10 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:20.456470 1306 raft_consensus.cc:2241] T ef58f477ebeb49639ed7d071e1a62c10 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:20.456699 1306 raft_consensus.cc:2270] T ef58f477ebeb49639ed7d071e1a62c10 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:20.457440 1306 ts_tablet_manager.cc:1905] T ef58f477ebeb49639ed7d071e1a62c10 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:20.460677 924 raft_consensus.cc:3058] T 5adf6b492640417ca071c94702b48cb9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:20.463817 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d886fa63f8f4996b087997b542e8bb1" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:20.440536 1036 tablet_service.cc:1468] Processing CreateTablet for tablet bd9b59f0864a497faf281edd6077668d (DEFAULT_TABLE table=default.Table_0c9bfebabc22494489b1c1251a412c81 [id=8b74044a78e0493dbaf1835f33334212]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:20.484465 1023 tablet_service.cc:1468] Processing CreateTablet for tablet d27c8d99528a42ed8dd1c5a7f373bbae (DEFAULT_TABLE table=default.Table_d6b752d82c2b4be88b762e67bac0d5a1 [id=ce82be9787f14aa6bcdcb070228abc51]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:20.464586 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 438ae684af234c868a419fbab43b0b98 (DEFAULT_TABLE table=default.Table_95515169b000454aa95025573560278f [id=1442b92adbbe43a6b8580cce1bbfe389]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:20.486387 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9473f0ef978c4bfaa41fb7b0729e8863" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:20.488476 1332 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44246 (request call id 461) took 1022 ms. Trace:
I20250901 01:52:20.489676 1151 consensus_queue.cc:260] T 60285f2ba037433999f5455fb9cc7acd P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:20.490330 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f62d96c37cca4143a4b098b7c69979e6" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:20.490370 1764 consensus_queue.cc:1035] T 34042ea5c1b645969c1de41fe587c343 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20250901 01:52:20.490864 1761 consensus_queue.cc:1035] T 186ec810d02e4748aad15228b5866ac1 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20250901 01:52:20.490833 1332 rpcz_store.cc:276] 0901 01:52:19.450441 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:19.509900 (+ 59459us) service_pool.cc:225] Handling call
0901 01:52:19.971567 (+461667us) ts_tablet_manager.cc:1145] Deleting tablet d60d571b2c2840a2b20629bb832d1309
0901 01:52:19.971578 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:20.220771 (+249193us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:20.360959 (+140188us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:20.472894 (+111935us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":450629}
I20250901 01:52:20.491868 1332 tablet_replica.cc:331] T 3df908c3096a4df7bddd8ac88b49cfe0 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:20.492081 1761 consensus_queue.cc:1035] T fcb7b5b7d63d4f0b906fea97cb548de2 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250901 01:52:20.493036 992 leader_election.cc:343] T a7ff996482164776b468cf305989866b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: a7ff996482164776b468cf305989866b
I20250901 01:52:20.495682 992 leader_election.cc:304] T 6c05eff658924ca4a9a1e7a37f917d67 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:20.497048 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd9b59f0864a497faf281edd6077668d. 1 dirs total, 0 dirs full, 0 dirs failed
W20250901 01:52:20.497745 992 consensus_peers.cc:489] T 7442c6ff699141e1aff208ebdc9d192a P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:20.496985 1024 tablet_service.cc:1468] Processing CreateTablet for tablet d0463a8d655d4db4800b4e1b221c8315 (DEFAULT_TABLE table=default.Table_0b769136b0664331945b30cfe0482fc6 [id=135c19f9795c4195bb211d0be5fba6a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250901 01:52:20.504372 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 438ae684af234c868a419fbab43b0b98. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:20.505892 1151 ts_tablet_manager.cc:1428] T 60285f2ba037433999f5455fb9cc7acd P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.107s user 0.001s sys 0.001s
I20250901 01:52:20.508679 1761 raft_consensus.cc:2802] T 6c05eff658924ca4a9a1e7a37f917d67 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:20.511183 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d27c8d99528a42ed8dd1c5a7f373bbae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:20.516577 1761 raft_consensus.cc:491] T 6c05eff658924ca4a9a1e7a37f917d67 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:20.518828 1332 raft_consensus.cc:2241] T 3df908c3096a4df7bddd8ac88b49cfe0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:20.520005 1150 tablet_bootstrap.cc:492] T 1286c0ab85704837a99b6e5a62184d77 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:20.520179 1150 ts_tablet_manager.cc:1397] T 1286c0ab85704837a99b6e5a62184d77 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.531s user 0.017s sys 0.020s
I20250901 01:52:20.521050 1150 raft_consensus.cc:357] T 1286c0ab85704837a99b6e5a62184d77 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:20.521242 1150 raft_consensus.cc:383] T 1286c0ab85704837a99b6e5a62184d77 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:20.521325 1150 raft_consensus.cc:738] T 1286c0ab85704837a99b6e5a62184d77 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:20.526450 1332 raft_consensus.cc:2270] T 3df908c3096a4df7bddd8ac88b49cfe0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:20.526683 1151 tablet_bootstrap.cc:492] T b541a6e14f114ab29a864cca11806316 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:20.527091 924 raft_consensus.cc:2466] T 5adf6b492640417ca071c94702b48cb9 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:20.527407 1306 ts_tablet_manager.cc:1918] T ef58f477ebeb49639ed7d071e1a62c10 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:20.527683 1306 log.cc:1199] T ef58f477ebeb49639ed7d071e1a62c10 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/ef58f477ebeb49639ed7d071e1a62c10
I20250901 01:52:20.528338 1306 ts_tablet_manager.cc:1939] T ef58f477ebeb49639ed7d071e1a62c10 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:20.528750 1306 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52846 (request call id 433) took 1429 ms. Trace:
I20250901 01:52:20.528882 1306 rpcz_store.cc:276] 0901 01:52:19.098871 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:19.099213 (+ 342us) service_pool.cc:225] Handling call
0901 01:52:20.456084 (+1356871us) ts_tablet_manager.cc:1145] Deleting tablet ef58f477ebeb49639ed7d071e1a62c10
0901 01:52:20.456097 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:20.508060 (+ 51963us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:20.514904 (+ 6844us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:20.528702 (+ 13798us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1356692}
W20250901 01:52:20.529467 992 consensus_peers.cc:489] T fcb7b5b7d63d4f0b906fea97cb548de2 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:20.531930 1150 consensus_queue.cc:260] T 1286c0ab85704837a99b6e5a62184d77 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:20.532842 1306 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52846 (request call id 440) took 1164 ms. Trace:
I20250901 01:52:20.534267 1761 raft_consensus.cc:3058] T 6c05eff658924ca4a9a1e7a37f917d67 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:20.537567 1306 rpcz_store.cc:276] 0901 01:52:19.368375 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:19.368445 (+ 70us) service_pool.cc:225] Handling call
0901 01:52:20.532625 (+1164180us) ts_tablet_manager.cc:1145] Deleting tablet 4902062977ea42149e3466e999117322
0901 01:52:20.532639 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:20.532805 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1163960}
I20250901 01:52:20.538265 1332 ts_tablet_manager.cc:1905] T 3df908c3096a4df7bddd8ac88b49cfe0 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:20.539011 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0463a8d655d4db4800b4e1b221c8315. 1 dirs total, 0 dirs full, 0 dirs failed
I20250901 01:52:20.539625 1150 ts_tablet_manager.cc:1428] T 1286c0ab85704837a99b6e5a62184d77 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.019s user 0.000s sys 0.002s
W20250901 01:52:20.539829 992 leader_election.cc:336] T 5adf6b492640417ca071c94702b48cb9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:20.540364 1306 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52846 (request call id 444) took 1163 ms. Trace:
I20250901 01:52:20.540503 1306 rpcz_store.cc:276] 0901 01:52:19.376594 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:19.376860 (+ 266us) service_pool.cc:225] Handling call
0901 01:52:20.540219 (+1163359us) ts_tablet_manager.cc:1145] Deleting tablet 1a60c10e42104502ac2d3b80b8126ceb
0901 01:52:20.540230 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:20.540330 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1163160}
I20250901 01:52:20.542261 1150 tablet_bootstrap.cc:492] T 3a8231590ea643b0a1f83eec4d9a8191 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:20.543987 1306 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52846 (request call id 453) took 1112 ms. Trace:
I20250901 01:52:20.544086 1306 rpcz_store.cc:276] 0901 01:52:19.431351 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:19.431405 (+ 54us) service_pool.cc:225] Handling call
0901 01:52:20.543827 (+1112422us) ts_tablet_manager.cc:1145] Deleting tablet a54af99bc10045e8b0b6868e5aed98b4
0901 01:52:20.543839 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:20.543959 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1112155}
I20250901 01:52:20.545055 1306 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52846 (request call id 457) took 1111 ms. Trace:
I20250901 01:52:20.545166 1306 rpcz_store.cc:276] 0901 01:52:19.433676 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:19.433730 (+ 54us) service_pool.cc:225] Handling call
0901 01:52:20.544937 (+1111207us) ts_tablet_manager.cc:1145] Deleting tablet 74b709cb5ec64006b2f12bead8217d80
0901 01:52:20.544946 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:20.545024 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1110993}
I20250901 01:52:20.549922 1306 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52846 (request call id 463) took 1105 ms. Trace:
I20250901 01:52:20.550454 1306 rpcz_store.cc:276] 0901 01:52:19.444433 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:19.444478 (+ 45us) service_pool.cc:225] Handling call
0901 01:52:20.549764 (+1105286us) ts_tablet_manager.cc:1145] Deleting tablet 64fb46b942ff4e4c8749986ecc3dfe65
0901 01:52:20.549777 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:20.549885 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1105081}
I20250901 01:52:20.555974 1306 tablet_replica.cc:331] T 49d30da95b14400b845270122487a2a0 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:20.556471 1306 raft_consensus.cc:2241] T 49d30da95b14400b845270122487a2a0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:20.557006 1306 raft_consensus.cc:2270] T 49d30da95b14400b845270122487a2a0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:20.547019 1150 tablet_bootstrap.cc:654] T 3a8231590ea643b0a1f83eec4d9a8191 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:20.559578 992 consensus_peers.cc:489] T 34042ea5c1b645969c1de41fe587c343 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:20.560711 1306 ts_tablet_manager.cc:1905] T 49d30da95b14400b845270122487a2a0 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:20.570256 992 leader_election.cc:336] T dd694cfa09564d3da5d279c0861a4c32 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:20.578790 992 leader_election.cc:304] T dd694cfa09564d3da5d279c0861a4c32 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:20.579160 1771 raft_consensus.cc:2747] T dd694cfa09564d3da5d279c0861a4c32 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:20.579233 992 leader_election.cc:336] T d60d571b2c2840a2b20629bb832d1309 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:20.579375 992 leader_election.cc:304] T d60d571b2c2840a2b20629bb832d1309 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
W20250901 01:52:20.579537 992 raft_consensus.cc:2684] T d60d571b2c2840a2b20629bb832d1309 P fd27ab6665304aaea53adbd102dd4f58: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250901 01:52:20.580127 992 consensus_peers.cc:489] T 4410da9e95454ecf95faf80221b0a71c P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:20.581552 1150 tablet_bootstrap.cc:492] T 3a8231590ea643b0a1f83eec4d9a8191 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:20.581704 1150 ts_tablet_manager.cc:1397] T 3a8231590ea643b0a1f83eec4d9a8191 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.042s user 0.001s sys 0.003s
I20250901 01:52:20.582561 1150 raft_consensus.cc:357] T 3a8231590ea643b0a1f83eec4d9a8191 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:20.582772 1150 raft_consensus.cc:383] T 3a8231590ea643b0a1f83eec4d9a8191 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:20.582872 1150 raft_consensus.cc:738] T 3a8231590ea643b0a1f83eec4d9a8191 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:20.584983 1150 consensus_queue.cc:260] T 3a8231590ea643b0a1f83eec4d9a8191 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:20.586763 1150 ts_tablet_manager.cc:1428] T 3a8231590ea643b0a1f83eec4d9a8191 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250901 01:52:20.587523 1150 tablet_bootstrap.cc:492] T 40d74d7b5a3b4d1dae09eebd9e07a0c4 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:20.589309 1150 tablet_bootstrap.cc:654] T 40d74d7b5a3b4d1dae09eebd9e07a0c4 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:20.596760 1306 ts_tablet_manager.cc:1918] T 49d30da95b14400b845270122487a2a0 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:20.596890 1306 log.cc:1199] T 49d30da95b14400b845270122487a2a0 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/49d30da95b14400b845270122487a2a0
I20250901 01:52:20.597260 1306 ts_tablet_manager.cc:1939] T 49d30da95b14400b845270122487a2a0 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:20.598047 1306 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52846 (request call id 465) took 1153 ms. Trace:
I20250901 01:52:20.598138 1306 rpcz_store.cc:276] 0901 01:52:19.445036 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:19.445080 (+ 44us) service_pool.cc:225] Handling call
0901 01:52:20.555908 (+1110828us) ts_tablet_manager.cc:1145] Deleting tablet 49d30da95b14400b845270122487a2a0
0901 01:52:20.555919 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:20.583803 (+ 27884us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:20.596628 (+ 12825us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:20.598028 (+ 1400us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1110596}
I20250901 01:52:20.598534 1306 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52846 (request call id 466) took 1153 ms. Trace:
I20250901 01:52:20.598619 1306 rpcz_store.cc:276] 0901 01:52:19.445479 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:19.446458 (+ 979us) service_pool.cc:225] Handling call
0901 01:52:20.598389 (+1151931us) ts_tablet_manager.cc:1145] Deleting tablet 6741c56499b44f648fb032c8cf50205b
0901 01:52:20.598397 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:20.598519 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1151758}
I20250901 01:52:20.598853 1306 tablet_replica.cc:331] T b73e362321284df1b18ef755dfe930a5 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:20.599023 1306 raft_consensus.cc:2241] T b73e362321284df1b18ef755dfe930a5 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:20.599170 1306 raft_consensus.cc:2270] T b73e362321284df1b18ef755dfe930a5 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:20.599689 1306 ts_tablet_manager.cc:1905] T b73e362321284df1b18ef755dfe930a5 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:20.605589 1306 ts_tablet_manager.cc:1918] T b73e362321284df1b18ef755dfe930a5 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:20.605697 1306 log.cc:1199] T b73e362321284df1b18ef755dfe930a5 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/b73e362321284df1b18ef755dfe930a5
I20250901 01:52:20.606031 1306 ts_tablet_manager.cc:1939] T b73e362321284df1b18ef755dfe930a5 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:20.606813 1306 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52846 (request call id 469) took 1160 ms. Trace:
I20250901 01:52:20.606966 1150 tablet_bootstrap.cc:492] T 40d74d7b5a3b4d1dae09eebd9e07a0c4 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:20.607100 1151 tablet_bootstrap.cc:654] T b541a6e14f114ab29a864cca11806316 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:20.607108 1150 ts_tablet_manager.cc:1397] T 40d74d7b5a3b4d1dae09eebd9e07a0c4 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.006s
I20250901 01:52:20.607868 1150 raft_consensus.cc:357] T 40d74d7b5a3b4d1dae09eebd9e07a0c4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:20.608019 1150 raft_consensus.cc:383] T 40d74d7b5a3b4d1dae09eebd9e07a0c4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:20.608070 1150 raft_consensus.cc:738] T 40d74d7b5a3b4d1dae09eebd9e07a0c4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:20.608256 1150 consensus_queue.cc:260] T 40d74d7b5a3b4d1dae09eebd9e07a0c4 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:20.608767 1150 ts_tablet_manager.cc:1428] T 40d74d7b5a3b4d1dae09eebd9e07a0c4 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250901 01:52:20.609030 1150 tablet_bootstrap.cc:492] T 831e42de95854541a49ae6b3e039d3e9 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:20.610574 1150 tablet_bootstrap.cc:654] T 831e42de95854541a49ae6b3e039d3e9 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:20.625816 859 leader_election.cc:343] T 438cd35ff3f74c49b908d072560077db P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:20.625998 859 leader_election.cc:304] T 438cd35ff3f74c49b908d072560077db P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:20.626436 859 leader_election.cc:343] T 3fda93767e494a4684978c8522b829dd P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:20.626526 859 leader_election.cc:304] T 3fda93767e494a4684978c8522b829dd P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:20.626874 859 leader_election.cc:343] T 9d886fa63f8f4996b087997b542e8bb1 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:20.626957 859 leader_election.cc:304] T 9d886fa63f8f4996b087997b542e8bb1 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:20.627405 1711 raft_consensus.cc:2747] T 438cd35ff3f74c49b908d072560077db P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:20.627584 1711 raft_consensus.cc:2747] T 3fda93767e494a4684978c8522b829dd P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:20.627696 1711 raft_consensus.cc:2747] T 9d886fa63f8f4996b087997b542e8bb1 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:20.629845 859 leader_election.cc:343] T 9473f0ef978c4bfaa41fb7b0729e8863 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:20.629978 859 leader_election.cc:304] T 9473f0ef978c4bfaa41fb7b0729e8863 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:20.630339 859 leader_election.cc:343] T f62d96c37cca4143a4b098b7c69979e6 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:20.630430 859 leader_election.cc:304] T f62d96c37cca4143a4b098b7c69979e6 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:20.630935 1769 raft_consensus.cc:2747] T 9473f0ef978c4bfaa41fb7b0729e8863 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:20.631085 1769 raft_consensus.cc:2747] T f62d96c37cca4143a4b098b7c69979e6 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:20.606905 1306 rpcz_store.cc:276] 0901 01:52:19.446784 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:19.446822 (+ 38us) service_pool.cc:225] Handling call
0901 01:52:20.598816 (+1151994us) ts_tablet_manager.cc:1145] Deleting tablet b73e362321284df1b18ef755dfe930a5
0901 01:52:20.598823 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:20.602629 (+ 3806us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:20.605511 (+ 2882us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:20.606792 (+ 1281us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1151823}
I20250901 01:52:20.635823 1306 tablet_replica.cc:331] T 23af154a6f1c4e2e9352fb17c9024f10 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:20.637207 1306 raft_consensus.cc:2241] T 23af154a6f1c4e2e9352fb17c9024f10 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:20.638407 1306 raft_consensus.cc:2270] T 23af154a6f1c4e2e9352fb17c9024f10 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:20.639830 1306 ts_tablet_manager.cc:1905] T 23af154a6f1c4e2e9352fb17c9024f10 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:20.647300 1761 raft_consensus.cc:513] T 6c05eff658924ca4a9a1e7a37f917d67 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:20.649971 1782 raft_consensus.cc:491] T 00000000000000000000000000000000 P d2ae2cb6af0f4b3a8f47ab1cdd6567d3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e3add7edb97a40ef9f7c6bc011d13227)
I20250901 01:52:20.650151 1782 raft_consensus.cc:513] T 00000000000000000000000000000000 P d2ae2cb6af0f4b3a8f47ab1cdd6567d3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3add7edb97a40ef9f7c6bc011d13227" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 43871 } } peers { permanent_uuid: "d2ae2cb6af0f4b3a8f47ab1cdd6567d3" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42375 } } peers { permanent_uuid: "7ce2288f784044eea010664e3def133f" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 44231 } }
I20250901 01:52:20.651530 1782 leader_election.cc:290] T 00000000000000000000000000000000 P d2ae2cb6af0f4b3a8f47ab1cdd6567d3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e3add7edb97a40ef9f7c6bc011d13227 (127.0.104.62:43871), 7ce2288f784044eea010664e3def133f (127.0.104.60:44231)
I20250901 01:52:20.651942 1783 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e3add7edb97a40ef9f7c6bc011d13227)
I20250901 01:52:20.655980 1306 ts_tablet_manager.cc:1918] T 23af154a6f1c4e2e9352fb17c9024f10 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:20.656134 1306 log.cc:1199] T 23af154a6f1c4e2e9352fb17c9024f10 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/23af154a6f1c4e2e9352fb17c9024f10
I20250901 01:52:20.656428 1306 ts_tablet_manager.cc:1939] T 23af154a6f1c4e2e9352fb17c9024f10 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:20.657773 1306 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52846 (request call id 471) took 1210 ms. Trace:
I20250901 01:52:20.657939 1306 rpcz_store.cc:276] 0901 01:52:19.447279 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:19.447351 (+ 72us) service_pool.cc:225] Handling call
0901 01:52:20.635751 (+1188400us) ts_tablet_manager.cc:1145] Deleting tablet 23af154a6f1c4e2e9352fb17c9024f10
0901 01:52:20.635777 (+ 26us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:20.654092 (+ 18315us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:20.655861 (+ 1769us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:20.657745 (+ 1884us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1188210}
I20250901 01:52:20.658427 1306 tablet_replica.cc:331] T 9473f0ef978c4bfaa41fb7b0729e8863 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:20.658703 1306 raft_consensus.cc:2241] T 9473f0ef978c4bfaa41fb7b0729e8863 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:20.658906 1306 raft_consensus.cc:2270] T 9473f0ef978c4bfaa41fb7b0729e8863 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:20.659546 1761 leader_election.cc:290] T 6c05eff658924ca4a9a1e7a37f917d67 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:20.659581 1306 ts_tablet_manager.cc:1905] T 9473f0ef978c4bfaa41fb7b0729e8863 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:20.667078 1306 ts_tablet_manager.cc:1918] T 9473f0ef978c4bfaa41fb7b0729e8863 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:20.667299 1306 log.cc:1199] T 9473f0ef978c4bfaa41fb7b0729e8863 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/9473f0ef978c4bfaa41fb7b0729e8863
I20250901 01:52:20.667734 1306 ts_tablet_manager.cc:1939] T 9473f0ef978c4bfaa41fb7b0729e8863 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:20.668686 1306 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52846 (request call id 472) took 1220 ms. Trace:
I20250901 01:52:20.668864 1306 rpcz_store.cc:276] 0901 01:52:19.447824 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:19.447866 (+ 42us) service_pool.cc:225] Handling call
0901 01:52:20.658367 (+1210501us) ts_tablet_manager.cc:1145] Deleting tablet 9473f0ef978c4bfaa41fb7b0729e8863
0901 01:52:20.658380 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:20.663436 (+ 5056us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:20.666984 (+ 3548us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:20.668661 (+ 1677us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1210313}
I20250901 01:52:20.669338 1306 tablet_replica.cc:331] T 3fda93767e494a4684978c8522b829dd P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:20.670032 1306 raft_consensus.cc:2241] T 3fda93767e494a4684978c8522b829dd P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:20.670358 1306 raft_consensus.cc:2270] T 3fda93767e494a4684978c8522b829dd P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:20.671015 1306 ts_tablet_manager.cc:1905] T 3fda93767e494a4684978c8522b829dd P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:20.678442 1306 ts_tablet_manager.cc:1918] T 3fda93767e494a4684978c8522b829dd P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:20.678808 1306 log.cc:1199] T 3fda93767e494a4684978c8522b829dd P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/3fda93767e494a4684978c8522b829dd
I20250901 01:52:20.679224 1306 ts_tablet_manager.cc:1939] T 3fda93767e494a4684978c8522b829dd P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:20.679437 1332 ts_tablet_manager.cc:1918] T 3df908c3096a4df7bddd8ac88b49cfe0 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:20.679620 1332 log.cc:1199] T 3df908c3096a4df7bddd8ac88b49cfe0 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/3df908c3096a4df7bddd8ac88b49cfe0
I20250901 01:52:20.679965 1332 ts_tablet_manager.cc:1939] T 3df908c3096a4df7bddd8ac88b49cfe0 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:20.680334 1306 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52846 (request call id 476) took 1173 ms. Trace:
I20250901 01:52:20.680531 1306 rpcz_store.cc:276] 0901 01:52:19.506755 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:19.506808 (+ 53us) service_pool.cc:225] Handling call
0901 01:52:20.669283 (+1162475us) ts_tablet_manager.cc:1145] Deleting tablet 3fda93767e494a4684978c8522b829dd
0901 01:52:20.669296 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:20.674838 (+ 5542us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:20.678331 (+ 3493us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:20.680306 (+ 1975us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1162203}
I20250901 01:52:20.681130 1306 tablet_replica.cc:331] T 9d886fa63f8f4996b087997b542e8bb1 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:20.681388 1306 raft_consensus.cc:2241] T 9d886fa63f8f4996b087997b542e8bb1 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:20.681615 1306 raft_consensus.cc:2270] T 9d886fa63f8f4996b087997b542e8bb1 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:20.682238 1306 ts_tablet_manager.cc:1905] T 9d886fa63f8f4996b087997b542e8bb1 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:20.688032 992 leader_election.cc:336] T a7ff996482164776b468cf305989866b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:20.688200 992 leader_election.cc:304] T a7ff996482164776b468cf305989866b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:20.688454 1761 raft_consensus.cc:2747] T a7ff996482164776b468cf305989866b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:20.688737 992 leader_election.cc:336] T 6c05eff658924ca4a9a1e7a37f917d67 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:20.688951 992 consensus_peers.cc:489] T 186ec810d02e4748aad15228b5866ac1 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250901 01:52:20.689041 992 leader_election.cc:336] T 5adf6b492640417ca071c94702b48cb9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:20.690479 992 leader_election.cc:304] T 5adf6b492640417ca071c94702b48cb9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:20.690747 1761 raft_consensus.cc:2802] T 5adf6b492640417ca071c94702b48cb9 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:20.690888 1761 raft_consensus.cc:695] T 5adf6b492640417ca071c94702b48cb9 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:20.691140 1761 consensus_queue.cc:237] T 5adf6b492640417ca071c94702b48cb9 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:20.692371 1332 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44246 (request call id 463) took 1193 ms. Trace:
I20250901 01:52:20.692523 1332 rpcz_store.cc:276] 0901 01:52:19.499085 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:19.520217 (+ 21132us) service_pool.cc:225] Handling call
0901 01:52:20.491514 (+971297us) ts_tablet_manager.cc:1145] Deleting tablet 3df908c3096a4df7bddd8ac88b49cfe0
0901 01:52:20.491523 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:20.665412 (+173889us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:20.679330 (+ 13918us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:20.692328 (+ 12998us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6824,"tablet-delete.queue_time_us":956532}
I20250901 01:52:20.692930 1332 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44246 (request call id 485) took 1030 ms. Trace:
I20250901 01:52:20.693014 1332 rpcz_store.cc:276] 0901 01:52:19.662650 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:20.266328 (+603678us) service_pool.cc:225] Handling call
0901 01:52:20.692782 (+426454us) ts_tablet_manager.cc:1145] Deleting tablet 56f2c4d2e6ab4bb4bdd6d025843589b6
0901 01:52:20.692791 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:20.692889 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":418396}
I20250901 01:52:20.693356 1332 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44246 (request call id 490) took 1016 ms. Trace:
I20250901 01:52:20.693439 1332 rpcz_store.cc:276] 0901 01:52:19.677027 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:20.302637 (+625610us) service_pool.cc:225] Handling call
0901 01:52:20.693241 (+390604us) ts_tablet_manager.cc:1145] Deleting tablet e8e7d2a72f704f52a8f8d46911a8a9e4
0901 01:52:20.693248 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:20.693313 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":390362}
I20250901 01:52:20.693713 1306 ts_tablet_manager.cc:1918] T 9d886fa63f8f4996b087997b542e8bb1 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:20.693755 1332 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44246 (request call id 486) took 1031 ms. Trace:
I20250901 01:52:20.693841 1332 rpcz_store.cc:276] 0901 01:52:19.662735 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:20.270649 (+607914us) service_pool.cc:225] Handling call
0901 01:52:20.693614 (+422965us) ts_tablet_manager.cc:1145] Deleting tablet 46bfbdff3f904715a8c24907b20fcab0
0901 01:52:20.693621 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:20.693719 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":387545}
I20250901 01:52:20.693929 1306 log.cc:1199] T 9d886fa63f8f4996b087997b542e8bb1 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/9d886fa63f8f4996b087997b542e8bb1
I20250901 01:52:20.694482 1306 ts_tablet_manager.cc:1939] T 9d886fa63f8f4996b087997b542e8bb1 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:20.695484 1306 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52846 (request call id 477) took 1164 ms. Trace:
I20250901 01:52:20.695658 1306 rpcz_store.cc:276] 0901 01:52:19.530651 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:19.530722 (+ 71us) service_pool.cc:225] Handling call
0901 01:52:20.681071 (+1150349us) ts_tablet_manager.cc:1145] Deleting tablet 9d886fa63f8f4996b087997b542e8bb1
0901 01:52:20.681084 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:20.685968 (+ 4884us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:20.693514 (+ 7546us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:20.695443 (+ 1929us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1150086}
I20250901 01:52:20.706199 1306 tablet_replica.cc:331] T 438cd35ff3f74c49b908d072560077db P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:20.706645 1306 raft_consensus.cc:2241] T 438cd35ff3f74c49b908d072560077db P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:20.706928 1306 raft_consensus.cc:2270] T 438cd35ff3f74c49b908d072560077db P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:20.707756 1306 ts_tablet_manager.cc:1905] T 438cd35ff3f74c49b908d072560077db P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:20.651805 1503 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d2ae2cb6af0f4b3a8f47ab1cdd6567d3" candidate_term: 3 candidate_status { last_received { term: 2 index: 633 } } ignore_live_leader: false dest_uuid: "7ce2288f784044eea010664e3def133f" is_pre_election: true
I20250901 01:52:20.708614 1783 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3add7edb97a40ef9f7c6bc011d13227" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 43871 } } peers { permanent_uuid: "d2ae2cb6af0f4b3a8f47ab1cdd6567d3" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42375 } } peers { permanent_uuid: "7ce2288f784044eea010664e3def133f" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 44231 } }
I20250901 01:52:20.709141 1503 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2ae2cb6af0f4b3a8f47ab1cdd6567d3 in term 2.
I20250901 01:52:20.709254 1151 tablet_bootstrap.cc:492] T b541a6e14f114ab29a864cca11806316 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:20.709398 1151 ts_tablet_manager.cc:1397] T b541a6e14f114ab29a864cca11806316 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.201s user 0.008s sys 0.000s
I20250901 01:52:20.709762 513 leader_election.cc:304] T 00000000000000000000000000000000 P d2ae2cb6af0f4b3a8f47ab1cdd6567d3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ce2288f784044eea010664e3def133f, d2ae2cb6af0f4b3a8f47ab1cdd6567d3; no voters:
I20250901 01:52:20.710251 1151 raft_consensus.cc:357] T b541a6e14f114ab29a864cca11806316 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:20.710487 1151 raft_consensus.cc:383] T b541a6e14f114ab29a864cca11806316 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:20.710584 1783 leader_election.cc:290] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e3add7edb97a40ef9f7c6bc011d13227 (127.0.104.62:43871), d2ae2cb6af0f4b3a8f47ab1cdd6567d3 (127.0.104.61:42375)
I20250901 01:52:20.710597 1151 raft_consensus.cc:738] T b541a6e14f114ab29a864cca11806316 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:20.710639 534 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7ce2288f784044eea010664e3def133f" candidate_term: 3 candidate_status { last_received { term: 2 index: 633 } } ignore_live_leader: false dest_uuid: "d2ae2cb6af0f4b3a8f47ab1cdd6567d3" is_pre_election: true
I20250901 01:52:20.710793 1782 raft_consensus.cc:2802] T 00000000000000000000000000000000 P d2ae2cb6af0f4b3a8f47ab1cdd6567d3 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250901 01:52:20.710808 1151 consensus_queue.cc:260] T b541a6e14f114ab29a864cca11806316 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:20.710986 534 raft_consensus.cc:2466] T 00000000000000000000000000000000 P d2ae2cb6af0f4b3a8f47ab1cdd6567d3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ce2288f784044eea010664e3def133f in term 2.
I20250901 01:52:20.711421 1474 leader_election.cc:304] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ce2288f784044eea010664e3def133f, d2ae2cb6af0f4b3a8f47ab1cdd6567d3; no voters:
I20250901 01:52:20.711788 1782 raft_consensus.cc:491] T 00000000000000000000000000000000 P d2ae2cb6af0f4b3a8f47ab1cdd6567d3 [term 2 FOLLOWER]: Starting leader election (detected failure of leader e3add7edb97a40ef9f7c6bc011d13227)
I20250901 01:52:20.711937 1782 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d2ae2cb6af0f4b3a8f47ab1cdd6567d3 [term 2 FOLLOWER]: Advancing to term 3
I20250901 01:52:20.716679 1151 ts_tablet_manager.cc:1428] T b541a6e14f114ab29a864cca11806316 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250901 01:52:20.717015 1151 tablet_bootstrap.cc:492] T 12c04e0c1f804179ae23509cdb24d395 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:20.717296 919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c05eff658924ca4a9a1e7a37f917d67" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:20.717445 919 raft_consensus.cc:3058] T 6c05eff658924ca4a9a1e7a37f917d67 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:20.718672 1151 tablet_bootstrap.cc:654] T 12c04e0c1f804179ae23509cdb24d395 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:20.719040 992 consensus_peers.cc:489] T 85322a864d154f299f2d670fdd2a7bfb P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250901 01:52:20.720031 919 raft_consensus.cc:2466] T 6c05eff658924ca4a9a1e7a37f917d67 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
W20250901 01:52:20.721541 992 leader_election.cc:336] T 6c05eff658924ca4a9a1e7a37f917d67 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:20.721791 1783 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250901 01:52:20.721907 1783 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f [term 2 FOLLOWER]: Starting leader election (detected failure of leader e3add7edb97a40ef9f7c6bc011d13227)
I20250901 01:52:20.722011 1783 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f [term 2 FOLLOWER]: Advancing to term 3
I20250901 01:52:20.727633 1783 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3add7edb97a40ef9f7c6bc011d13227" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 43871 } } peers { permanent_uuid: "d2ae2cb6af0f4b3a8f47ab1cdd6567d3" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42375 } } peers { permanent_uuid: "7ce2288f784044eea010664e3def133f" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 44231 } }
I20250901 01:52:20.732204 1761 raft_consensus.cc:491] T 154574e17f214cc0ad56e6e6be3af899 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:20.733188 1761 raft_consensus.cc:513] T 154574e17f214cc0ad56e6e6be3af899 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:20.735296 1761 leader_election.cc:290] T 154574e17f214cc0ad56e6e6be3af899 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:20.737622 992 leader_election.cc:336] T 154574e17f214cc0ad56e6e6be3af899 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:20.716287 1782 raft_consensus.cc:513] T 00000000000000000000000000000000 P d2ae2cb6af0f4b3a8f47ab1cdd6567d3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3add7edb97a40ef9f7c6bc011d13227" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 43871 } } peers { permanent_uuid: "d2ae2cb6af0f4b3a8f47ab1cdd6567d3" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42375 } } peers { permanent_uuid: "7ce2288f784044eea010664e3def133f" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 44231 } }
I20250901 01:52:20.746351 1151 tablet_bootstrap.cc:492] T 12c04e0c1f804179ae23509cdb24d395 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:20.746920 1150 tablet_bootstrap.cc:492] T 831e42de95854541a49ae6b3e039d3e9 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:20.747156 1151 ts_tablet_manager.cc:1397] T 12c04e0c1f804179ae23509cdb24d395 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.030s user 0.001s sys 0.014s
I20250901 01:52:20.747980 1151 raft_consensus.cc:357] T 12c04e0c1f804179ae23509cdb24d395 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:20.748251 1150 ts_tablet_manager.cc:1397] T 831e42de95854541a49ae6b3e039d3e9 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.139s user 0.021s sys 0.030s
I20250901 01:52:20.748940 1150 raft_consensus.cc:357] T 831e42de95854541a49ae6b3e039d3e9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:20.749195 1151 raft_consensus.cc:383] T 12c04e0c1f804179ae23509cdb24d395 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:20.749300 1150 raft_consensus.cc:383] T 831e42de95854541a49ae6b3e039d3e9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:20.749406 1151 raft_consensus.cc:738] T 12c04e0c1f804179ae23509cdb24d395 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:20.749498 1150 raft_consensus.cc:738] T 831e42de95854541a49ae6b3e039d3e9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:20.749773 1151 consensus_queue.cc:260] T 12c04e0c1f804179ae23509cdb24d395 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:20.750139 1150 consensus_queue.cc:260] T 831e42de95854541a49ae6b3e039d3e9 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:20.750626 1150 ts_tablet_manager.cc:1428] T 831e42de95854541a49ae6b3e039d3e9 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250901 01:52:20.751066 1151 ts_tablet_manager.cc:1428] T 12c04e0c1f804179ae23509cdb24d395 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250901 01:52:20.751525 1150 tablet_bootstrap.cc:492] T f06a77ecc841462396a8d5026c7327f4 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:20.751857 1151 tablet_bootstrap.cc:492] T 4d27fb61943c45d4aca67689cc3a9ff3 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:20.753300 1782 leader_election.cc:290] T 00000000000000000000000000000000 P d2ae2cb6af0f4b3a8f47ab1cdd6567d3 [CANDIDATE]: Term 3 election: Requested vote from peers e3add7edb97a40ef9f7c6bc011d13227 (127.0.104.62:43871), 7ce2288f784044eea010664e3def133f (127.0.104.60:44231)
I20250901 01:52:20.756126 1151 tablet_bootstrap.cc:654] T 4d27fb61943c45d4aca67689cc3a9ff3 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:20.759236 1306 ts_tablet_manager.cc:1918] T 438cd35ff3f74c49b908d072560077db P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:20.753418 1503 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d2ae2cb6af0f4b3a8f47ab1cdd6567d3" candidate_term: 3 candidate_status { last_received { term: 2 index: 633 } } ignore_live_leader: false dest_uuid: "7ce2288f784044eea010664e3def133f"
I20250901 01:52:20.761071 1503 raft_consensus.cc:2391] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d2ae2cb6af0f4b3a8f47ab1cdd6567d3 in current term 3: Already voted for candidate 7ce2288f784044eea010664e3def133f in this term.
I20250901 01:52:20.762513 1783 leader_election.cc:290] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f [CANDIDATE]: Term 3 election: Requested vote from peers e3add7edb97a40ef9f7c6bc011d13227 (127.0.104.62:43871), d2ae2cb6af0f4b3a8f47ab1cdd6567d3 (127.0.104.61:42375)
I20250901 01:52:20.762579 534 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7ce2288f784044eea010664e3def133f" candidate_term: 3 candidate_status { last_received { term: 2 index: 633 } } ignore_live_leader: false dest_uuid: "d2ae2cb6af0f4b3a8f47ab1cdd6567d3"
I20250901 01:52:20.762866 534 raft_consensus.cc:2391] T 00000000000000000000000000000000 P d2ae2cb6af0f4b3a8f47ab1cdd6567d3 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7ce2288f784044eea010664e3def133f in current term 3: Already voted for candidate d2ae2cb6af0f4b3a8f47ab1cdd6567d3 in this term.
I20250901 01:52:20.765138 1306 log.cc:1199] T 438cd35ff3f74c49b908d072560077db P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/438cd35ff3f74c49b908d072560077db
I20250901 01:52:20.766124 1306 ts_tablet_manager.cc:1939] T 438cd35ff3f74c49b908d072560077db P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:20.764598 1150 tablet_bootstrap.cc:654] T f06a77ecc841462396a8d5026c7327f4 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:20.768040 992 leader_election.cc:304] T 6c05eff658924ca4a9a1e7a37f917d67 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:20.768558 1669 raft_consensus.cc:2802] T 6c05eff658924ca4a9a1e7a37f917d67 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:20.768793 1669 raft_consensus.cc:695] T 6c05eff658924ca4a9a1e7a37f917d67 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:20.769178 1669 consensus_queue.cc:237] T 6c05eff658924ca4a9a1e7a37f917d67 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
W20250901 01:52:20.812353 992 leader_election.cc:343] T 154574e17f214cc0ad56e6e6be3af899 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 154574e17f214cc0ad56e6e6be3af899
I20250901 01:52:20.812616 992 leader_election.cc:304] T 154574e17f214cc0ad56e6e6be3af899 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:20.813158 1771 raft_consensus.cc:2747] T 154574e17f214cc0ad56e6e6be3af899 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:20.891435 1150 tablet_bootstrap.cc:492] T f06a77ecc841462396a8d5026c7327f4 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:20.891599 1150 ts_tablet_manager.cc:1397] T f06a77ecc841462396a8d5026c7327f4 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.140s user 0.019s sys 0.012s
I20250901 01:52:20.892390 1150 raft_consensus.cc:357] T f06a77ecc841462396a8d5026c7327f4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:20.892619 1150 raft_consensus.cc:383] T f06a77ecc841462396a8d5026c7327f4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:20.892683 1150 raft_consensus.cc:738] T f06a77ecc841462396a8d5026c7327f4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:20.896323 1802 tablet.cc:1722] T f65ec84958eb488d8b0f812b59b5b3d5 P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250901 01:52:20.902860 1150 consensus_queue.cc:260] T f06a77ecc841462396a8d5026c7327f4 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:20.897089 1804 tablet.cc:1722] T 4410da9e95454ecf95faf80221b0a71c P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:20.901350 1151 tablet_bootstrap.cc:492] T 4d27fb61943c45d4aca67689cc3a9ff3 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:20.903393 1151 ts_tablet_manager.cc:1397] T 4d27fb61943c45d4aca67689cc3a9ff3 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.152s user 0.020s sys 0.043s
I20250901 01:52:20.904270 1150 ts_tablet_manager.cc:1428] T f06a77ecc841462396a8d5026c7327f4 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20250901 01:52:20.904289 1151 raft_consensus.cc:357] T 4d27fb61943c45d4aca67689cc3a9ff3 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:20.904488 1151 raft_consensus.cc:383] T 4d27fb61943c45d4aca67689cc3a9ff3 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:20.904580 1151 raft_consensus.cc:738] T 4d27fb61943c45d4aca67689cc3a9ff3 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:20.904857 1151 consensus_queue.cc:260] T 4d27fb61943c45d4aca67689cc3a9ff3 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:20.905412 1151 ts_tablet_manager.cc:1428] T 4d27fb61943c45d4aca67689cc3a9ff3 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250901 01:52:20.905711 1151 tablet_bootstrap.cc:492] T a774cc6ae3924d299127fd9ffb01ca57 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:20.906798 1805 tablet.cc:1722] T f65ec84958eb488d8b0f812b59b5b3d5 P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250901 01:52:20.906798 1150 tablet_bootstrap.cc:492] T ddbc15685fca4c708329420109ad0c57 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:20.907454 1803 tablet.cc:1722] T 4410da9e95454ecf95faf80221b0a71c P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:20.908821 1150 tablet_bootstrap.cc:654] T ddbc15685fca4c708329420109ad0c57 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:20.917824 1151 tablet_bootstrap.cc:654] T a774cc6ae3924d299127fd9ffb01ca57 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:20.918833 1150 tablet_bootstrap.cc:492] T ddbc15685fca4c708329420109ad0c57 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:20.918968 1150 ts_tablet_manager.cc:1397] T ddbc15685fca4c708329420109ad0c57 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.001s
I20250901 01:52:20.919649 1150 raft_consensus.cc:357] T ddbc15685fca4c708329420109ad0c57 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:20.919857 1150 raft_consensus.cc:383] T ddbc15685fca4c708329420109ad0c57 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:20.919911 1150 raft_consensus.cc:738] T ddbc15685fca4c708329420109ad0c57 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:20.920107 1150 consensus_queue.cc:260] T ddbc15685fca4c708329420109ad0c57 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:20.920576 1150 ts_tablet_manager.cc:1428] T ddbc15685fca4c708329420109ad0c57 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250901 01:52:20.920855 1150 tablet_bootstrap.cc:492] T ded33e0950b544509af11739353c620d P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:20.928800 1150 tablet_bootstrap.cc:654] T ded33e0950b544509af11739353c620d P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:20.929405 1151 tablet_bootstrap.cc:492] T a774cc6ae3924d299127fd9ffb01ca57 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:20.929584 1151 ts_tablet_manager.cc:1397] T a774cc6ae3924d299127fd9ffb01ca57 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.000s
I20250901 01:52:20.930413 1151 raft_consensus.cc:357] T a774cc6ae3924d299127fd9ffb01ca57 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:20.930680 1151 raft_consensus.cc:383] T a774cc6ae3924d299127fd9ffb01ca57 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:20.930770 1151 raft_consensus.cc:738] T a774cc6ae3924d299127fd9ffb01ca57 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:20.931005 1151 consensus_queue.cc:260] T a774cc6ae3924d299127fd9ffb01ca57 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:20.931514 1151 ts_tablet_manager.cc:1428] T a774cc6ae3924d299127fd9ffb01ca57 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250901 01:52:20.942236 1151 tablet_bootstrap.cc:492] T 824229183e024feaa975c7f0387c9c1a P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:20.944773 1151 tablet_bootstrap.cc:654] T 824229183e024feaa975c7f0387c9c1a P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:20.951107 1151 tablet_bootstrap.cc:492] T 824229183e024feaa975c7f0387c9c1a P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:20.951107 1150 tablet_bootstrap.cc:492] T ded33e0950b544509af11739353c620d P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:20.951259 1151 ts_tablet_manager.cc:1397] T 824229183e024feaa975c7f0387c9c1a P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.002s
I20250901 01:52:20.951256 1150 ts_tablet_manager.cc:1397] T ded33e0950b544509af11739353c620d P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.030s user 0.005s sys 0.019s
I20250901 01:52:20.952020 1150 raft_consensus.cc:357] T ded33e0950b544509af11739353c620d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:20.952049 1151 raft_consensus.cc:357] T 824229183e024feaa975c7f0387c9c1a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:20.952195 1150 raft_consensus.cc:383] T ded33e0950b544509af11739353c620d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:20.952224 1151 raft_consensus.cc:383] T 824229183e024feaa975c7f0387c9c1a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:20.952253 1150 raft_consensus.cc:738] T ded33e0950b544509af11739353c620d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:20.952302 1151 raft_consensus.cc:738] T 824229183e024feaa975c7f0387c9c1a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:20.952534 1150 consensus_queue.cc:260] T ded33e0950b544509af11739353c620d P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:20.952579 1151 consensus_queue.cc:260] T 824229183e024feaa975c7f0387c9c1a P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:20.953173 1150 ts_tablet_manager.cc:1428] T ded33e0950b544509af11739353c620d P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250901 01:52:20.955955 1151 ts_tablet_manager.cc:1428] T 824229183e024feaa975c7f0387c9c1a P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20250901 01:52:20.956259 1151 tablet_bootstrap.cc:492] T b633b461dbf34e13bc6e1cc08e6d20c1 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:20.958053 1151 tablet_bootstrap.cc:654] T b633b461dbf34e13bc6e1cc08e6d20c1 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:20.962685 1150 tablet_bootstrap.cc:492] T e25923b2daf9429683b2d44a75a56189 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:20.964814 1150 tablet_bootstrap.cc:654] T e25923b2daf9429683b2d44a75a56189 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:20.971144 1761 raft_consensus.cc:491] T 60285f2ba037433999f5455fb9cc7acd P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:20.971344 1761 raft_consensus.cc:513] T 60285f2ba037433999f5455fb9cc7acd P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:20.971961 1761 leader_election.cc:290] T 60285f2ba037433999f5455fb9cc7acd P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:20.974020 992 leader_election.cc:336] T 60285f2ba037433999f5455fb9cc7acd P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:20.974305 992 leader_election.cc:343] T 60285f2ba037433999f5455fb9cc7acd P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 60285f2ba037433999f5455fb9cc7acd
I20250901 01:52:20.974457 992 leader_election.cc:304] T 60285f2ba037433999f5455fb9cc7acd P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:20.974843 1761 raft_consensus.cc:2747] T 60285f2ba037433999f5455fb9cc7acd P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:20.981050 1150 tablet_bootstrap.cc:492] T e25923b2daf9429683b2d44a75a56189 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:20.981195 1150 ts_tablet_manager.cc:1397] T e25923b2daf9429683b2d44a75a56189 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.005s
I20250901 01:52:20.981972 1150 raft_consensus.cc:357] T e25923b2daf9429683b2d44a75a56189 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:20.982146 1150 raft_consensus.cc:383] T e25923b2daf9429683b2d44a75a56189 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:20.982211 1150 raft_consensus.cc:738] T e25923b2daf9429683b2d44a75a56189 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:20.982447 1150 consensus_queue.cc:260] T e25923b2daf9429683b2d44a75a56189 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:20.982932 1150 ts_tablet_manager.cc:1428] T e25923b2daf9429683b2d44a75a56189 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250901 01:52:20.984498 1150 tablet_bootstrap.cc:492] T c763d252598745a088344aa0eb5fa3d3 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:20.988219 1150 tablet_bootstrap.cc:654] T c763d252598745a088344aa0eb5fa3d3 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:20.988943 1151 tablet_bootstrap.cc:492] T b633b461dbf34e13bc6e1cc08e6d20c1 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:20.989114 1151 ts_tablet_manager.cc:1397] T b633b461dbf34e13bc6e1cc08e6d20c1 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.033s user 0.015s sys 0.016s
I20250901 01:52:20.989899 1151 raft_consensus.cc:357] T b633b461dbf34e13bc6e1cc08e6d20c1 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:20.990123 1151 raft_consensus.cc:383] T b633b461dbf34e13bc6e1cc08e6d20c1 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:20.990211 1151 raft_consensus.cc:738] T b633b461dbf34e13bc6e1cc08e6d20c1 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:20.990463 1151 consensus_queue.cc:260] T b633b461dbf34e13bc6e1cc08e6d20c1 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:20.991145 1151 ts_tablet_manager.cc:1428] T b633b461dbf34e13bc6e1cc08e6d20c1 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250901 01:52:20.992156 1151 tablet_bootstrap.cc:492] T 49d30da95b14400b845270122487a2a0 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:20.994481 1151 tablet_bootstrap.cc:654] T 49d30da95b14400b845270122487a2a0 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:21.004194 439 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 33f637f6a8134215bb1995ec38440073 (table default.Table_2ece91972d494680a8ad00273c10dee4 [id=557afe29976d40f6bfb6ae4ca348f887], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 33f637f6a8134215bb1995ec38440073 consensus state has no leader
I20250901 01:52:21.004583 1150 tablet_bootstrap.cc:492] T c763d252598745a088344aa0eb5fa3d3 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:21.004665 1151 tablet_bootstrap.cc:492] T 49d30da95b14400b845270122487a2a0 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:21.004731 1150 ts_tablet_manager.cc:1397] T c763d252598745a088344aa0eb5fa3d3 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.008s
I20250901 01:52:21.004838 1151 ts_tablet_manager.cc:1397] T 49d30da95b14400b845270122487a2a0 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.013s user 0.012s sys 0.000s
I20250901 01:52:21.005455 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet 50472a7aa2f04e0fbd65ac6c86c34b3e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:20 UTC) from {username='slave'} at 127.0.0.1:44246
I20250901 01:52:21.005528 1150 raft_consensus.cc:357] T c763d252598745a088344aa0eb5fa3d3 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.005677 1150 raft_consensus.cc:383] T c763d252598745a088344aa0eb5fa3d3 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.005728 1150 raft_consensus.cc:738] T c763d252598745a088344aa0eb5fa3d3 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:21.005909 1150 consensus_queue.cc:260] T c763d252598745a088344aa0eb5fa3d3 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.006281 1151 raft_consensus.cc:357] T 49d30da95b14400b845270122487a2a0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:21.006384 1150 ts_tablet_manager.cc:1428] T c763d252598745a088344aa0eb5fa3d3 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250901 01:52:21.006464 1151 raft_consensus.cc:383] T 49d30da95b14400b845270122487a2a0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.006929 1150 tablet_bootstrap.cc:492] T 5d794471dbed4477b9d52fd411fa4d10 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:21.007150 465 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7ce2288f784044eea010664e3def133f" candidate_term: 3 candidate_status { last_received { term: 2 index: 633 } } ignore_live_leader: false dest_uuid: "e3add7edb97a40ef9f7c6bc011d13227" is_pre_election: true
I20250901 01:52:21.006546 1151 raft_consensus.cc:738] T 49d30da95b14400b845270122487a2a0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:21.007766 1151 consensus_queue.cc:260] T 49d30da95b14400b845270122487a2a0 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:21.008853 1150 tablet_bootstrap.cc:654] T 5d794471dbed4477b9d52fd411fa4d10 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:21.014663 464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7ce2288f784044eea010664e3def133f" candidate_term: 3 candidate_status { last_received { term: 2 index: 633 } } ignore_live_leader: false dest_uuid: "e3add7edb97a40ef9f7c6bc011d13227"
I20250901 01:52:21.015539 1472 leader_election.cc:304] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ce2288f784044eea010664e3def133f; no voters: d2ae2cb6af0f4b3a8f47ab1cdd6567d3, e3add7edb97a40ef9f7c6bc011d13227
I20250901 01:52:21.015967 1783 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250901 01:52:21.015043 451 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_4401d83f75f8464da49d5b811d7dd89d" } modify_external_catalogs: true
I20250901 01:52:21.018715 1151 ts_tablet_manager.cc:1428] T 49d30da95b14400b845270122487a2a0 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.014s user 0.000s sys 0.003s
W20250901 01:52:21.019490 440 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.1:40731, user_credentials={real_user=slave}} blocked reactor thread for 905626us
I20250901 01:52:21.020923 1151 tablet_bootstrap.cc:492] T 533d14f5903e49599a34ea353ffd8c00 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:21.023068 1151 tablet_bootstrap.cc:654] T 533d14f5903e49599a34ea353ffd8c00 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:21.023794 1150 tablet_bootstrap.cc:492] T 5d794471dbed4477b9d52fd411fa4d10 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:21.023933 1150 ts_tablet_manager.cc:1397] T 5d794471dbed4477b9d52fd411fa4d10 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.004s
I20250901 01:52:21.024704 1150 raft_consensus.cc:357] T 5d794471dbed4477b9d52fd411fa4d10 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.024901 1150 raft_consensus.cc:383] T 5d794471dbed4477b9d52fd411fa4d10 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.024962 1150 raft_consensus.cc:738] T 5d794471dbed4477b9d52fd411fa4d10 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:21.025153 1150 consensus_queue.cc:260] T 5d794471dbed4477b9d52fd411fa4d10 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.020148 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_241e1d06219f4aa2963f2c326cb82a0a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:21.025655 1150 ts_tablet_manager.cc:1428] T 5d794471dbed4477b9d52fd411fa4d10 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250901 01:52:21.025902 1150 tablet_bootstrap.cc:492] T f8a5886ddb644a86b76df44aab309057 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:21.026257 534 raft_consensus.cc:1238] T 00000000000000000000000000000000 P d2ae2cb6af0f4b3a8f47ab1cdd6567d3 [term 3 FOLLOWER]: Rejecting Update request from peer e3add7edb97a40ef9f7c6bc011d13227 for earlier term 2. Current term is 3. Ops: []
I20250901 01:52:21.026440 1503 raft_consensus.cc:1238] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f [term 3 FOLLOWER]: Rejecting Update request from peer e3add7edb97a40ef9f7c6bc011d13227 for earlier term 2. Current term is 3. Ops: []
I20250901 01:52:21.026607 452 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_d9e28b97a2e94ca8bf0ee8a01acd76e8" } modify_external_catalogs: true
I20250901 01:52:21.027408 1150 tablet_bootstrap.cc:654] T f8a5886ddb644a86b76df44aab309057 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:21.030615 446 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_d51ef24ec0f846c7ad212360eb478cd7" } new_table_name: "default.Table_8445fcd774874cadaf4acc3581eee417" modify_external_catalogs: true
I20250901 01:52:21.030941 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_429d55410fd14d2e9e5af8fae4d60c34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:21.032456 448 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_0e73f80424cd48408f18c2eff1411706" } modify_external_catalogs: true
I20250901 01:52:21.032649 448 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_0e73f80424cd48408f18c2eff1411706" } modify_external_catalogs: true
I20250901 01:52:21.029335 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_b304df88f34540a98dc27e701bbbcae5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:21.035977 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_241e1d06219f4aa2963f2c326cb82a0a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:21.038638 452 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57810:
table { table_name: "default.Table_d9e28b97a2e94ca8bf0ee8a01acd76e8" } modify_external_catalogs: true
W20250901 01:52:21.040186 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_429d55410fd14d2e9e5af8fae4d60c34 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:21.041301 455 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet f62d96c37cca4143a4b098b7c69979e6 (table default.Table_0877ac8202a24e9da650a589bc6dbfcc [id=1f65b3e09c9748f189ecb2b56c5d1132]). Expected version 1 got 0
W20250901 01:52:21.045972 455 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:41695 (request call id 63) took 934 ms (client timeout 1000 ms). Trace:
W20250901 01:52:21.046111 455 rpcz_store.cc:269] 0901 01:52:20.107414 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:20.107479 (+ 65us) service_pool.cc:225] Handling call
0901 01:52:21.041969 (+934490us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":1}
W20250901 01:52:21.048091 485 rpcz_store.cc:267] Call kudu.server.GenericService.CheckLeaks from 127.0.0.1:57800 (request call id 31) took 931 ms (client timeout 1000 ms). Trace:
W20250901 01:52:21.068559 440 catalog_manager.cc:4671] TS 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): CreateTablet RPC failed for tablet 36e10328c34242ebb4dbd8f294162696: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:21.068521 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57810:
name: "default.Table_d4d52bbc912d428fa018d1044927c76f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:21.069648 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b304df88f34540a98dc27e701bbbcae5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:21.070088 1398 consensus_queue.cc:1046] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d2ae2cb6af0f4b3a8f47ab1cdd6567d3" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42375 }, Status: INVALID_TERM, Last received: 2.633, Next index: 634, Last known committed idx: 632, Time since last communication: 0.023s
I20250901 01:52:21.070874 1409 consensus_queue.cc:1046] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7ce2288f784044eea010664e3def133f" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 44231 }, Status: INVALID_TERM, Last received: 2.633, Next index: 634, Last known committed idx: 632, Time since last communication: 0.000s
I20250901 01:52:21.072391 454 catalog_manager.cc:5582] T fcb7b5b7d63d4f0b906fea97cb548de2 P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: UNKNOWN } } }
I20250901 01:52:21.072947 454 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet fcb7b5b7d63d4f0b906fea97cb548de2 (table default.Table_1c8baaaa506548bd85b9f6a7483e5c7a [id=4d9ea39016ca4bf4859f4f80ea4e9ed6]). Expected version 3 got 0
W20250901 01:52:21.073822 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d4d52bbc912d428fa018d1044927c76f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250901 01:52:21.074162 454 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 5adf6b492640417ca071c94702b48cb9 (table default.Table_589c302e02474fc3bc5d3a290a8bd54a [id=5d14706774d345c48ca3575feabb17b0]). Expected version 2 got 0
I20250901 01:52:21.078553 1150 tablet_bootstrap.cc:492] T f8a5886ddb644a86b76df44aab309057 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:21.078747 1150 ts_tablet_manager.cc:1397] T f8a5886ddb644a86b76df44aab309057 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.053s user 0.018s sys 0.033s
I20250901 01:52:21.079589 1150 raft_consensus.cc:357] T f8a5886ddb644a86b76df44aab309057 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.079797 1150 raft_consensus.cc:383] T f8a5886ddb644a86b76df44aab309057 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.079860 1150 raft_consensus.cc:738] T f8a5886ddb644a86b76df44aab309057 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
W20250901 01:52:21.078876 485 rpcz_store.cc:269] 0901 01:52:20.104071 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:20.104131 (+ 60us) service_pool.cc:225] Handling call
0901 01:52:21.035096 (+930965us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250901 01:52:21.080104 1150 consensus_queue.cc:260] T f8a5886ddb644a86b76df44aab309057 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.082067 1150 ts_tablet_manager.cc:1428] T f8a5886ddb644a86b76df44aab309057 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20250901 01:52:21.082348 1150 tablet_bootstrap.cc:492] T 92f0ad59fbbb435ea872c669a52f854a P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:21.084532 1150 tablet_bootstrap.cc:654] T 92f0ad59fbbb435ea872c669a52f854a P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:21.087652 1711 raft_consensus.cc:491] T 40d74d7b5a3b4d1dae09eebd9e07a0c4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:21.087831 1711 raft_consensus.cc:513] T 40d74d7b5a3b4d1dae09eebd9e07a0c4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.088696 1711 leader_election.cc:290] T 40d74d7b5a3b4d1dae09eebd9e07a0c4 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:21.089912 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "40d74d7b5a3b4d1dae09eebd9e07a0c4" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:21.090665 859 leader_election.cc:343] T 40d74d7b5a3b4d1dae09eebd9e07a0c4 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:21.092860 1150 tablet_bootstrap.cc:492] T 92f0ad59fbbb435ea872c669a52f854a P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:21.092983 1150 ts_tablet_manager.cc:1397] T 92f0ad59fbbb435ea872c669a52f854a P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.009s
I20250901 01:52:21.093720 1150 raft_consensus.cc:357] T 92f0ad59fbbb435ea872c669a52f854a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.093863 1150 raft_consensus.cc:383] T 92f0ad59fbbb435ea872c669a52f854a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.093919 1150 raft_consensus.cc:738] T 92f0ad59fbbb435ea872c669a52f854a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:21.094125 1150 consensus_queue.cc:260] T 92f0ad59fbbb435ea872c669a52f854a P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.094583 1150 ts_tablet_manager.cc:1428] T 92f0ad59fbbb435ea872c669a52f854a P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250901 01:52:21.094833 1150 tablet_bootstrap.cc:492] T b2e8b2cd10154de08ddd30c214592a4a P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:21.096328 1150 tablet_bootstrap.cc:654] T b2e8b2cd10154de08ddd30c214592a4a P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:21.099330 416 external_mini_cluster.cc:1715] Unable to check leaks on /tmp/dist-test-taskddMNdm/build/asan/bin/kudu pid 424: Timed out: CheckLeaks RPC to 127.0.104.62:43871 timed out after 1.000s (SENT)
I20250901 01:52:21.099560 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskddMNdm/build/asan/bin/kudu with pid 424
I20250901 01:52:21.099625 1150 tablet_bootstrap.cc:492] T b2e8b2cd10154de08ddd30c214592a4a P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:21.099747 1150 ts_tablet_manager.cc:1397] T b2e8b2cd10154de08ddd30c214592a4a P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250901 01:52:21.100634 1150 raft_consensus.cc:357] T b2e8b2cd10154de08ddd30c214592a4a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.100788 1150 raft_consensus.cc:383] T b2e8b2cd10154de08ddd30c214592a4a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.100834 1150 raft_consensus.cc:738] T b2e8b2cd10154de08ddd30c214592a4a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:21.101029 1150 consensus_queue.cc:260] T b2e8b2cd10154de08ddd30c214592a4a P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.101536 1150 ts_tablet_manager.cc:1428] T b2e8b2cd10154de08ddd30c214592a4a P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250901 01:52:21.101789 1150 tablet_bootstrap.cc:492] T 670d90a37b934579b353bb4e4820127b P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:21.103286 1150 tablet_bootstrap.cc:654] T 670d90a37b934579b353bb4e4820127b P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:21.130336 1151 tablet_bootstrap.cc:492] T 533d14f5903e49599a34ea353ffd8c00 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:21.130553 1151 ts_tablet_manager.cc:1397] T 533d14f5903e49599a34ea353ffd8c00 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.110s user 0.001s sys 0.010s
I20250901 01:52:21.131386 1151 raft_consensus.cc:357] T 533d14f5903e49599a34ea353ffd8c00 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.131610 1151 raft_consensus.cc:383] T 533d14f5903e49599a34ea353ffd8c00 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.131697 1151 raft_consensus.cc:738] T 533d14f5903e49599a34ea353ffd8c00 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:21.131946 1151 consensus_queue.cc:260] T 533d14f5903e49599a34ea353ffd8c00 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.132472 1151 ts_tablet_manager.cc:1428] T 533d14f5903e49599a34ea353ffd8c00 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20250901 01:52:21.132795 1151 tablet_bootstrap.cc:492] T 82d163681fda47f38d79863bd28f576d P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:21.134622 1151 tablet_bootstrap.cc:654] T 82d163681fda47f38d79863bd28f576d P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:21.144168 1151 tablet_bootstrap.cc:492] T 82d163681fda47f38d79863bd28f576d P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:21.144343 1151 ts_tablet_manager.cc:1397] T 82d163681fda47f38d79863bd28f576d P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20250901 01:52:21.145131 1151 raft_consensus.cc:357] T 82d163681fda47f38d79863bd28f576d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.145383 1151 raft_consensus.cc:383] T 82d163681fda47f38d79863bd28f576d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.145484 1151 raft_consensus.cc:738] T 82d163681fda47f38d79863bd28f576d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:21.145738 1151 consensus_queue.cc:260] T 82d163681fda47f38d79863bd28f576d P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.146440 1151 ts_tablet_manager.cc:1428] T 82d163681fda47f38d79863bd28f576d P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250901 01:52:21.146770 1151 tablet_bootstrap.cc:492] T abe27e749fa0485195e0064a49d5431e P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
W20250901 01:52:21.146968 858 leader_election.cc:336] T 40d74d7b5a3b4d1dae09eebd9e07a0c4 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:21.147168 1150 tablet_bootstrap.cc:492] T 670d90a37b934579b353bb4e4820127b P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:21.147378 1150 ts_tablet_manager.cc:1397] T 670d90a37b934579b353bb4e4820127b P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.046s user 0.015s sys 0.027s
I20250901 01:52:21.148176 1150 raft_consensus.cc:357] T 670d90a37b934579b353bb4e4820127b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.148330 1150 raft_consensus.cc:383] T 670d90a37b934579b353bb4e4820127b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.148383 1150 raft_consensus.cc:738] T 670d90a37b934579b353bb4e4820127b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:21.147153 858 leader_election.cc:304] T 40d74d7b5a3b4d1dae09eebd9e07a0c4 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:21.148612 1150 consensus_queue.cc:260] T 670d90a37b934579b353bb4e4820127b P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.148988 1711 raft_consensus.cc:2747] T 40d74d7b5a3b4d1dae09eebd9e07a0c4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:21.149178 1150 ts_tablet_manager.cc:1428] T 670d90a37b934579b353bb4e4820127b P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250901 01:52:21.149585 1150 tablet_bootstrap.cc:492] T bb0cf5a6a27045a08df6845d7473184d P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:21.151253 1150 tablet_bootstrap.cc:654] T bb0cf5a6a27045a08df6845d7473184d P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:21.151546 1151 tablet_bootstrap.cc:654] T abe27e749fa0485195e0064a49d5431e P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:21.161541 1151 tablet_bootstrap.cc:492] T abe27e749fa0485195e0064a49d5431e P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:21.161746 1151 ts_tablet_manager.cc:1397] T abe27e749fa0485195e0064a49d5431e P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.008s
I20250901 01:52:21.162613 1151 raft_consensus.cc:357] T abe27e749fa0485195e0064a49d5431e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.162843 1151 raft_consensus.cc:383] T abe27e749fa0485195e0064a49d5431e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.162927 1151 raft_consensus.cc:738] T abe27e749fa0485195e0064a49d5431e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:21.163167 1151 consensus_queue.cc:260] T abe27e749fa0485195e0064a49d5431e P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.163906 1151 ts_tablet_manager.cc:1428] T abe27e749fa0485195e0064a49d5431e P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250901 01:52:21.164229 1151 tablet_bootstrap.cc:492] T c9a4c80bec1d41458974a0c449aecaf7 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:21.166344 1151 tablet_bootstrap.cc:654] T c9a4c80bec1d41458974a0c449aecaf7 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:21.168488 1151 tablet_bootstrap.cc:492] T c9a4c80bec1d41458974a0c449aecaf7 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:21.168627 1151 ts_tablet_manager.cc:1397] T c9a4c80bec1d41458974a0c449aecaf7 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20250901 01:52:21.169525 1151 raft_consensus.cc:357] T c9a4c80bec1d41458974a0c449aecaf7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.169783 1151 raft_consensus.cc:383] T c9a4c80bec1d41458974a0c449aecaf7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.169940 1151 raft_consensus.cc:738] T c9a4c80bec1d41458974a0c449aecaf7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:21.170217 1151 consensus_queue.cc:260] T c9a4c80bec1d41458974a0c449aecaf7 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.170925 1151 ts_tablet_manager.cc:1428] T c9a4c80bec1d41458974a0c449aecaf7 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250901 01:52:21.171252 1151 tablet_bootstrap.cc:492] T 5a8c21aee9f84216b6f90538a5bf4c17 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:21.172255 1711 raft_consensus.cc:491] T b2e8b2cd10154de08ddd30c214592a4a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:21.172462 1711 raft_consensus.cc:513] T b2e8b2cd10154de08ddd30c214592a4a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.173307 1711 leader_election.cc:290] T b2e8b2cd10154de08ddd30c214592a4a P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:21.173990 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2e8b2cd10154de08ddd30c214592a4a" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:21.174388 859 leader_election.cc:343] T b2e8b2cd10154de08ddd30c214592a4a P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
W20250901 01:52:21.175037 858 leader_election.cc:336] T b2e8b2cd10154de08ddd30c214592a4a P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:21.175137 858 leader_election.cc:304] T b2e8b2cd10154de08ddd30c214592a4a P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:21.175522 1711 raft_consensus.cc:2747] T b2e8b2cd10154de08ddd30c214592a4a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:21.176043 1151 tablet_bootstrap.cc:654] T 5a8c21aee9f84216b6f90538a5bf4c17 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:21.187978 992 connection.cc:537] server connection from 127.0.0.1:44246 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 01:52:21.187992 857 connection.cc:537] server connection from 127.0.0.1:52846 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250901 01:52:21.188915 1150 tablet_bootstrap.cc:492] T bb0cf5a6a27045a08df6845d7473184d P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
W20250901 01:52:21.189060 511 connection.cc:537] client connection to 127.0.104.62:43871 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250901 01:52:21.189158 1150 ts_tablet_manager.cc:1397] T bb0cf5a6a27045a08df6845d7473184d P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.040s user 0.014s sys 0.024s
W20250901 01:52:21.189203 511 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 1 similar messages]
W20250901 01:52:21.189775 1111 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.0.104.62:43871)
I20250901 01:52:21.189934 1150 raft_consensus.cc:357] T bb0cf5a6a27045a08df6845d7473184d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.190092 1150 raft_consensus.cc:383] T bb0cf5a6a27045a08df6845d7473184d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.190142 1150 raft_consensus.cc:738] T bb0cf5a6a27045a08df6845d7473184d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:21.190363 1150 consensus_queue.cc:260] T bb0cf5a6a27045a08df6845d7473184d P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:21.190577 1111 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.0.104.62:43871)
W20250901 01:52:21.190884 1111 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.0.104.62:43871)
I20250901 01:52:21.191030 1150 ts_tablet_manager.cc:1428] T bb0cf5a6a27045a08df6845d7473184d P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
W20250901 01:52:21.191285 1111 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.104.62:43871)
I20250901 01:52:21.191335 1150 tablet_bootstrap.cc:492] T fae2779fac164b9f8052c3666db803b8 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
W20250901 01:52:21.192732 858 connection.cc:537] client connection to 127.0.104.62:43871 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250901 01:52:21.192950 1150 tablet_bootstrap.cc:654] T fae2779fac164b9f8052c3666db803b8 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:21.197485 1822 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 01:52:21.194556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:43871 (local address 127.0.104.2:37281)
0901 01:52:21.194824 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0901 01:52:21.194846 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0901 01:52:21.194984 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 01:52:21.197266 (+ 2282us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:43871: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":139}
W20250901 01:52:21.197515 1759 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 01:52:21.195357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:43871 (local address 127.0.104.3:54271)
0901 01:52:21.195588 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0901 01:52:21.195607 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0901 01:52:21.195788 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 01:52:21.197240 (+ 1452us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:43871: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":112}
I20250901 01:52:21.200021 1151 tablet_bootstrap.cc:492] T 5a8c21aee9f84216b6f90538a5bf4c17 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:21.200690 1150 tablet_bootstrap.cc:492] T fae2779fac164b9f8052c3666db803b8 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:21.200839 1150 ts_tablet_manager.cc:1397] T fae2779fac164b9f8052c3666db803b8 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20250901 01:52:21.201373 1151 ts_tablet_manager.cc:1397] T 5a8c21aee9f84216b6f90538a5bf4c17 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.030s user 0.012s sys 0.001s
I20250901 01:52:21.201634 1150 raft_consensus.cc:357] T fae2779fac164b9f8052c3666db803b8 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.201814 1150 raft_consensus.cc:383] T fae2779fac164b9f8052c3666db803b8 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.201874 1150 raft_consensus.cc:738] T fae2779fac164b9f8052c3666db803b8 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:21.202096 1150 consensus_queue.cc:260] T fae2779fac164b9f8052c3666db803b8 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.202581 1150 ts_tablet_manager.cc:1428] T fae2779fac164b9f8052c3666db803b8 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250901 01:52:21.202826 1150 tablet_bootstrap.cc:492] T 9127f9c5239e445ba86012951b7c75d9 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:21.204483 1150 tablet_bootstrap.cc:654] T 9127f9c5239e445ba86012951b7c75d9 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:21.206460 1151 raft_consensus.cc:357] T 5a8c21aee9f84216b6f90538a5bf4c17 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.207029 1151 raft_consensus.cc:383] T 5a8c21aee9f84216b6f90538a5bf4c17 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.207304 1151 raft_consensus.cc:738] T 5a8c21aee9f84216b6f90538a5bf4c17 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:21.207494 1150 tablet_bootstrap.cc:492] T 9127f9c5239e445ba86012951b7c75d9 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:21.207613 1150 ts_tablet_manager.cc:1397] T 9127f9c5239e445ba86012951b7c75d9 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250901 01:52:21.207953 1151 consensus_queue.cc:260] T 5a8c21aee9f84216b6f90538a5bf4c17 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.208343 1150 raft_consensus.cc:357] T 9127f9c5239e445ba86012951b7c75d9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.208536 1150 raft_consensus.cc:383] T 9127f9c5239e445ba86012951b7c75d9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.208618 1150 raft_consensus.cc:738] T 9127f9c5239e445ba86012951b7c75d9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:21.208882 1150 consensus_queue.cc:260] T 9127f9c5239e445ba86012951b7c75d9 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.209283 1711 raft_consensus.cc:491] T 1286c0ab85704837a99b6e5a62184d77 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:21.209458 1150 ts_tablet_manager.cc:1428] T 9127f9c5239e445ba86012951b7c75d9 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250901 01:52:21.209429 1711 raft_consensus.cc:513] T 1286c0ab85704837a99b6e5a62184d77 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.209790 1150 tablet_bootstrap.cc:492] T 75e7a90815e449c39f386901a504143c P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:21.210168 1711 leader_election.cc:290] T 1286c0ab85704837a99b6e5a62184d77 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:21.210680 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1286c0ab85704837a99b6e5a62184d77" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:21.211159 859 leader_election.cc:343] T 1286c0ab85704837a99b6e5a62184d77 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
W20250901 01:52:21.212394 858 leader_election.cc:336] T 1286c0ab85704837a99b6e5a62184d77 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:21.212532 858 leader_election.cc:304] T 1286c0ab85704837a99b6e5a62184d77 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:21.212942 1711 raft_consensus.cc:2747] T 1286c0ab85704837a99b6e5a62184d77 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:21.213191 1150 tablet_bootstrap.cc:654] T 75e7a90815e449c39f386901a504143c P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:21.217993 1151 ts_tablet_manager.cc:1428] T 5a8c21aee9f84216b6f90538a5bf4c17 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
I20250901 01:52:21.218432 1151 tablet_bootstrap.cc:492] T b009cc95c819431ea9827a3e453b8da3 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:21.222512 1151 tablet_bootstrap.cc:654] T b009cc95c819431ea9827a3e453b8da3 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:21.223289 1711 raft_consensus.cc:491] T 5d794471dbed4477b9d52fd411fa4d10 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:21.223460 1711 raft_consensus.cc:513] T 5d794471dbed4477b9d52fd411fa4d10 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.224277 1711 leader_election.cc:290] T 5d794471dbed4477b9d52fd411fa4d10 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:21.224905 1761 raft_consensus.cc:491] T 12c04e0c1f804179ae23509cdb24d395 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:21.225069 1761 raft_consensus.cc:513] T 12c04e0c1f804179ae23509cdb24d395 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:21.225176 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d794471dbed4477b9d52fd411fa4d10" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:21.225729 1761 leader_election.cc:290] T 12c04e0c1f804179ae23509cdb24d395 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:21.227654 859 leader_election.cc:343] T 5d794471dbed4477b9d52fd411fa4d10 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
W20250901 01:52:21.228031 992 leader_election.cc:343] T 12c04e0c1f804179ae23509cdb24d395 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 12c04e0c1f804179ae23509cdb24d395
W20250901 01:52:21.228385 858 leader_election.cc:336] T 5d794471dbed4477b9d52fd411fa4d10 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:21.228536 858 leader_election.cc:304] T 5d794471dbed4477b9d52fd411fa4d10 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:21.228982 1711 raft_consensus.cc:2747] T 5d794471dbed4477b9d52fd411fa4d10 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:21.229823 992 leader_election.cc:336] T 12c04e0c1f804179ae23509cdb24d395 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:21.229945 992 leader_election.cc:304] T 12c04e0c1f804179ae23509cdb24d395 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:21.230331 1761 raft_consensus.cc:2747] T 12c04e0c1f804179ae23509cdb24d395 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:21.235155 1151 tablet_bootstrap.cc:492] T b009cc95c819431ea9827a3e453b8da3 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:21.235327 1151 ts_tablet_manager.cc:1397] T b009cc95c819431ea9827a3e453b8da3 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.003s
I20250901 01:52:21.236346 1151 raft_consensus.cc:357] T b009cc95c819431ea9827a3e453b8da3 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:21.236582 1151 raft_consensus.cc:383] T b009cc95c819431ea9827a3e453b8da3 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.236714 1151 raft_consensus.cc:738] T b009cc95c819431ea9827a3e453b8da3 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:21.236989 1151 consensus_queue.cc:260] T b009cc95c819431ea9827a3e453b8da3 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:21.237656 1151 ts_tablet_manager.cc:1428] T b009cc95c819431ea9827a3e453b8da3 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250901 01:52:21.238015 1151 tablet_bootstrap.cc:492] T f50f33de1b654663b9ef8a91e3436f11 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:21.240164 1151 tablet_bootstrap.cc:654] T f50f33de1b654663b9ef8a91e3436f11 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:21.247589 511 leader_election.cc:336] T 00000000000000000000000000000000 P d2ae2cb6af0f4b3a8f47ab1cdd6567d3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e3add7edb97a40ef9f7c6bc011d13227 (127.0.104.62:43871): Network error: Client connection negotiation failed: client connection to 127.0.104.62:43871: connect: Connection refused (error 111)
W20250901 01:52:21.248014 511 leader_election.cc:336] T 00000000000000000000000000000000 P d2ae2cb6af0f4b3a8f47ab1cdd6567d3 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer e3add7edb97a40ef9f7c6bc011d13227 (127.0.104.62:43871): Network error: Client connection negotiation failed: client connection to 127.0.104.62:43871: connect: Connection refused (error 111)
I20250901 01:52:21.248234 511 leader_election.cc:304] T 00000000000000000000000000000000 P d2ae2cb6af0f4b3a8f47ab1cdd6567d3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2ae2cb6af0f4b3a8f47ab1cdd6567d3; no voters: 7ce2288f784044eea010664e3def133f, e3add7edb97a40ef9f7c6bc011d13227
I20250901 01:52:21.248613 1782 raft_consensus.cc:2747] T 00000000000000000000000000000000 P d2ae2cb6af0f4b3a8f47ab1cdd6567d3 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250901 01:52:21.250308 1151 tablet_bootstrap.cc:492] T f50f33de1b654663b9ef8a91e3436f11 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:21.250495 1151 ts_tablet_manager.cc:1397] T f50f33de1b654663b9ef8a91e3436f11 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.004s
I20250901 01:52:21.251039 1150 tablet_bootstrap.cc:492] T 75e7a90815e449c39f386901a504143c P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:21.251217 1150 ts_tablet_manager.cc:1397] T 75e7a90815e449c39f386901a504143c P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.042s user 0.017s sys 0.021s
I20250901 01:52:21.251374 1151 raft_consensus.cc:357] T f50f33de1b654663b9ef8a91e3436f11 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:21.251592 1151 raft_consensus.cc:383] T f50f33de1b654663b9ef8a91e3436f11 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.251681 1151 raft_consensus.cc:738] T f50f33de1b654663b9ef8a91e3436f11 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:21.251955 1151 consensus_queue.cc:260] T f50f33de1b654663b9ef8a91e3436f11 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:21.252029 1150 raft_consensus.cc:357] T 75e7a90815e449c39f386901a504143c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.252172 1150 raft_consensus.cc:383] T 75e7a90815e449c39f386901a504143c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.252233 1150 raft_consensus.cc:738] T 75e7a90815e449c39f386901a504143c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:21.252740 1151 ts_tablet_manager.cc:1428] T f50f33de1b654663b9ef8a91e3436f11 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250901 01:52:21.253144 1151 tablet_bootstrap.cc:492] T 4a87e68850c4459eb2c8eafac89c6aa9 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:21.253202 1150 consensus_queue.cc:260] T 75e7a90815e449c39f386901a504143c P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.255235 1150 ts_tablet_manager.cc:1428] T 75e7a90815e449c39f386901a504143c P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250901 01:52:21.256587 1150 tablet_bootstrap.cc:492] T d58f69fad01f403eab74a47d3695ed83 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:21.258191 1150 tablet_bootstrap.cc:654] T d58f69fad01f403eab74a47d3695ed83 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:21.260175 1151 tablet_bootstrap.cc:654] T 4a87e68850c4459eb2c8eafac89c6aa9 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:21.261597 1711 raft_consensus.cc:491] T f06a77ecc841462396a8d5026c7327f4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:21.261742 1711 raft_consensus.cc:513] T f06a77ecc841462396a8d5026c7327f4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.262305 1711 leader_election.cc:290] T f06a77ecc841462396a8d5026c7327f4 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:21.262387 1150 tablet_bootstrap.cc:492] T d58f69fad01f403eab74a47d3695ed83 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:21.262526 1150 ts_tablet_manager.cc:1397] T d58f69fad01f403eab74a47d3695ed83 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20250901 01:52:21.263556 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f06a77ecc841462396a8d5026c7327f4" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:21.264076 1150 raft_consensus.cc:357] T d58f69fad01f403eab74a47d3695ed83 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.264257 1150 raft_consensus.cc:383] T d58f69fad01f403eab74a47d3695ed83 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.264312 1150 raft_consensus.cc:738] T d58f69fad01f403eab74a47d3695ed83 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
W20250901 01:52:21.264294 859 leader_election.cc:343] T f06a77ecc841462396a8d5026c7327f4 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
W20250901 01:52:21.264482 858 leader_election.cc:336] T f06a77ecc841462396a8d5026c7327f4 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:21.264573 858 leader_election.cc:304] T f06a77ecc841462396a8d5026c7327f4 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:21.264569 1150 consensus_queue.cc:260] T d58f69fad01f403eab74a47d3695ed83 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.264964 1711 raft_consensus.cc:2747] T f06a77ecc841462396a8d5026c7327f4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:21.265033 1150 ts_tablet_manager.cc:1428] T d58f69fad01f403eab74a47d3695ed83 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250901 01:52:21.265328 1150 tablet_bootstrap.cc:492] T a54af99bc10045e8b0b6868e5aed98b4 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:21.266875 1150 tablet_bootstrap.cc:654] T a54af99bc10045e8b0b6868e5aed98b4 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:21.268568 1761 raft_consensus.cc:491] T 824229183e024feaa975c7f0387c9c1a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:21.268850 1761 raft_consensus.cc:513] T 824229183e024feaa975c7f0387c9c1a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:21.269518 1761 leader_election.cc:290] T 824229183e024feaa975c7f0387c9c1a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:21.271497 992 leader_election.cc:343] T 824229183e024feaa975c7f0387c9c1a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 824229183e024feaa975c7f0387c9c1a
I20250901 01:52:21.271849 1150 tablet_bootstrap.cc:492] T a54af99bc10045e8b0b6868e5aed98b4 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
W20250901 01:52:21.271916 992 leader_election.cc:336] T 824229183e024feaa975c7f0387c9c1a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:21.271961 1150 ts_tablet_manager.cc:1397] T a54af99bc10045e8b0b6868e5aed98b4 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.002s
I20250901 01:52:21.272037 992 leader_election.cc:304] T 824229183e024feaa975c7f0387c9c1a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:21.272415 1761 raft_consensus.cc:2747] T 824229183e024feaa975c7f0387c9c1a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:21.272692 1150 raft_consensus.cc:357] T a54af99bc10045e8b0b6868e5aed98b4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:21.272862 1150 raft_consensus.cc:383] T a54af99bc10045e8b0b6868e5aed98b4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.272910 1150 raft_consensus.cc:738] T a54af99bc10045e8b0b6868e5aed98b4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:21.273133 1150 consensus_queue.cc:260] T a54af99bc10045e8b0b6868e5aed98b4 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:21.273667 1151 tablet_bootstrap.cc:492] T 4a87e68850c4459eb2c8eafac89c6aa9 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:21.273767 1150 ts_tablet_manager.cc:1428] T a54af99bc10045e8b0b6868e5aed98b4 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250901 01:52:21.273802 1151 ts_tablet_manager.cc:1397] T 4a87e68850c4459eb2c8eafac89c6aa9 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.002s
I20250901 01:52:21.274016 1150 tablet_bootstrap.cc:492] T 822382a1781b4304bce29355ce83730e P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:21.274510 1151 raft_consensus.cc:357] T 4a87e68850c4459eb2c8eafac89c6aa9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.274669 1151 raft_consensus.cc:383] T 4a87e68850c4459eb2c8eafac89c6aa9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.274767 1151 raft_consensus.cc:738] T 4a87e68850c4459eb2c8eafac89c6aa9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:21.275004 1151 consensus_queue.cc:260] T 4a87e68850c4459eb2c8eafac89c6aa9 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.275652 1150 tablet_bootstrap.cc:654] T 822382a1781b4304bce29355ce83730e P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:21.275660 1151 ts_tablet_manager.cc:1428] T 4a87e68850c4459eb2c8eafac89c6aa9 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250901 01:52:21.276885 1151 tablet_bootstrap.cc:492] T 2ff914352c7c43de800f3d54ecf63072 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:21.278772 1150 tablet_bootstrap.cc:492] T 822382a1781b4304bce29355ce83730e P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:21.278899 1150 ts_tablet_manager.cc:1397] T 822382a1781b4304bce29355ce83730e P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250901 01:52:21.279846 1151 tablet_bootstrap.cc:654] T 2ff914352c7c43de800f3d54ecf63072 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:21.283669 1151 tablet_bootstrap.cc:492] T 2ff914352c7c43de800f3d54ecf63072 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:21.283872 1151 ts_tablet_manager.cc:1397] T 2ff914352c7c43de800f3d54ecf63072 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.001s
I20250901 01:52:21.283994 1150 raft_consensus.cc:357] T 822382a1781b4304bce29355ce83730e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.284176 1150 raft_consensus.cc:383] T 822382a1781b4304bce29355ce83730e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.284228 1150 raft_consensus.cc:738] T 822382a1781b4304bce29355ce83730e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:21.284580 1151 raft_consensus.cc:357] T 2ff914352c7c43de800f3d54ecf63072 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.284767 1151 raft_consensus.cc:383] T 2ff914352c7c43de800f3d54ecf63072 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.284425 1150 consensus_queue.cc:260] T 822382a1781b4304bce29355ce83730e P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.284827 1151 raft_consensus.cc:738] T 2ff914352c7c43de800f3d54ecf63072 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:21.285045 1151 consensus_queue.cc:260] T 2ff914352c7c43de800f3d54ecf63072 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.285167 1150 ts_tablet_manager.cc:1428] T 822382a1781b4304bce29355ce83730e P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250901 01:52:21.286581 1151 ts_tablet_manager.cc:1428] T 2ff914352c7c43de800f3d54ecf63072 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250901 01:52:21.287530 1151 tablet_bootstrap.cc:492] T 606fac5c6efe48d1877104b01c23ba5b P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:21.289435 1150 tablet_bootstrap.cc:492] T 72ba689f9230411995dc96e2b45601df P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:21.290829 1151 tablet_bootstrap.cc:654] T 606fac5c6efe48d1877104b01c23ba5b P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:21.291148 1150 tablet_bootstrap.cc:654] T 72ba689f9230411995dc96e2b45601df P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:21.297793 1150 tablet_bootstrap.cc:492] T 72ba689f9230411995dc96e2b45601df P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:21.297940 1150 ts_tablet_manager.cc:1397] T 72ba689f9230411995dc96e2b45601df P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20250901 01:52:21.298554 1151 tablet_bootstrap.cc:492] T 606fac5c6efe48d1877104b01c23ba5b P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:21.298744 1151 ts_tablet_manager.cc:1397] T 606fac5c6efe48d1877104b01c23ba5b P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.004s
I20250901 01:52:21.298741 1150 raft_consensus.cc:357] T 72ba689f9230411995dc96e2b45601df P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.298914 1150 raft_consensus.cc:383] T 72ba689f9230411995dc96e2b45601df P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.298969 1150 raft_consensus.cc:738] T 72ba689f9230411995dc96e2b45601df P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:21.299156 1150 consensus_queue.cc:260] T 72ba689f9230411995dc96e2b45601df P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.299624 1150 ts_tablet_manager.cc:1428] T 72ba689f9230411995dc96e2b45601df P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250901 01:52:21.299901 1150 tablet_bootstrap.cc:492] T 121803e79ab64ec0b948b448646b9d0a P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:21.300372 1151 raft_consensus.cc:357] T 606fac5c6efe48d1877104b01c23ba5b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.300590 1151 raft_consensus.cc:383] T 606fac5c6efe48d1877104b01c23ba5b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.300678 1151 raft_consensus.cc:738] T 606fac5c6efe48d1877104b01c23ba5b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:21.300873 1151 consensus_queue.cc:260] T 606fac5c6efe48d1877104b01c23ba5b P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.301373 1151 ts_tablet_manager.cc:1428] T 606fac5c6efe48d1877104b01c23ba5b P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250901 01:52:21.301780 1150 tablet_bootstrap.cc:654] T 121803e79ab64ec0b948b448646b9d0a P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:21.301633 1151 tablet_bootstrap.cc:492] T 98cca86164db4252b1b49de88747822a P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:21.304051 1151 tablet_bootstrap.cc:654] T 98cca86164db4252b1b49de88747822a P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:21.305034 1761 raft_consensus.cc:491] T b541a6e14f114ab29a864cca11806316 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:21.305212 1761 raft_consensus.cc:513] T b541a6e14f114ab29a864cca11806316 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:21.306037 1761 leader_election.cc:290] T b541a6e14f114ab29a864cca11806316 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:21.316781 1150 tablet_bootstrap.cc:492] T 121803e79ab64ec0b948b448646b9d0a P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:21.316926 1150 ts_tablet_manager.cc:1397] T 121803e79ab64ec0b948b448646b9d0a P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.007s
I20250901 01:52:21.317237 1151 tablet_bootstrap.cc:492] T 98cca86164db4252b1b49de88747822a P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:21.317379 1151 ts_tablet_manager.cc:1397] T 98cca86164db4252b1b49de88747822a P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.012s
I20250901 01:52:21.318114 1150 raft_consensus.cc:357] T 121803e79ab64ec0b948b448646b9d0a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
W20250901 01:52:21.318212 992 leader_election.cc:343] T b541a6e14f114ab29a864cca11806316 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: b541a6e14f114ab29a864cca11806316
I20250901 01:52:21.318266 1150 raft_consensus.cc:383] T 121803e79ab64ec0b948b448646b9d0a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.318317 1150 raft_consensus.cc:738] T 121803e79ab64ec0b948b448646b9d0a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:21.318527 1150 consensus_queue.cc:260] T 121803e79ab64ec0b948b448646b9d0a P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
W20250901 01:52:21.318789 992 leader_election.cc:336] T b541a6e14f114ab29a864cca11806316 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:21.318888 992 leader_election.cc:304] T b541a6e14f114ab29a864cca11806316 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:21.319092 1150 ts_tablet_manager.cc:1428] T 121803e79ab64ec0b948b448646b9d0a P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250901 01:52:21.319377 1150 tablet_bootstrap.cc:492] T b6daeff0070a4db08e90020752d21c29 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:21.319713 1761 raft_consensus.cc:2747] T b541a6e14f114ab29a864cca11806316 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:21.319816 1151 raft_consensus.cc:357] T 98cca86164db4252b1b49de88747822a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.320004 1151 raft_consensus.cc:383] T 98cca86164db4252b1b49de88747822a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.320080 1151 raft_consensus.cc:738] T 98cca86164db4252b1b49de88747822a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:21.320346 1151 consensus_queue.cc:260] T 98cca86164db4252b1b49de88747822a P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.321360 1150 tablet_bootstrap.cc:654] T b6daeff0070a4db08e90020752d21c29 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:21.320899 1151 ts_tablet_manager.cc:1428] T 98cca86164db4252b1b49de88747822a P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250901 01:52:21.328338 1150 tablet_bootstrap.cc:492] T b6daeff0070a4db08e90020752d21c29 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:21.328356 1151 tablet_bootstrap.cc:492] T e6c10426f60f43e8bde69d9b1b9c8eb6 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:21.328558 1150 ts_tablet_manager.cc:1397] T b6daeff0070a4db08e90020752d21c29 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20250901 01:52:21.329412 1150 raft_consensus.cc:357] T b6daeff0070a4db08e90020752d21c29 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.329617 1150 raft_consensus.cc:383] T b6daeff0070a4db08e90020752d21c29 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.329701 1150 raft_consensus.cc:738] T b6daeff0070a4db08e90020752d21c29 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:21.329932 1150 consensus_queue.cc:260] T b6daeff0070a4db08e90020752d21c29 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.330276 1151 tablet_bootstrap.cc:654] T e6c10426f60f43e8bde69d9b1b9c8eb6 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:21.330444 1150 ts_tablet_manager.cc:1428] T b6daeff0070a4db08e90020752d21c29 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250901 01:52:21.330843 1150 tablet_bootstrap.cc:492] T 3c1fd0820d8a4995abe027f685da7f7b P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:21.333148 1150 tablet_bootstrap.cc:654] T 3c1fd0820d8a4995abe027f685da7f7b P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:21.338299 1150 tablet_bootstrap.cc:492] T 3c1fd0820d8a4995abe027f685da7f7b P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:21.338447 1150 ts_tablet_manager.cc:1397] T 3c1fd0820d8a4995abe027f685da7f7b P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20250901 01:52:21.338740 1151 tablet_bootstrap.cc:492] T e6c10426f60f43e8bde69d9b1b9c8eb6 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:21.338865 1151 ts_tablet_manager.cc:1397] T e6c10426f60f43e8bde69d9b1b9c8eb6 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.001s
I20250901 01:52:21.339259 1150 raft_consensus.cc:357] T 3c1fd0820d8a4995abe027f685da7f7b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.339452 1150 raft_consensus.cc:383] T 3c1fd0820d8a4995abe027f685da7f7b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.339556 1150 raft_consensus.cc:738] T 3c1fd0820d8a4995abe027f685da7f7b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:21.339604 1151 raft_consensus.cc:357] T e6c10426f60f43e8bde69d9b1b9c8eb6 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.339812 1151 raft_consensus.cc:383] T e6c10426f60f43e8bde69d9b1b9c8eb6 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.339905 1151 raft_consensus.cc:738] T e6c10426f60f43e8bde69d9b1b9c8eb6 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:21.339808 1150 consensus_queue.cc:260] T 3c1fd0820d8a4995abe027f685da7f7b P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.340178 1151 consensus_queue.cc:260] T e6c10426f60f43e8bde69d9b1b9c8eb6 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.340323 1150 ts_tablet_manager.cc:1428] T 3c1fd0820d8a4995abe027f685da7f7b P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250901 01:52:21.340634 1150 tablet_bootstrap.cc:492] T d77aadd7291647a0bae7c421f244c252 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:21.340772 1151 ts_tablet_manager.cc:1428] T e6c10426f60f43e8bde69d9b1b9c8eb6 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250901 01:52:21.341029 1151 tablet_bootstrap.cc:492] T 66a48a8ebe154b2cac89c234eaa378ff P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:21.342814 1150 tablet_bootstrap.cc:654] T d77aadd7291647a0bae7c421f244c252 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:21.342824 1151 tablet_bootstrap.cc:654] T 66a48a8ebe154b2cac89c234eaa378ff P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:21.353048 1711 raft_consensus.cc:491] T 75e7a90815e449c39f386901a504143c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:21.353368 1150 tablet_bootstrap.cc:492] T d77aadd7291647a0bae7c421f244c252 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:21.353258 1711 raft_consensus.cc:513] T 75e7a90815e449c39f386901a504143c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.353531 1150 ts_tablet_manager.cc:1397] T d77aadd7291647a0bae7c421f244c252 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.009s
I20250901 01:52:21.353961 1711 leader_election.cc:290] T 75e7a90815e449c39f386901a504143c P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:21.354547 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75e7a90815e449c39f386901a504143c" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:21.354816 1150 raft_consensus.cc:357] T d77aadd7291647a0bae7c421f244c252 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.355047 1150 raft_consensus.cc:383] T d77aadd7291647a0bae7c421f244c252 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.355139 1150 raft_consensus.cc:738] T d77aadd7291647a0bae7c421f244c252 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
W20250901 01:52:21.355156 859 leader_election.cc:343] T 75e7a90815e449c39f386901a504143c P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:21.355387 1150 consensus_queue.cc:260] T d77aadd7291647a0bae7c421f244c252 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.355932 1150 ts_tablet_manager.cc:1428] T d77aadd7291647a0bae7c421f244c252 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20250901 01:52:21.356094 858 leader_election.cc:336] T 75e7a90815e449c39f386901a504143c P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:21.356240 858 leader_election.cc:304] T 75e7a90815e449c39f386901a504143c P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:21.356509 1150 tablet_bootstrap.cc:492] T 0cb3262973934c8689b851adad60afe0 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:21.357216 1711 raft_consensus.cc:2747] T 75e7a90815e449c39f386901a504143c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:21.358610 1150 tablet_bootstrap.cc:654] T 0cb3262973934c8689b851adad60afe0 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:21.358867 1151 tablet_bootstrap.cc:492] T 66a48a8ebe154b2cac89c234eaa378ff P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:21.359045 1151 ts_tablet_manager.cc:1397] T 66a48a8ebe154b2cac89c234eaa378ff P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.007s
I20250901 01:52:21.360020 1151 raft_consensus.cc:357] T 66a48a8ebe154b2cac89c234eaa378ff P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.360280 1151 raft_consensus.cc:383] T 66a48a8ebe154b2cac89c234eaa378ff P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.360498 1151 raft_consensus.cc:738] T 66a48a8ebe154b2cac89c234eaa378ff P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:21.360803 1151 consensus_queue.cc:260] T 66a48a8ebe154b2cac89c234eaa378ff P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.362668 1151 ts_tablet_manager.cc:1428] T 66a48a8ebe154b2cac89c234eaa378ff P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250901 01:52:21.363761 1151 tablet_bootstrap.cc:492] T c95798f0daf4437db18a291c4464522e P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:21.364017 1711 raft_consensus.cc:491] T ddbc15685fca4c708329420109ad0c57 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:21.364159 1711 raft_consensus.cc:513] T ddbc15685fca4c708329420109ad0c57 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.364794 1711 leader_election.cc:290] T ddbc15685fca4c708329420109ad0c57 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:21.365404 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ddbc15685fca4c708329420109ad0c57" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:21.366266 859 leader_election.cc:343] T ddbc15685fca4c708329420109ad0c57 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:21.366902 1151 tablet_bootstrap.cc:654] T c95798f0daf4437db18a291c4464522e P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:21.369415 1150 tablet_bootstrap.cc:492] T 0cb3262973934c8689b851adad60afe0 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:21.369597 1150 ts_tablet_manager.cc:1397] T 0cb3262973934c8689b851adad60afe0 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.002s
I20250901 01:52:21.370608 1150 raft_consensus.cc:357] T 0cb3262973934c8689b851adad60afe0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.370860 1150 raft_consensus.cc:383] T 0cb3262973934c8689b851adad60afe0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.370977 1150 raft_consensus.cc:738] T 0cb3262973934c8689b851adad60afe0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:21.371276 1150 consensus_queue.cc:260] T 0cb3262973934c8689b851adad60afe0 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:21.371438 858 leader_election.cc:336] T ddbc15685fca4c708329420109ad0c57 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:21.371570 858 leader_election.cc:304] T ddbc15685fca4c708329420109ad0c57 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:21.371978 1150 ts_tablet_manager.cc:1428] T 0cb3262973934c8689b851adad60afe0 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250901 01:52:21.372066 1768 raft_consensus.cc:2747] T ddbc15685fca4c708329420109ad0c57 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:21.372360 1150 tablet_bootstrap.cc:492] T eac50cbae691416e903cb2cbc907f96d P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:21.374922 1150 tablet_bootstrap.cc:654] T eac50cbae691416e903cb2cbc907f96d P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:21.377596 1151 tablet_bootstrap.cc:492] T c95798f0daf4437db18a291c4464522e P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:21.377794 1151 ts_tablet_manager.cc:1397] T c95798f0daf4437db18a291c4464522e P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.008s
I20250901 01:52:21.378819 1151 raft_consensus.cc:357] T c95798f0daf4437db18a291c4464522e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.379052 1151 raft_consensus.cc:383] T c95798f0daf4437db18a291c4464522e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.379153 1151 raft_consensus.cc:738] T c95798f0daf4437db18a291c4464522e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:21.379468 1151 consensus_queue.cc:260] T c95798f0daf4437db18a291c4464522e P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.380980 1150 tablet_bootstrap.cc:492] T eac50cbae691416e903cb2cbc907f96d P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:21.381016 1151 ts_tablet_manager.cc:1428] T c95798f0daf4437db18a291c4464522e P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20250901 01:52:21.381315 1150 ts_tablet_manager.cc:1397] T eac50cbae691416e903cb2cbc907f96d P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20250901 01:52:21.382051 1151 tablet_bootstrap.cc:492] T 629db80a8ab444e582628578155638fd P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:21.382247 1150 raft_consensus.cc:357] T eac50cbae691416e903cb2cbc907f96d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.382432 1150 raft_consensus.cc:383] T eac50cbae691416e903cb2cbc907f96d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.382521 1150 raft_consensus.cc:738] T eac50cbae691416e903cb2cbc907f96d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:21.382755 1150 consensus_queue.cc:260] T eac50cbae691416e903cb2cbc907f96d P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.383293 1150 ts_tablet_manager.cc:1428] T eac50cbae691416e903cb2cbc907f96d P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250901 01:52:21.383652 1150 tablet_bootstrap.cc:492] T 50e3da53433f4c14b57db8a733fa53bc P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:21.383622 1768 raft_consensus.cc:491] T 831e42de95854541a49ae6b3e039d3e9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:21.383816 1768 raft_consensus.cc:513] T 831e42de95854541a49ae6b3e039d3e9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.383903 1151 tablet_bootstrap.cc:654] T 629db80a8ab444e582628578155638fd P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:21.384451 1768 leader_election.cc:290] T 831e42de95854541a49ae6b3e039d3e9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:21.385035 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "831e42de95854541a49ae6b3e039d3e9" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:21.386319 1150 tablet_bootstrap.cc:654] T 50e3da53433f4c14b57db8a733fa53bc P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:21.386873 859 leader_election.cc:343] T 831e42de95854541a49ae6b3e039d3e9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:21.393309 1768 raft_consensus.cc:491] T 3a8231590ea643b0a1f83eec4d9a8191 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:21.393497 1768 raft_consensus.cc:513] T 3a8231590ea643b0a1f83eec4d9a8191 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.394186 1768 leader_election.cc:290] T 3a8231590ea643b0a1f83eec4d9a8191 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:21.394642 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3a8231590ea643b0a1f83eec4d9a8191" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:21.395123 859 leader_election.cc:343] T 3a8231590ea643b0a1f83eec4d9a8191 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:21.395342 1150 tablet_bootstrap.cc:492] T 50e3da53433f4c14b57db8a733fa53bc P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:21.395501 1150 ts_tablet_manager.cc:1397] T 50e3da53433f4c14b57db8a733fa53bc P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.009s
I20250901 01:52:21.395601 1151 tablet_bootstrap.cc:492] T 629db80a8ab444e582628578155638fd P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:21.395769 1151 ts_tablet_manager.cc:1397] T 629db80a8ab444e582628578155638fd P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.000s
I20250901 01:52:21.396301 1150 raft_consensus.cc:357] T 50e3da53433f4c14b57db8a733fa53bc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:21.396420 858 leader_election.cc:336] T 831e42de95854541a49ae6b3e039d3e9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:21.396555 1150 raft_consensus.cc:383] T 50e3da53433f4c14b57db8a733fa53bc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.396644 858 leader_election.cc:304] T 831e42de95854541a49ae6b3e039d3e9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:21.396677 1150 raft_consensus.cc:738] T 50e3da53433f4c14b57db8a733fa53bc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:21.396906 1768 raft_consensus.cc:2747] T 831e42de95854541a49ae6b3e039d3e9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:21.396958 1150 consensus_queue.cc:260] T 50e3da53433f4c14b57db8a733fa53bc P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.397121 1151 raft_consensus.cc:357] T 629db80a8ab444e582628578155638fd P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.397291 1151 raft_consensus.cc:383] T 629db80a8ab444e582628578155638fd P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.397365 1151 raft_consensus.cc:738] T 629db80a8ab444e582628578155638fd P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:21.397540 1150 ts_tablet_manager.cc:1428] T 50e3da53433f4c14b57db8a733fa53bc P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250901 01:52:21.397619 1151 consensus_queue.cc:260] T 629db80a8ab444e582628578155638fd P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:21.397939 858 consensus_peers.cc:489] T 24ddc13d1bb04db4a7850acbff091577 P 00c282f779c1477d920f24dc0dc16628 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250901 01:52:21.398165 1150 tablet_bootstrap.cc:492] T 3c12d6a70ac24dce82e851782e6fe5aa P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:21.399050 1151 ts_tablet_manager.cc:1428] T 629db80a8ab444e582628578155638fd P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250901 01:52:21.400082 1151 tablet_bootstrap.cc:492] T ef58f477ebeb49639ed7d071e1a62c10 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:21.400584 1150 tablet_bootstrap.cc:654] T 3c12d6a70ac24dce82e851782e6fe5aa P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:21.401114 858 leader_election.cc:336] T 3a8231590ea643b0a1f83eec4d9a8191 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:21.401327 858 leader_election.cc:304] T 3a8231590ea643b0a1f83eec4d9a8191 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:21.401618 1768 raft_consensus.cc:2747] T 3a8231590ea643b0a1f83eec4d9a8191 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:21.402405 1151 tablet_bootstrap.cc:654] T ef58f477ebeb49639ed7d071e1a62c10 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:21.408756 1150 tablet_bootstrap.cc:492] T 3c12d6a70ac24dce82e851782e6fe5aa P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:21.408957 1150 ts_tablet_manager.cc:1397] T 3c12d6a70ac24dce82e851782e6fe5aa P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.008s
I20250901 01:52:21.409930 1150 raft_consensus.cc:357] T 3c12d6a70ac24dce82e851782e6fe5aa P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.410497 1150 raft_consensus.cc:383] T 3c12d6a70ac24dce82e851782e6fe5aa P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.410602 1150 raft_consensus.cc:738] T 3c12d6a70ac24dce82e851782e6fe5aa P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:21.410964 1150 consensus_queue.cc:260] T 3c12d6a70ac24dce82e851782e6fe5aa P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.411572 1150 ts_tablet_manager.cc:1428] T 3c12d6a70ac24dce82e851782e6fe5aa P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250901 01:52:21.413671 1150 tablet_bootstrap.cc:492] T 8dd587304e6342edbe96c9897b40e5c9 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:21.415596 1150 tablet_bootstrap.cc:654] T 8dd587304e6342edbe96c9897b40e5c9 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:21.420387 1150 tablet_bootstrap.cc:492] T 8dd587304e6342edbe96c9897b40e5c9 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:21.420557 1150 ts_tablet_manager.cc:1397] T 8dd587304e6342edbe96c9897b40e5c9 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20250901 01:52:21.421355 1150 raft_consensus.cc:357] T 8dd587304e6342edbe96c9897b40e5c9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.421553 1150 raft_consensus.cc:383] T 8dd587304e6342edbe96c9897b40e5c9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.421662 1150 raft_consensus.cc:738] T 8dd587304e6342edbe96c9897b40e5c9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:21.421793 1151 tablet_bootstrap.cc:492] T ef58f477ebeb49639ed7d071e1a62c10 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:21.421983 1151 ts_tablet_manager.cc:1397] T ef58f477ebeb49639ed7d071e1a62c10 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.006s
I20250901 01:52:21.421952 1150 consensus_queue.cc:260] T 8dd587304e6342edbe96c9897b40e5c9 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.423231 1150 ts_tablet_manager.cc:1428] T 8dd587304e6342edbe96c9897b40e5c9 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250901 01:52:21.423509 1150 tablet_bootstrap.cc:492] T b7b802f976f3471faaaecc964ec87e48 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:21.425351 1151 raft_consensus.cc:357] T ef58f477ebeb49639ed7d071e1a62c10 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.425628 1151 raft_consensus.cc:383] T ef58f477ebeb49639ed7d071e1a62c10 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.425769 1151 raft_consensus.cc:738] T ef58f477ebeb49639ed7d071e1a62c10 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:21.425949 1150 tablet_bootstrap.cc:654] T b7b802f976f3471faaaecc964ec87e48 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:21.426082 1151 consensus_queue.cc:260] T ef58f477ebeb49639ed7d071e1a62c10 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.426642 1151 ts_tablet_manager.cc:1428] T ef58f477ebeb49639ed7d071e1a62c10 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250901 01:52:21.427987 1768 raft_consensus.cc:491] T 50e3da53433f4c14b57db8a733fa53bc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:21.428186 1768 raft_consensus.cc:513] T 50e3da53433f4c14b57db8a733fa53bc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.429165 1151 tablet_bootstrap.cc:492] T 438cd35ff3f74c49b908d072560077db P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:21.429157 1768 leader_election.cc:290] T 50e3da53433f4c14b57db8a733fa53bc P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:21.430160 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50e3da53433f4c14b57db8a733fa53bc" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:21.431012 1151 tablet_bootstrap.cc:654] T 438cd35ff3f74c49b908d072560077db P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:21.432005 858 leader_election.cc:336] T 50e3da53433f4c14b57db8a733fa53bc P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:21.432262 859 leader_election.cc:343] T 50e3da53433f4c14b57db8a733fa53bc P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:21.432476 859 leader_election.cc:304] T 50e3da53433f4c14b57db8a733fa53bc P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:21.433087 1768 raft_consensus.cc:2747] T 50e3da53433f4c14b57db8a733fa53bc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:21.439045 1768 raft_consensus.cc:491] T b6daeff0070a4db08e90020752d21c29 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:21.439244 1768 raft_consensus.cc:513] T b6daeff0070a4db08e90020752d21c29 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.440066 1768 leader_election.cc:290] T b6daeff0070a4db08e90020752d21c29 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:21.441610 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b6daeff0070a4db08e90020752d21c29" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:21.442090 858 leader_election.cc:336] T b6daeff0070a4db08e90020752d21c29 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:21.442274 859 leader_election.cc:343] T b6daeff0070a4db08e90020752d21c29 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:21.442523 859 leader_election.cc:304] T b6daeff0070a4db08e90020752d21c29 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:21.443172 1768 raft_consensus.cc:2747] T b6daeff0070a4db08e90020752d21c29 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:21.448904 1768 raft_consensus.cc:491] T bb0cf5a6a27045a08df6845d7473184d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:21.449097 1768 raft_consensus.cc:513] T bb0cf5a6a27045a08df6845d7473184d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.449908 1768 leader_election.cc:290] T bb0cf5a6a27045a08df6845d7473184d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:21.451021 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb0cf5a6a27045a08df6845d7473184d" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:21.451498 1150 tablet_bootstrap.cc:492] T b7b802f976f3471faaaecc964ec87e48 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
W20250901 01:52:21.451709 859 leader_election.cc:343] T bb0cf5a6a27045a08df6845d7473184d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:21.452206 1150 ts_tablet_manager.cc:1397] T b7b802f976f3471faaaecc964ec87e48 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.029s user 0.007s sys 0.017s
W20250901 01:52:21.452263 858 leader_election.cc:336] T bb0cf5a6a27045a08df6845d7473184d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:21.452445 858 leader_election.cc:304] T bb0cf5a6a27045a08df6845d7473184d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:21.452981 1711 raft_consensus.cc:2747] T bb0cf5a6a27045a08df6845d7473184d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:21.453337 1150 raft_consensus.cc:357] T b7b802f976f3471faaaecc964ec87e48 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.453652 1150 raft_consensus.cc:383] T b7b802f976f3471faaaecc964ec87e48 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.453783 1150 raft_consensus.cc:738] T b7b802f976f3471faaaecc964ec87e48 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:21.454025 1150 consensus_queue.cc:260] T b7b802f976f3471faaaecc964ec87e48 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.454511 1151 tablet_bootstrap.cc:492] T 438cd35ff3f74c49b908d072560077db P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:21.454514 1150 ts_tablet_manager.cc:1428] T b7b802f976f3471faaaecc964ec87e48 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250901 01:52:21.454666 1151 ts_tablet_manager.cc:1397] T 438cd35ff3f74c49b908d072560077db P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.014s
I20250901 01:52:21.454909 1150 tablet_bootstrap.cc:492] T 579b9cdf819f4609a532e96bec093fd5 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:21.455597 1151 raft_consensus.cc:357] T 438cd35ff3f74c49b908d072560077db P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.455785 1151 raft_consensus.cc:383] T 438cd35ff3f74c49b908d072560077db P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.455857 1151 raft_consensus.cc:738] T 438cd35ff3f74c49b908d072560077db P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:21.456194 1151 consensus_queue.cc:260] T 438cd35ff3f74c49b908d072560077db P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.456773 1151 ts_tablet_manager.cc:1428] T 438cd35ff3f74c49b908d072560077db P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250901 01:52:21.456852 1150 tablet_bootstrap.cc:654] T 579b9cdf819f4609a532e96bec093fd5 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:21.463177 1151 tablet_bootstrap.cc:492] T 3fda93767e494a4684978c8522b829dd P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:21.464679 1150 tablet_bootstrap.cc:492] T 579b9cdf819f4609a532e96bec093fd5 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:21.465037 1150 ts_tablet_manager.cc:1397] T 579b9cdf819f4609a532e96bec093fd5 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.008s
I20250901 01:52:21.465924 1150 raft_consensus.cc:357] T 579b9cdf819f4609a532e96bec093fd5 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.466192 1150 raft_consensus.cc:383] T 579b9cdf819f4609a532e96bec093fd5 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.466276 1150 raft_consensus.cc:738] T 579b9cdf819f4609a532e96bec093fd5 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:21.466485 1150 consensus_queue.cc:260] T 579b9cdf819f4609a532e96bec093fd5 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.466903 1151 tablet_bootstrap.cc:654] T 3fda93767e494a4684978c8522b829dd P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:21.467003 1150 ts_tablet_manager.cc:1428] T 579b9cdf819f4609a532e96bec093fd5 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250901 01:52:21.467525 1150 tablet_bootstrap.cc:492] T 7b16ca1daf394a6da65a4af8da82ec33 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:21.468680 1711 raft_consensus.cc:491] T a54af99bc10045e8b0b6868e5aed98b4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:21.468818 1711 raft_consensus.cc:513] T a54af99bc10045e8b0b6868e5aed98b4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:21.469352 1150 tablet_bootstrap.cc:654] T 7b16ca1daf394a6da65a4af8da82ec33 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:21.469449 1711 leader_election.cc:290] T a54af99bc10045e8b0b6868e5aed98b4 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:21.470693 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a54af99bc10045e8b0b6868e5aed98b4" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:21.471267 859 leader_election.cc:343] T a54af99bc10045e8b0b6868e5aed98b4 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
W20250901 01:52:21.472165 858 leader_election.cc:336] T a54af99bc10045e8b0b6868e5aed98b4 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:21.472350 858 leader_election.cc:304] T a54af99bc10045e8b0b6868e5aed98b4 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:21.472739 1768 raft_consensus.cc:2747] T a54af99bc10045e8b0b6868e5aed98b4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:21.476598 1768 raft_consensus.cc:491] T 670d90a37b934579b353bb4e4820127b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:21.476805 1768 raft_consensus.cc:513] T 670d90a37b934579b353bb4e4820127b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.477447 1768 leader_election.cc:290] T 670d90a37b934579b353bb4e4820127b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:21.477800 1150 tablet_bootstrap.cc:492] T 7b16ca1daf394a6da65a4af8da82ec33 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:21.477980 1150 ts_tablet_manager.cc:1397] T 7b16ca1daf394a6da65a4af8da82ec33 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.007s
I20250901 01:52:21.478809 1150 raft_consensus.cc:357] T 7b16ca1daf394a6da65a4af8da82ec33 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.479018 1150 raft_consensus.cc:383] T 7b16ca1daf394a6da65a4af8da82ec33 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.479100 1150 raft_consensus.cc:738] T 7b16ca1daf394a6da65a4af8da82ec33 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
W20250901 01:52:21.479238 858 leader_election.cc:336] T 670d90a37b934579b353bb4e4820127b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:21.479380 1150 consensus_queue.cc:260] T 7b16ca1daf394a6da65a4af8da82ec33 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.479895 1150 ts_tablet_manager.cc:1428] T 7b16ca1daf394a6da65a4af8da82ec33 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250901 01:52:21.480191 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "670d90a37b934579b353bb4e4820127b" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:21.482426 859 leader_election.cc:343] T 670d90a37b934579b353bb4e4820127b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:21.482576 859 leader_election.cc:304] T 670d90a37b934579b353bb4e4820127b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:21.484100 1768 raft_consensus.cc:2747] T 670d90a37b934579b353bb4e4820127b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:21.485519 1151 tablet_bootstrap.cc:492] T 3fda93767e494a4684978c8522b829dd P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:21.485654 1151 ts_tablet_manager.cc:1397] T 3fda93767e494a4684978c8522b829dd P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.023s user 0.011s sys 0.006s
I20250901 01:52:21.485925 1150 tablet_bootstrap.cc:492] T c49d1143d54b4c319739488906b0e271 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:21.486375 1151 raft_consensus.cc:357] T 3fda93767e494a4684978c8522b829dd P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.486557 1151 raft_consensus.cc:383] T 3fda93767e494a4684978c8522b829dd P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.486649 1151 raft_consensus.cc:738] T 3fda93767e494a4684978c8522b829dd P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:21.486881 1151 consensus_queue.cc:260] T 3fda93767e494a4684978c8522b829dd P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.488160 1150 tablet_bootstrap.cc:654] T c49d1143d54b4c319739488906b0e271 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:21.495554 1151 ts_tablet_manager.cc:1428] T 3fda93767e494a4684978c8522b829dd P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20250901 01:52:21.499231 1761 raft_consensus.cc:491] T e6c10426f60f43e8bde69d9b1b9c8eb6 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:21.499420 1761 raft_consensus.cc:513] T e6c10426f60f43e8bde69d9b1b9c8eb6 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.500039 1761 leader_election.cc:290] T e6c10426f60f43e8bde69d9b1b9c8eb6 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:21.501315 917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e6c10426f60f43e8bde69d9b1b9c8eb6" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:21.501672 917 raft_consensus.cc:2466] T e6c10426f60f43e8bde69d9b1b9c8eb6 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
W20250901 01:52:21.504449 858 consensus_peers.cc:489] T b11e8f77243e4764b96548d1132b8e5a P 00c282f779c1477d920f24dc0dc16628 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250901 01:52:21.506585 992 leader_election.cc:304] T e6c10426f60f43e8bde69d9b1b9c8eb6 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:21.506950 1761 raft_consensus.cc:2802] T e6c10426f60f43e8bde69d9b1b9c8eb6 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:21.507063 1761 raft_consensus.cc:491] T e6c10426f60f43e8bde69d9b1b9c8eb6 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:21.507139 1761 raft_consensus.cc:3058] T e6c10426f60f43e8bde69d9b1b9c8eb6 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
W20250901 01:52:21.508494 992 leader_election.cc:336] T e6c10426f60f43e8bde69d9b1b9c8eb6 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:21.510339 1761 raft_consensus.cc:513] T e6c10426f60f43e8bde69d9b1b9c8eb6 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.510913 1761 leader_election.cc:290] T e6c10426f60f43e8bde69d9b1b9c8eb6 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:21.511929 917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e6c10426f60f43e8bde69d9b1b9c8eb6" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:21.512145 917 raft_consensus.cc:3058] T e6c10426f60f43e8bde69d9b1b9c8eb6 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:21.514443 1151 tablet_bootstrap.cc:492] T 1286c0ab85704837a99b6e5a62184d77 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:21.515714 917 raft_consensus.cc:2466] T e6c10426f60f43e8bde69d9b1b9c8eb6 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:21.516342 1768 raft_consensus.cc:491] T c763d252598745a088344aa0eb5fa3d3 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:21.516484 1151 tablet_bootstrap.cc:654] T 1286c0ab85704837a99b6e5a62184d77 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:21.516513 1768 raft_consensus.cc:513] T c763d252598745a088344aa0eb5fa3d3 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.517316 1768 leader_election.cc:290] T c763d252598745a088344aa0eb5fa3d3 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:21.520464 858 leader_election.cc:336] T c763d252598745a088344aa0eb5fa3d3 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:21.521055 1768 raft_consensus.cc:491] T e25923b2daf9429683b2d44a75a56189 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:21.521173 1768 raft_consensus.cc:513] T e25923b2daf9429683b2d44a75a56189 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.521608 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c763d252598745a088344aa0eb5fa3d3" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:21.521724 1768 leader_election.cc:290] T e25923b2daf9429683b2d44a75a56189 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:21.523550 858 leader_election.cc:336] T e25923b2daf9429683b2d44a75a56189 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:21.523954 992 leader_election.cc:336] T e6c10426f60f43e8bde69d9b1b9c8eb6 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:21.524336 859 leader_election.cc:343] T c763d252598745a088344aa0eb5fa3d3 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:21.524447 859 leader_election.cc:304] T c763d252598745a088344aa0eb5fa3d3 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:21.524643 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e25923b2daf9429683b2d44a75a56189" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:21.524816 1768 raft_consensus.cc:2747] T c763d252598745a088344aa0eb5fa3d3 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:21.525118 859 leader_election.cc:343] T e25923b2daf9429683b2d44a75a56189 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:21.525209 859 leader_election.cc:304] T e25923b2daf9429683b2d44a75a56189 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:21.525555 1768 raft_consensus.cc:2747] T e25923b2daf9429683b2d44a75a56189 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:21.529426 992 leader_election.cc:304] T e6c10426f60f43e8bde69d9b1b9c8eb6 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:21.529911 1761 raft_consensus.cc:2802] T e6c10426f60f43e8bde69d9b1b9c8eb6 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:21.531414 1150 tablet_bootstrap.cc:492] T c49d1143d54b4c319739488906b0e271 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:21.531551 1150 ts_tablet_manager.cc:1397] T c49d1143d54b4c319739488906b0e271 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.046s user 0.009s sys 0.033s
I20250901 01:52:21.532153 1150 raft_consensus.cc:357] T c49d1143d54b4c319739488906b0e271 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.532315 1150 raft_consensus.cc:383] T c49d1143d54b4c319739488906b0e271 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.532387 1150 raft_consensus.cc:738] T c49d1143d54b4c319739488906b0e271 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:21.532673 1150 consensus_queue.cc:260] T c49d1143d54b4c319739488906b0e271 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.534287 1761 raft_consensus.cc:695] T e6c10426f60f43e8bde69d9b1b9c8eb6 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:21.535068 1761 consensus_queue.cc:237] T e6c10426f60f43e8bde69d9b1b9c8eb6 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.535602 1872 consensus_queue.cc:1035] T 6c05eff658924ca4a9a1e7a37f917d67 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:21.536204 1150 ts_tablet_manager.cc:1428] T c49d1143d54b4c319739488906b0e271 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20250901 01:52:21.536506 1150 tablet_bootstrap.cc:492] T 49aaaec611784e6bac131efd58dbca51 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:21.537972 1150 tablet_bootstrap.cc:654] T 49aaaec611784e6bac131efd58dbca51 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:21.538832 1768 raft_consensus.cc:491] T ded33e0950b544509af11739353c620d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:21.539016 1768 raft_consensus.cc:513] T ded33e0950b544509af11739353c620d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.539594 1768 leader_election.cc:290] T ded33e0950b544509af11739353c620d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:21.541132 858 leader_election.cc:336] T ded33e0950b544509af11739353c620d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:21.544809 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ded33e0950b544509af11739353c620d" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:21.545696 859 leader_election.cc:343] T ded33e0950b544509af11739353c620d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:21.545809 859 leader_election.cc:304] T ded33e0950b544509af11739353c620d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:21.546128 1768 raft_consensus.cc:2747] T ded33e0950b544509af11739353c620d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:21.546756 1151 tablet_bootstrap.cc:492] T 1286c0ab85704837a99b6e5a62184d77 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:21.546932 1151 ts_tablet_manager.cc:1397] T 1286c0ab85704837a99b6e5a62184d77 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.033s user 0.008s sys 0.004s
I20250901 01:52:21.549456 1151 raft_consensus.cc:357] T 1286c0ab85704837a99b6e5a62184d77 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.549654 1151 raft_consensus.cc:383] T 1286c0ab85704837a99b6e5a62184d77 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.549737 1151 raft_consensus.cc:738] T 1286c0ab85704837a99b6e5a62184d77 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:21.552786 1151 consensus_queue.cc:260] T 1286c0ab85704837a99b6e5a62184d77 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.553321 1151 ts_tablet_manager.cc:1428] T 1286c0ab85704837a99b6e5a62184d77 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20250901 01:52:21.553617 1151 tablet_bootstrap.cc:492] T f62d96c37cca4143a4b098b7c69979e6 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:21.556071 1872 consensus_queue.cc:1035] T 5adf6b492640417ca071c94702b48cb9 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:21.591595 1151 tablet_bootstrap.cc:654] T f62d96c37cca4143a4b098b7c69979e6 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:21.593255 1881 raft_consensus.cc:491] T a774cc6ae3924d299127fd9ffb01ca57 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:21.593426 1881 raft_consensus.cc:513] T a774cc6ae3924d299127fd9ffb01ca57 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
W20250901 01:52:21.595351 992 leader_election.cc:336] T a774cc6ae3924d299127fd9ffb01ca57 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:21.595757 1881 leader_election.cc:290] T a774cc6ae3924d299127fd9ffb01ca57 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:21.596582 992 leader_election.cc:343] T a774cc6ae3924d299127fd9ffb01ca57 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: a774cc6ae3924d299127fd9ffb01ca57
I20250901 01:52:21.596721 992 leader_election.cc:304] T a774cc6ae3924d299127fd9ffb01ca57 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:21.597050 1881 raft_consensus.cc:2747] T a774cc6ae3924d299127fd9ffb01ca57 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:21.599180 1881 raft_consensus.cc:491] T c9a4c80bec1d41458974a0c449aecaf7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:21.599341 1881 raft_consensus.cc:513] T c9a4c80bec1d41458974a0c449aecaf7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.601372 1768 raft_consensus.cc:491] T b7b802f976f3471faaaecc964ec87e48 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:21.601534 1768 raft_consensus.cc:513] T b7b802f976f3471faaaecc964ec87e48 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.602312 1768 leader_election.cc:290] T b7b802f976f3471faaaecc964ec87e48 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:21.604336 858 leader_election.cc:336] T b7b802f976f3471faaaecc964ec87e48 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:21.604341 992 leader_election.cc:336] T c9a4c80bec1d41458974a0c449aecaf7 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:21.605019 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b7b802f976f3471faaaecc964ec87e48" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:21.605615 859 leader_election.cc:343] T b7b802f976f3471faaaecc964ec87e48 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:21.605734 859 leader_election.cc:304] T b7b802f976f3471faaaecc964ec87e48 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:21.606123 1768 raft_consensus.cc:2747] T b7b802f976f3471faaaecc964ec87e48 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:21.605402 1881 leader_election.cc:290] T c9a4c80bec1d41458974a0c449aecaf7 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:21.611572 1151 tablet_bootstrap.cc:492] T f62d96c37cca4143a4b098b7c69979e6 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:21.611761 1151 ts_tablet_manager.cc:1397] T f62d96c37cca4143a4b098b7c69979e6 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.058s user 0.001s sys 0.006s
W20250901 01:52:21.612542 992 leader_election.cc:343] T c9a4c80bec1d41458974a0c449aecaf7 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: c9a4c80bec1d41458974a0c449aecaf7
I20250901 01:52:21.612682 992 leader_election.cc:304] T c9a4c80bec1d41458974a0c449aecaf7 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:21.613032 1881 raft_consensus.cc:2747] T c9a4c80bec1d41458974a0c449aecaf7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:21.613562 1151 raft_consensus.cc:357] T f62d96c37cca4143a4b098b7c69979e6 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.613778 1151 raft_consensus.cc:383] T f62d96c37cca4143a4b098b7c69979e6 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.613911 1151 raft_consensus.cc:738] T f62d96c37cca4143a4b098b7c69979e6 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:21.614727 1881 raft_consensus.cc:491] T 533d14f5903e49599a34ea353ffd8c00 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:21.614877 1881 raft_consensus.cc:513] T 533d14f5903e49599a34ea353ffd8c00 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.616043 1881 leader_election.cc:290] T 533d14f5903e49599a34ea353ffd8c00 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:21.619089 992 leader_election.cc:343] T 533d14f5903e49599a34ea353ffd8c00 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 533d14f5903e49599a34ea353ffd8c00
W20250901 01:52:21.619855 992 leader_election.cc:336] T 533d14f5903e49599a34ea353ffd8c00 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:21.620079 992 leader_election.cc:304] T 533d14f5903e49599a34ea353ffd8c00 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:21.620487 1768 raft_consensus.cc:491] T f8a5886ddb644a86b76df44aab309057 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:21.620679 1768 raft_consensus.cc:513] T f8a5886ddb644a86b76df44aab309057 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.620991 1884 raft_consensus.cc:2747] T 533d14f5903e49599a34ea353ffd8c00 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:21.621272 992 consensus_peers.cc:489] T 5adf6b492640417ca071c94702b48cb9 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:21.621429 1768 leader_election.cc:290] T f8a5886ddb644a86b76df44aab309057 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:21.623903 858 leader_election.cc:336] T f8a5886ddb644a86b76df44aab309057 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:21.626089 1768 raft_consensus.cc:491] T 8dd587304e6342edbe96c9897b40e5c9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:21.626256 1768 raft_consensus.cc:513] T 8dd587304e6342edbe96c9897b40e5c9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.627717 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f8a5886ddb644a86b76df44aab309057" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:21.627856 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8dd587304e6342edbe96c9897b40e5c9" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:21.628652 859 leader_election.cc:343] T f8a5886ddb644a86b76df44aab309057 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:21.628804 859 leader_election.cc:304] T f8a5886ddb644a86b76df44aab309057 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:21.629163 992 consensus_peers.cc:489] T 6c05eff658924ca4a9a1e7a37f917d67 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250901 01:52:21.629284 859 leader_election.cc:343] T 8dd587304e6342edbe96c9897b40e5c9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:21.629412 1151 consensus_queue.cc:260] T f62d96c37cca4143a4b098b7c69979e6 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.629498 1711 raft_consensus.cc:2747] T f8a5886ddb644a86b76df44aab309057 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:21.629998 1151 ts_tablet_manager.cc:1428] T f62d96c37cca4143a4b098b7c69979e6 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.018s user 0.000s sys 0.001s
W20250901 01:52:21.630187 858 leader_election.cc:336] T 8dd587304e6342edbe96c9897b40e5c9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:21.630297 858 leader_election.cc:304] T 8dd587304e6342edbe96c9897b40e5c9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:21.630599 1711 raft_consensus.cc:2747] T 8dd587304e6342edbe96c9897b40e5c9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:21.630880 1768 leader_election.cc:290] T 8dd587304e6342edbe96c9897b40e5c9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:21.632767 1151 tablet_bootstrap.cc:492] T b73e362321284df1b18ef755dfe930a5 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:21.633760 1150 tablet_bootstrap.cc:492] T 49aaaec611784e6bac131efd58dbca51 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:21.633886 1150 ts_tablet_manager.cc:1397] T 49aaaec611784e6bac131efd58dbca51 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.097s user 0.040s sys 0.048s
I20250901 01:52:21.634508 1151 tablet_bootstrap.cc:654] T b73e362321284df1b18ef755dfe930a5 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:21.634579 1150 raft_consensus.cc:357] T 49aaaec611784e6bac131efd58dbca51 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.634721 1150 raft_consensus.cc:383] T 49aaaec611784e6bac131efd58dbca51 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.634768 1150 raft_consensus.cc:738] T 49aaaec611784e6bac131efd58dbca51 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:21.634961 1150 consensus_queue.cc:260] T 49aaaec611784e6bac131efd58dbca51 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.637728 1151 tablet_bootstrap.cc:492] T b73e362321284df1b18ef755dfe930a5 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:21.637879 1151 ts_tablet_manager.cc:1397] T b73e362321284df1b18ef755dfe930a5 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20250901 01:52:21.638067 1150 ts_tablet_manager.cc:1428] T 49aaaec611784e6bac131efd58dbca51 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250901 01:52:21.638338 1150 tablet_bootstrap.cc:492] T af2c4bcbb00741f789edee109252d8cb P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:21.638525 1151 raft_consensus.cc:357] T b73e362321284df1b18ef755dfe930a5 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.638729 1151 raft_consensus.cc:383] T b73e362321284df1b18ef755dfe930a5 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.638787 1151 raft_consensus.cc:738] T b73e362321284df1b18ef755dfe930a5 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:21.639021 1151 consensus_queue.cc:260] T b73e362321284df1b18ef755dfe930a5 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.640087 1150 tablet_bootstrap.cc:654] T af2c4bcbb00741f789edee109252d8cb P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:21.640187 1151 ts_tablet_manager.cc:1428] T b73e362321284df1b18ef755dfe930a5 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250901 01:52:21.641381 1151 tablet_bootstrap.cc:492] T 9473f0ef978c4bfaa41fb7b0729e8863 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:21.642895 1151 tablet_bootstrap.cc:654] T 9473f0ef978c4bfaa41fb7b0729e8863 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:21.650768 1151 tablet_bootstrap.cc:492] T 9473f0ef978c4bfaa41fb7b0729e8863 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:21.650921 1151 ts_tablet_manager.cc:1397] T 9473f0ef978c4bfaa41fb7b0729e8863 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.002s
I20250901 01:52:21.651685 1151 raft_consensus.cc:357] T 9473f0ef978c4bfaa41fb7b0729e8863 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.651854 1151 raft_consensus.cc:383] T 9473f0ef978c4bfaa41fb7b0729e8863 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.651921 1151 raft_consensus.cc:738] T 9473f0ef978c4bfaa41fb7b0729e8863 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:21.652118 1151 consensus_queue.cc:260] T 9473f0ef978c4bfaa41fb7b0729e8863 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.652143 1768 raft_consensus.cc:491] T c49d1143d54b4c319739488906b0e271 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:21.652329 1768 raft_consensus.cc:513] T c49d1143d54b4c319739488906b0e271 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.653124 1768 leader_election.cc:290] T c49d1143d54b4c319739488906b0e271 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:21.653422 1151 ts_tablet_manager.cc:1428] T 9473f0ef978c4bfaa41fb7b0729e8863 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250901 01:52:21.653796 1150 tablet_bootstrap.cc:492] T af2c4bcbb00741f789edee109252d8cb P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:21.653926 1150 ts_tablet_manager.cc:1397] T af2c4bcbb00741f789edee109252d8cb P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.003s
I20250901 01:52:21.654253 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c49d1143d54b4c319739488906b0e271" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:21.654666 859 leader_election.cc:343] T c49d1143d54b4c319739488906b0e271 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:21.654696 1150 raft_consensus.cc:357] T af2c4bcbb00741f789edee109252d8cb P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.654871 1150 raft_consensus.cc:383] T af2c4bcbb00741f789edee109252d8cb P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.654912 1150 raft_consensus.cc:738] T af2c4bcbb00741f789edee109252d8cb P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:21.655095 1150 consensus_queue.cc:260] T af2c4bcbb00741f789edee109252d8cb P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.655468 1884 raft_consensus.cc:491] T 4d27fb61943c45d4aca67689cc3a9ff3 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:21.655560 1150 ts_tablet_manager.cc:1428] T af2c4bcbb00741f789edee109252d8cb P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250901 01:52:21.655609 1884 raft_consensus.cc:513] T 4d27fb61943c45d4aca67689cc3a9ff3 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:21.656092 1884 leader_election.cc:290] T 4d27fb61943c45d4aca67689cc3a9ff3 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:21.656337 1768 raft_consensus.cc:491] T af2c4bcbb00741f789edee109252d8cb P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:21.656474 1768 raft_consensus.cc:513] T af2c4bcbb00741f789edee109252d8cb P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.656924 1768 leader_election.cc:290] T af2c4bcbb00741f789edee109252d8cb P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:21.657337 1151 tablet_bootstrap.cc:492] T 3a8231590ea643b0a1f83eec4d9a8191 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
W20250901 01:52:21.658831 858 leader_election.cc:336] T c49d1143d54b4c319739488906b0e271 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:21.658932 858 leader_election.cc:304] T c49d1143d54b4c319739488906b0e271 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:21.659090 1884 raft_consensus.cc:491] T b73e362321284df1b18ef755dfe930a5 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:21.659197 1884 raft_consensus.cc:513] T b73e362321284df1b18ef755dfe930a5 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.659638 1768 raft_consensus.cc:2747] T c49d1143d54b4c319739488906b0e271 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:21.659703 1884 leader_election.cc:290] T b73e362321284df1b18ef755dfe930a5 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:21.660207 1151 tablet_bootstrap.cc:654] T 3a8231590ea643b0a1f83eec4d9a8191 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:21.660499 992 leader_election.cc:343] T 4d27fb61943c45d4aca67689cc3a9ff3 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 4d27fb61943c45d4aca67689cc3a9ff3
W20250901 01:52:21.660898 992 leader_election.cc:336] T 4d27fb61943c45d4aca67689cc3a9ff3 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:21.661010 992 leader_election.cc:304] T 4d27fb61943c45d4aca67689cc3a9ff3 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
W20250901 01:52:21.661355 858 leader_election.cc:336] T af2c4bcbb00741f789edee109252d8cb P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:21.661417 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "af2c4bcbb00741f789edee109252d8cb" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:21.661792 1884 raft_consensus.cc:2747] T 4d27fb61943c45d4aca67689cc3a9ff3 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:21.662266 859 leader_election.cc:343] T af2c4bcbb00741f789edee109252d8cb P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:21.662367 859 leader_election.cc:304] T af2c4bcbb00741f789edee109252d8cb P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:21.662699 992 leader_election.cc:343] T b73e362321284df1b18ef755dfe930a5 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: b73e362321284df1b18ef755dfe930a5
I20250901 01:52:21.662735 1768 raft_consensus.cc:2747] T af2c4bcbb00741f789edee109252d8cb P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:21.662853 992 leader_election.cc:336] T b73e362321284df1b18ef755dfe930a5 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:21.662955 992 leader_election.cc:304] T b73e362321284df1b18ef755dfe930a5 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:21.663273 1884 raft_consensus.cc:2747] T b73e362321284df1b18ef755dfe930a5 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:21.665494 1150 tablet_bootstrap.cc:492] T e1d18e999b5e42b1873631e0f1445081 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:21.666915 1150 tablet_bootstrap.cc:654] T e1d18e999b5e42b1873631e0f1445081 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:21.671002 1768 raft_consensus.cc:491] T b009cc95c819431ea9827a3e453b8da3 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:21.671173 1768 raft_consensus.cc:513] T b009cc95c819431ea9827a3e453b8da3 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:21.671890 1768 leader_election.cc:290] T b009cc95c819431ea9827a3e453b8da3 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:21.672734 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b009cc95c819431ea9827a3e453b8da3" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:21.672945 1056 raft_consensus.cc:2466] T b009cc95c819431ea9827a3e453b8da3 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00c282f779c1477d920f24dc0dc16628 in term 0.
I20250901 01:52:21.673234 1768 raft_consensus.cc:491] T 7b16ca1daf394a6da65a4af8da82ec33 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:21.673331 1768 raft_consensus.cc:513] T 7b16ca1daf394a6da65a4af8da82ec33 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.673791 1768 leader_election.cc:290] T 7b16ca1daf394a6da65a4af8da82ec33 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:21.674852 859 leader_election.cc:304] T b009cc95c819431ea9827a3e453b8da3 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:21.675125 1768 raft_consensus.cc:2802] T b009cc95c819431ea9827a3e453b8da3 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:21.675191 1768 raft_consensus.cc:491] T b009cc95c819431ea9827a3e453b8da3 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:21.675247 1768 raft_consensus.cc:3058] T b009cc95c819431ea9827a3e453b8da3 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:21.675419 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7b16ca1daf394a6da65a4af8da82ec33" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:21.677583 859 leader_election.cc:343] T 7b16ca1daf394a6da65a4af8da82ec33 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:21.678241 1768 raft_consensus.cc:513] T b009cc95c819431ea9827a3e453b8da3 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:21.679270 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b009cc95c819431ea9827a3e453b8da3" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58"
I20250901 01:52:21.679443 1056 raft_consensus.cc:3058] T b009cc95c819431ea9827a3e453b8da3 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
W20250901 01:52:21.679772 858 leader_election.cc:336] T b009cc95c819431ea9827a3e453b8da3 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:21.681463 858 leader_election.cc:336] T b009cc95c819431ea9827a3e453b8da3 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:21.681595 858 leader_election.cc:336] T 7b16ca1daf394a6da65a4af8da82ec33 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:21.681691 858 leader_election.cc:304] T 7b16ca1daf394a6da65a4af8da82ec33 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:21.681939 1711 raft_consensus.cc:2747] T 7b16ca1daf394a6da65a4af8da82ec33 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:21.682160 1056 raft_consensus.cc:2466] T b009cc95c819431ea9827a3e453b8da3 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c282f779c1477d920f24dc0dc16628 in term 1.
I20250901 01:52:21.682721 859 leader_election.cc:304] T b009cc95c819431ea9827a3e453b8da3 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:21.682942 1711 raft_consensus.cc:2802] T b009cc95c819431ea9827a3e453b8da3 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:21.683923 1151 tablet_bootstrap.cc:492] T 3a8231590ea643b0a1f83eec4d9a8191 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:21.684100 1151 ts_tablet_manager.cc:1397] T 3a8231590ea643b0a1f83eec4d9a8191 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.027s user 0.005s sys 0.007s
I20250901 01:52:21.684875 1151 raft_consensus.cc:357] T 3a8231590ea643b0a1f83eec4d9a8191 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.685101 1151 raft_consensus.cc:383] T 3a8231590ea643b0a1f83eec4d9a8191 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.685187 1151 raft_consensus.cc:738] T 3a8231590ea643b0a1f83eec4d9a8191 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:21.685483 1151 consensus_queue.cc:260] T 3a8231590ea643b0a1f83eec4d9a8191 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.686762 1151 ts_tablet_manager.cc:1428] T 3a8231590ea643b0a1f83eec4d9a8191 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250901 01:52:21.682974 1768 leader_election.cc:290] T b009cc95c819431ea9827a3e453b8da3 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:21.688328 1711 raft_consensus.cc:695] T b009cc95c819431ea9827a3e453b8da3 P 00c282f779c1477d920f24dc0dc16628 [term 1 LEADER]: Becoming Leader. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Running, Role: LEADER
I20250901 01:52:21.689285 1711 consensus_queue.cc:237] T b009cc95c819431ea9827a3e453b8da3 P 00c282f779c1477d920f24dc0dc16628 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:21.698441 1884 raft_consensus.cc:491] T 49d30da95b14400b845270122487a2a0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:21.698604 1884 raft_consensus.cc:513] T 49d30da95b14400b845270122487a2a0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:21.699102 1884 leader_election.cc:290] T 49d30da95b14400b845270122487a2a0 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:21.699443 1884 raft_consensus.cc:491] T 9473f0ef978c4bfaa41fb7b0729e8863 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:21.699621 1884 raft_consensus.cc:513] T 9473f0ef978c4bfaa41fb7b0729e8863 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.700371 1884 leader_election.cc:290] T 9473f0ef978c4bfaa41fb7b0729e8863 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:21.701509 992 leader_election.cc:336] T 49d30da95b14400b845270122487a2a0 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:21.701838 992 leader_election.cc:343] T 49d30da95b14400b845270122487a2a0 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 49d30da95b14400b845270122487a2a0
I20250901 01:52:21.701951 992 leader_election.cc:304] T 49d30da95b14400b845270122487a2a0 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
W20250901 01:52:21.702993 992 leader_election.cc:336] T 9473f0ef978c4bfaa41fb7b0729e8863 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:21.703974 1151 tablet_bootstrap.cc:492] T f06a77ecc841462396a8d5026c7327f4 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:21.704736 1884 raft_consensus.cc:2747] T 49d30da95b14400b845270122487a2a0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:21.704845 1150 tablet_bootstrap.cc:492] T e1d18e999b5e42b1873631e0f1445081 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
W20250901 01:52:21.704892 992 leader_election.cc:343] T 9473f0ef978c4bfaa41fb7b0729e8863 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 9473f0ef978c4bfaa41fb7b0729e8863
I20250901 01:52:21.704969 1150 ts_tablet_manager.cc:1397] T e1d18e999b5e42b1873631e0f1445081 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.040s user 0.009s sys 0.029s
I20250901 01:52:21.705013 992 leader_election.cc:304] T 9473f0ef978c4bfaa41fb7b0729e8863 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:21.705395 1884 raft_consensus.cc:2747] T 9473f0ef978c4bfaa41fb7b0729e8863 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:21.705981 1150 raft_consensus.cc:357] T e1d18e999b5e42b1873631e0f1445081 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.706143 1150 raft_consensus.cc:383] T e1d18e999b5e42b1873631e0f1445081 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.706192 1150 raft_consensus.cc:738] T e1d18e999b5e42b1873631e0f1445081 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:21.706395 1150 consensus_queue.cc:260] T e1d18e999b5e42b1873631e0f1445081 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.706643 1890 raft_consensus.cc:491] T d58f69fad01f403eab74a47d3695ed83 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:21.706799 1151 tablet_bootstrap.cc:654] T f06a77ecc841462396a8d5026c7327f4 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:21.706759 1890 raft_consensus.cc:513] T d58f69fad01f403eab74a47d3695ed83 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.707288 1890 leader_election.cc:290] T d58f69fad01f403eab74a47d3695ed83 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:21.709132 1711 raft_consensus.cc:491] T 579b9cdf819f4609a532e96bec093fd5 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:21.709278 1711 raft_consensus.cc:513] T 579b9cdf819f4609a532e96bec093fd5 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.709836 1711 leader_election.cc:290] T 579b9cdf819f4609a532e96bec093fd5 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:21.710886 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "579b9cdf819f4609a532e96bec093fd5" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:21.711441 859 leader_election.cc:343] T 579b9cdf819f4609a532e96bec093fd5 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:21.711681 1150 ts_tablet_manager.cc:1428] T e1d18e999b5e42b1873631e0f1445081 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250901 01:52:21.711983 1150 tablet_bootstrap.cc:492] T 616ff34e8d314e0aa0e11ce60be11789 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:21.712595 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d58f69fad01f403eab74a47d3695ed83" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:21.713054 859 leader_election.cc:343] T d58f69fad01f403eab74a47d3695ed83 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:21.713696 1150 tablet_bootstrap.cc:654] T 616ff34e8d314e0aa0e11ce60be11789 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:21.726110 1150 tablet_bootstrap.cc:492] T 616ff34e8d314e0aa0e11ce60be11789 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:21.726260 1150 ts_tablet_manager.cc:1397] T 616ff34e8d314e0aa0e11ce60be11789 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.006s
W20250901 01:52:21.726722 858 leader_election.cc:336] T d58f69fad01f403eab74a47d3695ed83 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:21.726817 858 leader_election.cc:304] T d58f69fad01f403eab74a47d3695ed83 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:21.726960 1150 raft_consensus.cc:357] T 616ff34e8d314e0aa0e11ce60be11789 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.727031 1711 raft_consensus.cc:2747] T d58f69fad01f403eab74a47d3695ed83 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:21.727092 1150 raft_consensus.cc:383] T 616ff34e8d314e0aa0e11ce60be11789 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250901 01:52:21.727095 858 leader_election.cc:336] T 579b9cdf819f4609a532e96bec093fd5 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:21.727212 1150 raft_consensus.cc:738] T 616ff34e8d314e0aa0e11ce60be11789 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:21.727257 858 leader_election.cc:304] T 579b9cdf819f4609a532e96bec093fd5 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:21.727394 1150 consensus_queue.cc:260] T 616ff34e8d314e0aa0e11ce60be11789 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.727488 1711 raft_consensus.cc:2747] T 579b9cdf819f4609a532e96bec093fd5 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:21.727973 1150 ts_tablet_manager.cc:1428] T 616ff34e8d314e0aa0e11ce60be11789 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250901 01:52:21.728677 1150 tablet_bootstrap.cc:492] T 20f7bfab18134e079e91c52c89468cc3 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:21.729918 1151 tablet_bootstrap.cc:492] T f06a77ecc841462396a8d5026c7327f4 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:21.730046 1150 tablet_bootstrap.cc:654] T 20f7bfab18134e079e91c52c89468cc3 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:21.730075 1151 ts_tablet_manager.cc:1397] T f06a77ecc841462396a8d5026c7327f4 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.027s user 0.012s sys 0.005s
I20250901 01:52:21.730845 1151 raft_consensus.cc:357] T f06a77ecc841462396a8d5026c7327f4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.732060 1151 raft_consensus.cc:383] T f06a77ecc841462396a8d5026c7327f4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.732156 1151 raft_consensus.cc:738] T f06a77ecc841462396a8d5026c7327f4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:21.732374 1151 consensus_queue.cc:260] T f06a77ecc841462396a8d5026c7327f4 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.734138 1151 ts_tablet_manager.cc:1428] T f06a77ecc841462396a8d5026c7327f4 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250901 01:52:21.734651 1151 tablet_bootstrap.cc:492] T 9d886fa63f8f4996b087997b542e8bb1 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:21.735471 1890 raft_consensus.cc:491] T e1d18e999b5e42b1873631e0f1445081 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:21.735587 1890 raft_consensus.cc:513] T e1d18e999b5e42b1873631e0f1445081 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.736122 1890 leader_election.cc:290] T e1d18e999b5e42b1873631e0f1445081 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:21.736923 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e1d18e999b5e42b1873631e0f1445081" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:21.737381 859 leader_election.cc:343] T e1d18e999b5e42b1873631e0f1445081 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:21.738917 1151 tablet_bootstrap.cc:654] T 9d886fa63f8f4996b087997b542e8bb1 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:21.739310 858 leader_election.cc:336] T e1d18e999b5e42b1873631e0f1445081 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:21.739441 858 leader_election.cc:304] T e1d18e999b5e42b1873631e0f1445081 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:21.740129 1890 raft_consensus.cc:2747] T e1d18e999b5e42b1873631e0f1445081 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:21.741153 1150 tablet_bootstrap.cc:492] T 20f7bfab18134e079e91c52c89468cc3 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:21.741122 1884 raft_consensus.cc:491] T 1286c0ab85704837a99b6e5a62184d77 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:21.741269 1150 ts_tablet_manager.cc:1397] T 20f7bfab18134e079e91c52c89468cc3 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.002s
I20250901 01:52:21.741277 1884 raft_consensus.cc:513] T 1286c0ab85704837a99b6e5a62184d77 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.741865 1884 leader_election.cc:290] T 1286c0ab85704837a99b6e5a62184d77 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:21.741998 1150 raft_consensus.cc:357] T 20f7bfab18134e079e91c52c89468cc3 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.742141 1150 raft_consensus.cc:383] T 20f7bfab18134e079e91c52c89468cc3 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.742434 1150 raft_consensus.cc:738] T 20f7bfab18134e079e91c52c89468cc3 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:21.742640 1150 consensus_queue.cc:260] T 20f7bfab18134e079e91c52c89468cc3 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.742728 919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1286c0ab85704837a99b6e5a62184d77" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:21.742918 919 raft_consensus.cc:2466] T 1286c0ab85704837a99b6e5a62184d77 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:21.743203 1150 ts_tablet_manager.cc:1428] T 20f7bfab18134e079e91c52c89468cc3 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250901 01:52:21.743460 1150 tablet_bootstrap.cc:492] T 2240302d9178445d8700e35f5ea274f7 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:21.745198 1150 tablet_bootstrap.cc:654] T 2240302d9178445d8700e35f5ea274f7 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:21.745805 992 leader_election.cc:304] T 1286c0ab85704837a99b6e5a62184d77 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
W20250901 01:52:21.746094 992 leader_election.cc:336] T 1286c0ab85704837a99b6e5a62184d77 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:21.746389 1872 raft_consensus.cc:2802] T 1286c0ab85704837a99b6e5a62184d77 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:21.746536 1872 raft_consensus.cc:491] T 1286c0ab85704837a99b6e5a62184d77 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:21.746659 1872 raft_consensus.cc:3058] T 1286c0ab85704837a99b6e5a62184d77 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:21.751653 1872 raft_consensus.cc:513] T 1286c0ab85704837a99b6e5a62184d77 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:21.754323 992 leader_election.cc:336] T 1286c0ab85704837a99b6e5a62184d77 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:21.754638 1150 tablet_bootstrap.cc:492] T 2240302d9178445d8700e35f5ea274f7 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:21.754784 1150 ts_tablet_manager.cc:1397] T 2240302d9178445d8700e35f5ea274f7 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.005s
I20250901 01:52:21.755081 919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1286c0ab85704837a99b6e5a62184d77" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:21.755249 919 raft_consensus.cc:3058] T 1286c0ab85704837a99b6e5a62184d77 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:21.755661 1150 raft_consensus.cc:357] T 2240302d9178445d8700e35f5ea274f7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.755848 1150 raft_consensus.cc:383] T 2240302d9178445d8700e35f5ea274f7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.755906 1150 raft_consensus.cc:738] T 2240302d9178445d8700e35f5ea274f7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:21.756171 1150 consensus_queue.cc:260] T 2240302d9178445d8700e35f5ea274f7 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.756762 1150 ts_tablet_manager.cc:1428] T 2240302d9178445d8700e35f5ea274f7 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250901 01:52:21.757014 1150 tablet_bootstrap.cc:492] T 15e4d972e8254cf899c3e0b71b6e2bd9 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:21.758041 919 raft_consensus.cc:2466] T 1286c0ab85704837a99b6e5a62184d77 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:21.758724 1150 tablet_bootstrap.cc:654] T 15e4d972e8254cf899c3e0b71b6e2bd9 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:21.759222 992 leader_election.cc:304] T 1286c0ab85704837a99b6e5a62184d77 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:21.759820 1881 raft_consensus.cc:2802] T 1286c0ab85704837a99b6e5a62184d77 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:21.759975 1881 raft_consensus.cc:695] T 1286c0ab85704837a99b6e5a62184d77 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:21.760247 1881 consensus_queue.cc:237] T 1286c0ab85704837a99b6e5a62184d77 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.763006 1872 leader_election.cc:290] T 1286c0ab85704837a99b6e5a62184d77 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:21.763335 1761 raft_consensus.cc:491] T f50f33de1b654663b9ef8a91e3436f11 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:21.763547 1761 raft_consensus.cc:513] T f50f33de1b654663b9ef8a91e3436f11 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:21.764263 1761 leader_election.cc:290] T f50f33de1b654663b9ef8a91e3436f11 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:21.766641 992 leader_election.cc:336] T f50f33de1b654663b9ef8a91e3436f11 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:21.767365 992 leader_election.cc:343] T f50f33de1b654663b9ef8a91e3436f11 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: f50f33de1b654663b9ef8a91e3436f11
I20250901 01:52:21.767511 992 leader_election.cc:304] T f50f33de1b654663b9ef8a91e3436f11 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:21.767627 1150 tablet_bootstrap.cc:492] T 15e4d972e8254cf899c3e0b71b6e2bd9 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:21.767765 1150 ts_tablet_manager.cc:1397] T 15e4d972e8254cf899c3e0b71b6e2bd9 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.008s
I20250901 01:52:21.768011 1761 raft_consensus.cc:2747] T f50f33de1b654663b9ef8a91e3436f11 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:21.768577 1150 raft_consensus.cc:357] T 15e4d972e8254cf899c3e0b71b6e2bd9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:21.768734 1150 raft_consensus.cc:383] T 15e4d972e8254cf899c3e0b71b6e2bd9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.768796 1150 raft_consensus.cc:738] T 15e4d972e8254cf899c3e0b71b6e2bd9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:21.768676 1897 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:21.768981 1150 consensus_queue.cc:260] T 15e4d972e8254cf899c3e0b71b6e2bd9 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:21.769212 1897 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3add7edb97a40ef9f7c6bc011d13227" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 43871 } } peers { permanent_uuid: "d2ae2cb6af0f4b3a8f47ab1cdd6567d3" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42375 } } peers { permanent_uuid: "7ce2288f784044eea010664e3def133f" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 44231 } }
I20250901 01:52:21.772132 1150 ts_tablet_manager.cc:1428] T 15e4d972e8254cf899c3e0b71b6e2bd9 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250901 01:52:21.772112 1897 leader_election.cc:290] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e3add7edb97a40ef9f7c6bc011d13227 (127.0.104.62:43871), d2ae2cb6af0f4b3a8f47ab1cdd6567d3 (127.0.104.61:42375)
I20250901 01:52:21.772599 534 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7ce2288f784044eea010664e3def133f" candidate_term: 4 candidate_status { last_received { term: 2 index: 633 } } ignore_live_leader: false dest_uuid: "d2ae2cb6af0f4b3a8f47ab1cdd6567d3" is_pre_election: true
I20250901 01:52:21.772818 534 raft_consensus.cc:2466] T 00000000000000000000000000000000 P d2ae2cb6af0f4b3a8f47ab1cdd6567d3 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ce2288f784044eea010664e3def133f in term 3.
I20250901 01:52:21.773064 1150 tablet_bootstrap.cc:492] T 64fb46b942ff4e4c8749986ecc3dfe65 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:21.773464 1474 leader_election.cc:304] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ce2288f784044eea010664e3def133f, d2ae2cb6af0f4b3a8f47ab1cdd6567d3; no voters:
I20250901 01:52:21.774397 1897 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250901 01:52:21.774572 1897 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:21.774722 1897 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f [term 3 FOLLOWER]: Advancing to term 4
I20250901 01:52:21.774757 1150 tablet_bootstrap.cc:654] T 64fb46b942ff4e4c8749986ecc3dfe65 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:21.777706 1897 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3add7edb97a40ef9f7c6bc011d13227" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 43871 } } peers { permanent_uuid: "d2ae2cb6af0f4b3a8f47ab1cdd6567d3" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42375 } } peers { permanent_uuid: "7ce2288f784044eea010664e3def133f" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 44231 } }
W20250901 01:52:21.778699 1472 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.62:43871: connect: Connection refused (error 111)
I20250901 01:52:21.781769 1897 leader_election.cc:290] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f [CANDIDATE]: Term 4 election: Requested vote from peers e3add7edb97a40ef9f7c6bc011d13227 (127.0.104.62:43871), d2ae2cb6af0f4b3a8f47ab1cdd6567d3 (127.0.104.61:42375)
I20250901 01:52:21.781896 534 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7ce2288f784044eea010664e3def133f" candidate_term: 4 candidate_status { last_received { term: 2 index: 633 } } ignore_live_leader: false dest_uuid: "d2ae2cb6af0f4b3a8f47ab1cdd6567d3"
I20250901 01:52:21.782161 534 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d2ae2cb6af0f4b3a8f47ab1cdd6567d3 [term 3 FOLLOWER]: Advancing to term 4
I20250901 01:52:21.783215 1151 tablet_bootstrap.cc:492] T 9d886fa63f8f4996b087997b542e8bb1 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:21.783499 1151 ts_tablet_manager.cc:1397] T 9d886fa63f8f4996b087997b542e8bb1 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.049s user 0.010s sys 0.012s
I20250901 01:52:21.784726 1151 raft_consensus.cc:357] T 9d886fa63f8f4996b087997b542e8bb1 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.785081 1151 raft_consensus.cc:383] T 9d886fa63f8f4996b087997b542e8bb1 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.785219 1151 raft_consensus.cc:738] T 9d886fa63f8f4996b087997b542e8bb1 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:21.785616 1151 consensus_queue.cc:260] T 9d886fa63f8f4996b087997b542e8bb1 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.786944 1151 ts_tablet_manager.cc:1428] T 9d886fa63f8f4996b087997b542e8bb1 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250901 01:52:21.787385 1151 tablet_bootstrap.cc:492] T a6445b29591548be9f9577db1ba0bfa5 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:21.788235 1890 raft_consensus.cc:491] T 822382a1781b4304bce29355ce83730e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:21.788381 1890 raft_consensus.cc:513] T 822382a1781b4304bce29355ce83730e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.789088 1890 leader_election.cc:290] T 822382a1781b4304bce29355ce83730e P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:21.789561 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "822382a1781b4304bce29355ce83730e" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:21.790226 1151 tablet_bootstrap.cc:654] T a6445b29591548be9f9577db1ba0bfa5 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:21.790398 859 leader_election.cc:343] T 822382a1781b4304bce29355ce83730e P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
W20250901 01:52:21.791152 858 leader_election.cc:336] T 822382a1781b4304bce29355ce83730e P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:21.791249 858 leader_election.cc:304] T 822382a1781b4304bce29355ce83730e P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:21.791610 1881 raft_consensus.cc:491] T 98cca86164db4252b1b49de88747822a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:21.791751 1881 raft_consensus.cc:513] T 98cca86164db4252b1b49de88747822a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.791858 1890 raft_consensus.cc:2747] T 822382a1781b4304bce29355ce83730e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:21.792310 1881 leader_election.cc:290] T 98cca86164db4252b1b49de88747822a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:21.794504 992 leader_election.cc:336] T 98cca86164db4252b1b49de88747822a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:21.794772 992 leader_election.cc:343] T 98cca86164db4252b1b49de88747822a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 98cca86164db4252b1b49de88747822a
I20250901 01:52:21.794914 992 leader_election.cc:304] T 98cca86164db4252b1b49de88747822a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:21.795258 1881 raft_consensus.cc:2747] T 98cca86164db4252b1b49de88747822a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:21.795703 1472 leader_election.cc:336] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer e3add7edb97a40ef9f7c6bc011d13227 (127.0.104.62:43871): Network error: Client connection negotiation failed: client connection to 127.0.104.62:43871: connect: Connection refused (error 111)
I20250901 01:52:21.792289 534 raft_consensus.cc:2466] T 00000000000000000000000000000000 P d2ae2cb6af0f4b3a8f47ab1cdd6567d3 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ce2288f784044eea010664e3def133f in term 4.
I20250901 01:52:21.797065 1474 leader_election.cc:304] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ce2288f784044eea010664e3def133f, d2ae2cb6af0f4b3a8f47ab1cdd6567d3; no voters:
W20250901 01:52:21.797281 1472 leader_election.cc:336] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer e3add7edb97a40ef9f7c6bc011d13227 (127.0.104.62:43871): Network error: Client connection negotiation failed: client connection to 127.0.104.62:43871: connect: Connection refused (error 111)
I20250901 01:52:21.797405 1897 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f [term 4 FOLLOWER]: Leader election won for term 4
I20250901 01:52:21.798800 1151 tablet_bootstrap.cc:492] T a6445b29591548be9f9577db1ba0bfa5 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:21.799013 1151 ts_tablet_manager.cc:1397] T a6445b29591548be9f9577db1ba0bfa5 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.004s
I20250901 01:52:21.799919 1151 raft_consensus.cc:357] T a6445b29591548be9f9577db1ba0bfa5 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:21.800221 1151 raft_consensus.cc:383] T a6445b29591548be9f9577db1ba0bfa5 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.800376 1151 raft_consensus.cc:738] T a6445b29591548be9f9577db1ba0bfa5 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:21.800737 1151 consensus_queue.cc:260] T a6445b29591548be9f9577db1ba0bfa5 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:21.801750 1890 raft_consensus.cc:491] T 2240302d9178445d8700e35f5ea274f7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:21.801869 1890 raft_consensus.cc:513] T 2240302d9178445d8700e35f5ea274f7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.802551 1890 leader_election.cc:290] T 2240302d9178445d8700e35f5ea274f7 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:21.803115 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2240302d9178445d8700e35f5ea274f7" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:21.803942 859 leader_election.cc:343] T 2240302d9178445d8700e35f5ea274f7 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
W20250901 01:52:21.804617 858 leader_election.cc:336] T 2240302d9178445d8700e35f5ea274f7 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:21.804715 858 leader_election.cc:304] T 2240302d9178445d8700e35f5ea274f7 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:21.805138 1890 raft_consensus.cc:2747] T 2240302d9178445d8700e35f5ea274f7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:21.801579 1151 ts_tablet_manager.cc:1428] T a6445b29591548be9f9577db1ba0bfa5 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250901 01:52:21.805985 1151 tablet_bootstrap.cc:492] T 40d74d7b5a3b4d1dae09eebd9e07a0c4 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:21.807937 1151 tablet_bootstrap.cc:654] T 40d74d7b5a3b4d1dae09eebd9e07a0c4 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:21.816812 1881 raft_consensus.cc:491] T 2ff914352c7c43de800f3d54ecf63072 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:21.816984 1881 raft_consensus.cc:513] T 2ff914352c7c43de800f3d54ecf63072 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.817591 1881 leader_election.cc:290] T 2ff914352c7c43de800f3d54ecf63072 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:21.818629 919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2ff914352c7c43de800f3d54ecf63072" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:21.818818 919 raft_consensus.cc:2466] T 2ff914352c7c43de800f3d54ecf63072 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:21.820021 992 leader_election.cc:304] T 2ff914352c7c43de800f3d54ecf63072 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:21.820374 1881 raft_consensus.cc:2802] T 2ff914352c7c43de800f3d54ecf63072 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:21.820546 1881 raft_consensus.cc:491] T 2ff914352c7c43de800f3d54ecf63072 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:21.820659 1881 raft_consensus.cc:3058] T 2ff914352c7c43de800f3d54ecf63072 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:21.824630 1761 raft_consensus.cc:491] T 5a8c21aee9f84216b6f90538a5bf4c17 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:21.824858 1761 raft_consensus.cc:513] T 5a8c21aee9f84216b6f90538a5bf4c17 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.825655 1761 leader_election.cc:290] T 5a8c21aee9f84216b6f90538a5bf4c17 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:21.825912 992 leader_election.cc:336] T 2ff914352c7c43de800f3d54ecf63072 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:21.829062 992 leader_election.cc:343] T 5a8c21aee9f84216b6f90538a5bf4c17 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 5a8c21aee9f84216b6f90538a5bf4c17
W20250901 01:52:21.829398 992 leader_election.cc:336] T 5a8c21aee9f84216b6f90538a5bf4c17 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:21.830091 992 leader_election.cc:304] T 5a8c21aee9f84216b6f90538a5bf4c17 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:21.830497 1761 raft_consensus.cc:2747] T 5a8c21aee9f84216b6f90538a5bf4c17 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:21.831147 1881 raft_consensus.cc:513] T 2ff914352c7c43de800f3d54ecf63072 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.831753 1881 leader_election.cc:290] T 2ff914352c7c43de800f3d54ecf63072 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:21.832046 1151 tablet_bootstrap.cc:492] T 40d74d7b5a3b4d1dae09eebd9e07a0c4 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:21.832213 1151 ts_tablet_manager.cc:1397] T 40d74d7b5a3b4d1dae09eebd9e07a0c4 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.026s user 0.012s sys 0.010s
I20250901 01:52:21.832824 919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2ff914352c7c43de800f3d54ecf63072" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:21.833029 919 raft_consensus.cc:3058] T 2ff914352c7c43de800f3d54ecf63072 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:21.835650 919 raft_consensus.cc:2466] T 2ff914352c7c43de800f3d54ecf63072 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:21.836606 1151 raft_consensus.cc:357] T 40d74d7b5a3b4d1dae09eebd9e07a0c4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.836836 1151 raft_consensus.cc:383] T 40d74d7b5a3b4d1dae09eebd9e07a0c4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.836931 1151 raft_consensus.cc:738] T 40d74d7b5a3b4d1dae09eebd9e07a0c4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:21.837203 1151 consensus_queue.cc:260] T 40d74d7b5a3b4d1dae09eebd9e07a0c4 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.838078 1151 ts_tablet_manager.cc:1428] T 40d74d7b5a3b4d1dae09eebd9e07a0c4 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20250901 01:52:21.838626 1151 tablet_bootstrap.cc:492] T 23af154a6f1c4e2e9352fb17c9024f10 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:21.840385 1151 tablet_bootstrap.cc:654] T 23af154a6f1c4e2e9352fb17c9024f10 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:21.844602 992 leader_election.cc:336] T 2ff914352c7c43de800f3d54ecf63072 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:21.845359 1897 raft_consensus.cc:695] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f [term 4 LEADER]: Becoming Leader. State: Replica: 7ce2288f784044eea010664e3def133f, State: Running, Role: LEADER
I20250901 01:52:21.846653 992 leader_election.cc:304] T 2ff914352c7c43de800f3d54ecf63072 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:21.852110 1881 raft_consensus.cc:2802] T 2ff914352c7c43de800f3d54ecf63072 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:21.852257 1881 raft_consensus.cc:695] T 2ff914352c7c43de800f3d54ecf63072 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:21.852583 1881 consensus_queue.cc:237] T 2ff914352c7c43de800f3d54ecf63072 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.847501 1897 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 632, Committed index: 632, Last appended: 2.633, Last appended by leader: 633, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3add7edb97a40ef9f7c6bc011d13227" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 43871 } } peers { permanent_uuid: "d2ae2cb6af0f4b3a8f47ab1cdd6567d3" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42375 } } peers { permanent_uuid: "7ce2288f784044eea010664e3def133f" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 44231 } }
I20250901 01:52:21.859231 1151 tablet_bootstrap.cc:492] T 23af154a6f1c4e2e9352fb17c9024f10 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:21.859380 1151 ts_tablet_manager.cc:1397] T 23af154a6f1c4e2e9352fb17c9024f10 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.021s user 0.011s sys 0.001s
I20250901 01:52:21.860262 1151 raft_consensus.cc:357] T 23af154a6f1c4e2e9352fb17c9024f10 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.860486 1151 raft_consensus.cc:383] T 23af154a6f1c4e2e9352fb17c9024f10 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.860585 1151 raft_consensus.cc:738] T 23af154a6f1c4e2e9352fb17c9024f10 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:21.860843 1151 consensus_queue.cc:260] T 23af154a6f1c4e2e9352fb17c9024f10 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.861385 1151 ts_tablet_manager.cc:1428] T 23af154a6f1c4e2e9352fb17c9024f10 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250901 01:52:21.862174 1151 tablet_bootstrap.cc:492] T ddbc15685fca4c708329420109ad0c57 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:21.863718 1150 tablet_bootstrap.cc:492] T 64fb46b942ff4e4c8749986ecc3dfe65 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:21.863849 1150 ts_tablet_manager.cc:1397] T 64fb46b942ff4e4c8749986ecc3dfe65 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.092s user 0.037s sys 0.044s
I20250901 01:52:21.864025 1151 tablet_bootstrap.cc:654] T ddbc15685fca4c708329420109ad0c57 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:21.863843 1904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ce2288f784044eea010664e3def133f. Latest consensus state: current_term: 4 leader_uuid: "7ce2288f784044eea010664e3def133f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3add7edb97a40ef9f7c6bc011d13227" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 43871 } } peers { permanent_uuid: "d2ae2cb6af0f4b3a8f47ab1cdd6567d3" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42375 } } peers { permanent_uuid: "7ce2288f784044eea010664e3def133f" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 44231 } } }
I20250901 01:52:21.864208 1904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f [sys.catalog]: This master's current role is: LEADER
I20250901 01:52:21.864686 1150 raft_consensus.cc:357] T 64fb46b942ff4e4c8749986ecc3dfe65 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.864871 1150 raft_consensus.cc:383] T 64fb46b942ff4e4c8749986ecc3dfe65 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.864928 1150 raft_consensus.cc:738] T 64fb46b942ff4e4c8749986ecc3dfe65 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:21.865111 1150 consensus_queue.cc:260] T 64fb46b942ff4e4c8749986ecc3dfe65 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.865619 1150 ts_tablet_manager.cc:1428] T 64fb46b942ff4e4c8749986ecc3dfe65 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250901 01:52:21.865878 1150 tablet_bootstrap.cc:492] T 7f64ebd080944190be2e64d02d70a8a7 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:21.867480 1150 tablet_bootstrap.cc:654] T 7f64ebd080944190be2e64d02d70a8a7 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:21.877146 1881 raft_consensus.cc:491] T 9d886fa63f8f4996b087997b542e8bb1 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:21.877413 1881 raft_consensus.cc:513] T 9d886fa63f8f4996b087997b542e8bb1 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.878207 1881 leader_election.cc:290] T 9d886fa63f8f4996b087997b542e8bb1 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:21.880452 992 leader_election.cc:336] T 9d886fa63f8f4996b087997b542e8bb1 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:21.880753 992 leader_election.cc:343] T 9d886fa63f8f4996b087997b542e8bb1 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 9d886fa63f8f4996b087997b542e8bb1
I20250901 01:52:21.880923 992 leader_election.cc:304] T 9d886fa63f8f4996b087997b542e8bb1 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:21.881385 1881 raft_consensus.cc:2747] T 9d886fa63f8f4996b087997b542e8bb1 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:21.887127 1881 raft_consensus.cc:491] T 66a48a8ebe154b2cac89c234eaa378ff P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:21.887355 1881 raft_consensus.cc:513] T 66a48a8ebe154b2cac89c234eaa378ff P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.888140 1881 leader_election.cc:290] T 66a48a8ebe154b2cac89c234eaa378ff P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:21.890614 992 leader_election.cc:343] T 66a48a8ebe154b2cac89c234eaa378ff P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 66a48a8ebe154b2cac89c234eaa378ff
I20250901 01:52:21.890841 1151 tablet_bootstrap.cc:492] T ddbc15685fca4c708329420109ad0c57 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:21.891042 1151 ts_tablet_manager.cc:1397] T ddbc15685fca4c708329420109ad0c57 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.029s user 0.009s sys 0.000s
W20250901 01:52:21.891202 992 leader_election.cc:336] T 66a48a8ebe154b2cac89c234eaa378ff P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:21.891369 992 leader_election.cc:304] T 66a48a8ebe154b2cac89c234eaa378ff P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:21.891892 1881 raft_consensus.cc:2747] T 66a48a8ebe154b2cac89c234eaa378ff P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:21.892056 1151 raft_consensus.cc:357] T ddbc15685fca4c708329420109ad0c57 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.892308 1151 raft_consensus.cc:383] T ddbc15685fca4c708329420109ad0c57 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.892431 1151 raft_consensus.cc:738] T ddbc15685fca4c708329420109ad0c57 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:21.892728 1151 consensus_queue.cc:260] T ddbc15685fca4c708329420109ad0c57 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.893196 1151 ts_tablet_manager.cc:1428] T ddbc15685fca4c708329420109ad0c57 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250901 01:52:21.893726 1151 tablet_bootstrap.cc:492] T ded33e0950b544509af11739353c620d P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:21.893929 1150 tablet_bootstrap.cc:492] T 7f64ebd080944190be2e64d02d70a8a7 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:21.894057 1150 ts_tablet_manager.cc:1397] T 7f64ebd080944190be2e64d02d70a8a7 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.028s user 0.016s sys 0.010s
I20250901 01:52:21.894872 1150 raft_consensus.cc:357] T 7f64ebd080944190be2e64d02d70a8a7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.895020 1150 raft_consensus.cc:383] T 7f64ebd080944190be2e64d02d70a8a7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.895071 1150 raft_consensus.cc:738] T 7f64ebd080944190be2e64d02d70a8a7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:21.895248 1150 consensus_queue.cc:260] T 7f64ebd080944190be2e64d02d70a8a7 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.895812 1151 tablet_bootstrap.cc:654] T ded33e0950b544509af11739353c620d P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:21.895982 1150 ts_tablet_manager.cc:1428] T 7f64ebd080944190be2e64d02d70a8a7 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250901 01:52:21.896284 1150 tablet_bootstrap.cc:492] T aa424ac6ef1441e6b1cc2867b1c48186 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:21.897323 1890 raft_consensus.cc:491] T 629db80a8ab444e582628578155638fd P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:21.897521 1890 raft_consensus.cc:513] T 629db80a8ab444e582628578155638fd P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.898195 1890 leader_election.cc:290] T 629db80a8ab444e582628578155638fd P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:21.898244 1150 tablet_bootstrap.cc:654] T aa424ac6ef1441e6b1cc2867b1c48186 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:21.899287 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "629db80a8ab444e582628578155638fd" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:21.899498 1056 raft_consensus.cc:2466] T 629db80a8ab444e582628578155638fd P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00c282f779c1477d920f24dc0dc16628 in term 0.
I20250901 01:52:21.900597 859 leader_election.cc:304] T 629db80a8ab444e582628578155638fd P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:21.904958 1150 tablet_bootstrap.cc:492] T aa424ac6ef1441e6b1cc2867b1c48186 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:21.905051 1150 ts_tablet_manager.cc:1397] T aa424ac6ef1441e6b1cc2867b1c48186 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20250901 01:52:21.905658 1150 raft_consensus.cc:357] T aa424ac6ef1441e6b1cc2867b1c48186 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.905809 1150 raft_consensus.cc:383] T aa424ac6ef1441e6b1cc2867b1c48186 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.905854 1150 raft_consensus.cc:738] T aa424ac6ef1441e6b1cc2867b1c48186 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:21.906039 1890 raft_consensus.cc:2802] T 629db80a8ab444e582628578155638fd P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:21.906131 1890 raft_consensus.cc:491] T 629db80a8ab444e582628578155638fd P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:21.906193 1890 raft_consensus.cc:3058] T 629db80a8ab444e582628578155638fd P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:21.906539 1151 tablet_bootstrap.cc:492] T ded33e0950b544509af11739353c620d P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:21.906680 1151 ts_tablet_manager.cc:1397] T ded33e0950b544509af11739353c620d P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
W20250901 01:52:21.907335 858 leader_election.cc:336] T 629db80a8ab444e582628578155638fd P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:21.907384 1151 raft_consensus.cc:357] T ded33e0950b544509af11739353c620d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.907536 1151 raft_consensus.cc:383] T ded33e0950b544509af11739353c620d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.907603 1151 raft_consensus.cc:738] T ded33e0950b544509af11739353c620d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:21.907775 1151 consensus_queue.cc:260] T ded33e0950b544509af11739353c620d P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.908298 1150 consensus_queue.cc:260] T aa424ac6ef1441e6b1cc2867b1c48186 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.908803 1150 ts_tablet_manager.cc:1428] T aa424ac6ef1441e6b1cc2867b1c48186 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250901 01:52:21.909057 1150 tablet_bootstrap.cc:492] T d0c3c363dda444519a17dca63ad9caa7 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:21.909439 1890 raft_consensus.cc:513] T 629db80a8ab444e582628578155638fd P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.909957 1151 ts_tablet_manager.cc:1428] T ded33e0950b544509af11739353c620d P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250901 01:52:21.910032 1890 leader_election.cc:290] T 629db80a8ab444e582628578155638fd P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:21.910809 1151 tablet_bootstrap.cc:492] T 831e42de95854541a49ae6b3e039d3e9 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:21.912844 1150 tablet_bootstrap.cc:654] T d0c3c363dda444519a17dca63ad9caa7 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:21.914458 1151 tablet_bootstrap.cc:654] T 831e42de95854541a49ae6b3e039d3e9 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:21.915639 858 leader_election.cc:336] T 629db80a8ab444e582628578155638fd P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:21.919564 1150 tablet_bootstrap.cc:492] T d0c3c363dda444519a17dca63ad9caa7 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:21.920154 1150 ts_tablet_manager.cc:1397] T d0c3c363dda444519a17dca63ad9caa7 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20250901 01:52:21.920794 1761 raft_consensus.cc:491] T 4a87e68850c4459eb2c8eafac89c6aa9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:21.920897 1151 tablet_bootstrap.cc:492] T 831e42de95854541a49ae6b3e039d3e9 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:21.921061 1151 ts_tablet_manager.cc:1397] T 831e42de95854541a49ae6b3e039d3e9 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20250901 01:52:21.920984 1761 raft_consensus.cc:513] T 4a87e68850c4459eb2c8eafac89c6aa9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.921243 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "629db80a8ab444e582628578155638fd" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58"
I20250901 01:52:21.921427 1056 raft_consensus.cc:3058] T 629db80a8ab444e582628578155638fd P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:21.921617 1761 leader_election.cc:290] T 4a87e68850c4459eb2c8eafac89c6aa9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:21.921092 1150 raft_consensus.cc:357] T d0c3c363dda444519a17dca63ad9caa7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:21.922122 1150 raft_consensus.cc:383] T d0c3c363dda444519a17dca63ad9caa7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.922224 1150 raft_consensus.cc:738] T d0c3c363dda444519a17dca63ad9caa7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:21.922435 1150 consensus_queue.cc:260] T d0c3c363dda444519a17dca63ad9caa7 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:21.922961 1150 ts_tablet_manager.cc:1428] T d0c3c363dda444519a17dca63ad9caa7 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
W20250901 01:52:21.923520 992 leader_election.cc:336] T 4a87e68850c4459eb2c8eafac89c6aa9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:21.924126 1150 tablet_bootstrap.cc:492] T 42c1156e8dd84c6fa1396b46de13eb6f P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:21.924345 1056 raft_consensus.cc:2466] T 629db80a8ab444e582628578155638fd P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c282f779c1477d920f24dc0dc16628 in term 1.
I20250901 01:52:21.927103 1150 tablet_bootstrap.cc:654] T 42c1156e8dd84c6fa1396b46de13eb6f P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:21.924341 1151 raft_consensus.cc:357] T 831e42de95854541a49ae6b3e039d3e9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:21.928838 992 leader_election.cc:343] T 4a87e68850c4459eb2c8eafac89c6aa9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 4a87e68850c4459eb2c8eafac89c6aa9
I20250901 01:52:21.931354 992 leader_election.cc:304] T 4a87e68850c4459eb2c8eafac89c6aa9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:21.931809 859 leader_election.cc:304] T 629db80a8ab444e582628578155638fd P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:21.931840 1761 raft_consensus.cc:2747] T 4a87e68850c4459eb2c8eafac89c6aa9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:21.932042 1151 raft_consensus.cc:383] T 831e42de95854541a49ae6b3e039d3e9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.932071 1890 raft_consensus.cc:2802] T 629db80a8ab444e582628578155638fd P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:21.932116 1151 raft_consensus.cc:738] T 831e42de95854541a49ae6b3e039d3e9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:21.932178 1890 raft_consensus.cc:695] T 629db80a8ab444e582628578155638fd P 00c282f779c1477d920f24dc0dc16628 [term 1 LEADER]: Becoming Leader. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Running, Role: LEADER
I20250901 01:52:21.932339 1151 consensus_queue.cc:260] T 831e42de95854541a49ae6b3e039d3e9 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.932477 1890 consensus_queue.cc:237] T 629db80a8ab444e582628578155638fd P 00c282f779c1477d920f24dc0dc16628 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.933808 1151 ts_tablet_manager.cc:1428] T 831e42de95854541a49ae6b3e039d3e9 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20250901 01:52:21.934930 1151 tablet_bootstrap.cc:492] T e25923b2daf9429683b2d44a75a56189 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:21.935922 1761 raft_consensus.cc:491] T b633b461dbf34e13bc6e1cc08e6d20c1 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:21.936100 1761 raft_consensus.cc:513] T b633b461dbf34e13bc6e1cc08e6d20c1 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:21.936645 1761 leader_election.cc:290] T b633b461dbf34e13bc6e1cc08e6d20c1 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:21.938040 992 leader_election.cc:343] T b633b461dbf34e13bc6e1cc08e6d20c1 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: b633b461dbf34e13bc6e1cc08e6d20c1
W20250901 01:52:21.938449 992 leader_election.cc:336] T b633b461dbf34e13bc6e1cc08e6d20c1 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:21.938561 992 leader_election.cc:304] T b633b461dbf34e13bc6e1cc08e6d20c1 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:21.938936 1761 raft_consensus.cc:2747] T b633b461dbf34e13bc6e1cc08e6d20c1 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:21.940014 1151 tablet_bootstrap.cc:654] T e25923b2daf9429683b2d44a75a56189 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:21.950644 1711 raft_consensus.cc:491] T aa424ac6ef1441e6b1cc2867b1c48186 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:21.950870 1711 raft_consensus.cc:513] T aa424ac6ef1441e6b1cc2867b1c48186 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.951728 1711 leader_election.cc:290] T aa424ac6ef1441e6b1cc2867b1c48186 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:21.953076 858 leader_election.cc:336] T aa424ac6ef1441e6b1cc2867b1c48186 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:21.954248 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa424ac6ef1441e6b1cc2867b1c48186" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:21.954785 859 leader_election.cc:343] T aa424ac6ef1441e6b1cc2867b1c48186 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:21.954963 859 leader_election.cc:304] T aa424ac6ef1441e6b1cc2867b1c48186 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:21.955541 1711 raft_consensus.cc:2747] T aa424ac6ef1441e6b1cc2867b1c48186 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:21.956411 1761 raft_consensus.cc:491] T abe27e749fa0485195e0064a49d5431e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:21.956620 1761 raft_consensus.cc:513] T abe27e749fa0485195e0064a49d5431e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.957401 1761 leader_election.cc:290] T abe27e749fa0485195e0064a49d5431e P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:21.958613 992 leader_election.cc:343] T abe27e749fa0485195e0064a49d5431e P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: abe27e749fa0485195e0064a49d5431e
W20250901 01:52:21.959048 992 leader_election.cc:336] T abe27e749fa0485195e0064a49d5431e P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:21.959184 992 leader_election.cc:304] T abe27e749fa0485195e0064a49d5431e P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:21.959610 1761 raft_consensus.cc:2747] T abe27e749fa0485195e0064a49d5431e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:21.963210 1150 tablet_bootstrap.cc:492] T 42c1156e8dd84c6fa1396b46de13eb6f P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:21.963229 1151 tablet_bootstrap.cc:492] T e25923b2daf9429683b2d44a75a56189 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:21.963335 1150 ts_tablet_manager.cc:1397] T 42c1156e8dd84c6fa1396b46de13eb6f P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.039s user 0.007s sys 0.022s
I20250901 01:52:21.963361 1151 ts_tablet_manager.cc:1397] T e25923b2daf9429683b2d44a75a56189 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.018s
I20250901 01:52:21.963976 1151 raft_consensus.cc:357] T e25923b2daf9429683b2d44a75a56189 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.964023 1150 raft_consensus.cc:357] T 42c1156e8dd84c6fa1396b46de13eb6f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:21.964105 1761 raft_consensus.cc:491] T 438cd35ff3f74c49b908d072560077db P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:21.964169 1150 raft_consensus.cc:383] T 42c1156e8dd84c6fa1396b46de13eb6f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.964172 1151 raft_consensus.cc:383] T e25923b2daf9429683b2d44a75a56189 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:21.964213 1150 raft_consensus.cc:738] T 42c1156e8dd84c6fa1396b46de13eb6f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:21.964295 1151 raft_consensus.cc:738] T e25923b2daf9429683b2d44a75a56189 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:21.964269 1761 raft_consensus.cc:513] T 438cd35ff3f74c49b908d072560077db P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.964447 1150 consensus_queue.cc:260] T 42c1156e8dd84c6fa1396b46de13eb6f P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:21.964534 1151 consensus_queue.cc:260] T e25923b2daf9429683b2d44a75a56189 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.964890 1761 leader_election.cc:290] T 438cd35ff3f74c49b908d072560077db P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:21.965584 1150 ts_tablet_manager.cc:1428] T 42c1156e8dd84c6fa1396b46de13eb6f P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250901 01:52:21.966214 1150 tablet_bootstrap.cc:492] T f4fd117c395e41c2ab679c684908c5b7 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
W20250901 01:52:21.969192 992 leader_election.cc:336] T 438cd35ff3f74c49b908d072560077db P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:21.972637 992 leader_election.cc:343] T 438cd35ff3f74c49b908d072560077db P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 438cd35ff3f74c49b908d072560077db
I20250901 01:52:21.972760 992 leader_election.cc:304] T 438cd35ff3f74c49b908d072560077db P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:21.973141 1761 raft_consensus.cc:2747] T 438cd35ff3f74c49b908d072560077db P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:21.974001 1150 tablet_bootstrap.cc:654] T f4fd117c395e41c2ab679c684908c5b7 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:21.975296 1151 ts_tablet_manager.cc:1428] T e25923b2daf9429683b2d44a75a56189 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20250901 01:52:21.975656 1151 tablet_bootstrap.cc:492] T c763d252598745a088344aa0eb5fa3d3 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:21.977476 1151 tablet_bootstrap.cc:654] T c763d252598745a088344aa0eb5fa3d3 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:21.982262 1890 raft_consensus.cc:491] T 616ff34e8d314e0aa0e11ce60be11789 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:21.982441 1890 raft_consensus.cc:513] T 616ff34e8d314e0aa0e11ce60be11789 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.983197 1890 leader_election.cc:290] T 616ff34e8d314e0aa0e11ce60be11789 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:21.984748 1881 raft_consensus.cc:491] T 3fda93767e494a4684978c8522b829dd P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:21.984755 1890 raft_consensus.cc:491] T 121803e79ab64ec0b948b448646b9d0a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:21.984898 1881 raft_consensus.cc:513] T 3fda93767e494a4684978c8522b829dd P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:21.984930 1890 raft_consensus.cc:513] T 121803e79ab64ec0b948b448646b9d0a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:21.985589 1881 leader_election.cc:290] T 3fda93767e494a4684978c8522b829dd P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:21.985672 1890 leader_election.cc:290] T 121803e79ab64ec0b948b448646b9d0a P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:21.986980 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "121803e79ab64ec0b948b448646b9d0a" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:21.987560 992 leader_election.cc:343] T 3fda93767e494a4684978c8522b829dd P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 3fda93767e494a4684978c8522b829dd
I20250901 01:52:21.988639 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "616ff34e8d314e0aa0e11ce60be11789" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:21.989161 859 leader_election.cc:343] T 121803e79ab64ec0b948b448646b9d0a P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
W20250901 01:52:21.990741 858 leader_election.cc:336] T 616ff34e8d314e0aa0e11ce60be11789 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:21.991953 858 leader_election.cc:336] T 121803e79ab64ec0b948b448646b9d0a P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:21.992066 858 leader_election.cc:304] T 121803e79ab64ec0b948b448646b9d0a P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:21.993836 1890 raft_consensus.cc:2747] T 121803e79ab64ec0b948b448646b9d0a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:22.004223 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskddMNdm/build/asan/bin/kudu
/tmp/dist-test-taskddMNdm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:43871
--webserver_interface=127.0.104.62
--webserver_port=34679
--builtin_ntp_servers=127.0.104.20:46333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:43871,127.0.104.61:42375,127.0.104.60:44231
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250901 01:52:22.004850 859 leader_election.cc:343] T 616ff34e8d314e0aa0e11ce60be11789 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:22.004989 859 leader_election.cc:304] T 616ff34e8d314e0aa0e11ce60be11789 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:22.005501 1890 raft_consensus.cc:2747] T 616ff34e8d314e0aa0e11ce60be11789 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:22.006654 992 leader_election.cc:336] T 3fda93767e494a4684978c8522b829dd P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:22.011194 1151 tablet_bootstrap.cc:492] T c763d252598745a088344aa0eb5fa3d3 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:22.011170 1890 raft_consensus.cc:491] T d77aadd7291647a0bae7c421f244c252 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:22.011390 1151 ts_tablet_manager.cc:1397] T c763d252598745a088344aa0eb5fa3d3 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.036s user 0.024s sys 0.010s
I20250901 01:52:22.011332 1890 raft_consensus.cc:513] T d77aadd7291647a0bae7c421f244c252 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.012104 1890 leader_election.cc:290] T d77aadd7291647a0bae7c421f244c252 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:22.012282 1151 raft_consensus.cc:357] T c763d252598745a088344aa0eb5fa3d3 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.012517 1151 raft_consensus.cc:383] T c763d252598745a088344aa0eb5fa3d3 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:22.012611 1151 raft_consensus.cc:738] T c763d252598745a088344aa0eb5fa3d3 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:22.012890 1151 consensus_queue.cc:260] T c763d252598745a088344aa0eb5fa3d3 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.013513 1151 ts_tablet_manager.cc:1428] T c763d252598745a088344aa0eb5fa3d3 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250901 01:52:22.013828 1151 tablet_bootstrap.cc:492] T 5d794471dbed4477b9d52fd411fa4d10 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:22.015628 1151 tablet_bootstrap.cc:654] T 5d794471dbed4477b9d52fd411fa4d10 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:22.006861 992 leader_election.cc:304] T 3fda93767e494a4684978c8522b829dd P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:22.016449 1881 raft_consensus.cc:2747] T 3fda93767e494a4684978c8522b829dd P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:22.018697 858 leader_election.cc:336] T d77aadd7291647a0bae7c421f244c252 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:22.027374 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d77aadd7291647a0bae7c421f244c252" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:22.028893 859 leader_election.cc:343] T d77aadd7291647a0bae7c421f244c252 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:22.029055 859 leader_election.cc:304] T d77aadd7291647a0bae7c421f244c252 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:22.029515 1890 raft_consensus.cc:2747] T d77aadd7291647a0bae7c421f244c252 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:22.037223 1151 tablet_bootstrap.cc:492] T 5d794471dbed4477b9d52fd411fa4d10 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:22.037400 1151 ts_tablet_manager.cc:1397] T 5d794471dbed4477b9d52fd411fa4d10 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.010s
I20250901 01:52:22.038038 1150 tablet_bootstrap.cc:492] T f4fd117c395e41c2ab679c684908c5b7 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:22.038241 1150 ts_tablet_manager.cc:1397] T f4fd117c395e41c2ab679c684908c5b7 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.072s user 0.020s sys 0.034s
I20250901 01:52:22.038379 1151 raft_consensus.cc:357] T 5d794471dbed4477b9d52fd411fa4d10 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.038591 1151 raft_consensus.cc:383] T 5d794471dbed4477b9d52fd411fa4d10 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:22.038698 1151 raft_consensus.cc:738] T 5d794471dbed4477b9d52fd411fa4d10 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:22.038949 1151 consensus_queue.cc:260] T 5d794471dbed4477b9d52fd411fa4d10 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.039300 1150 raft_consensus.cc:357] T f4fd117c395e41c2ab679c684908c5b7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.039525 1150 raft_consensus.cc:383] T f4fd117c395e41c2ab679c684908c5b7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:22.039626 1150 raft_consensus.cc:738] T f4fd117c395e41c2ab679c684908c5b7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:22.039896 1150 consensus_queue.cc:260] T f4fd117c395e41c2ab679c684908c5b7 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.040439 1150 ts_tablet_manager.cc:1428] T f4fd117c395e41c2ab679c684908c5b7 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250901 01:52:22.043886 1150 tablet_bootstrap.cc:492] T e9b179dba2cc49428ba8090c962d6136 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:22.045673 1150 tablet_bootstrap.cc:654] T e9b179dba2cc49428ba8090c962d6136 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:22.048877 1881 raft_consensus.cc:491] T 606fac5c6efe48d1877104b01c23ba5b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:22.050043 1151 ts_tablet_manager.cc:1428] T 5d794471dbed4477b9d52fd411fa4d10 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250901 01:52:22.050243 1881 raft_consensus.cc:513] T 606fac5c6efe48d1877104b01c23ba5b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.050961 1881 leader_election.cc:290] T 606fac5c6efe48d1877104b01c23ba5b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:22.052776 1151 tablet_bootstrap.cc:492] T f8a5886ddb644a86b76df44aab309057 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:22.053587 1881 raft_consensus.cc:491] T f62d96c37cca4143a4b098b7c69979e6 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:22.053740 1881 raft_consensus.cc:513] T f62d96c37cca4143a4b098b7c69979e6 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.054350 1881 leader_election.cc:290] T f62d96c37cca4143a4b098b7c69979e6 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:22.054773 1151 tablet_bootstrap.cc:654] T f8a5886ddb644a86b76df44aab309057 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:22.055266 919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "606fac5c6efe48d1877104b01c23ba5b" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:22.055473 919 raft_consensus.cc:2466] T 606fac5c6efe48d1877104b01c23ba5b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:22.055668 1890 raft_consensus.cc:491] T 49aaaec611784e6bac131efd58dbca51 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:22.055773 1890 raft_consensus.cc:513] T 49aaaec611784e6bac131efd58dbca51 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.056344 1890 leader_election.cc:290] T 49aaaec611784e6bac131efd58dbca51 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:22.056911 992 leader_election.cc:336] T 606fac5c6efe48d1877104b01c23ba5b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:22.057067 992 leader_election.cc:336] T f62d96c37cca4143a4b098b7c69979e6 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:22.057250 992 leader_election.cc:304] T 606fac5c6efe48d1877104b01c23ba5b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:22.057221 919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f62d96c37cca4143a4b098b7c69979e6" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:22.057395 919 raft_consensus.cc:2466] T f62d96c37cca4143a4b098b7c69979e6 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:22.057751 1881 raft_consensus.cc:2802] T 606fac5c6efe48d1877104b01c23ba5b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:22.057859 1881 raft_consensus.cc:491] T 606fac5c6efe48d1877104b01c23ba5b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:22.057826 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "49aaaec611784e6bac131efd58dbca51" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:22.057934 1881 raft_consensus.cc:3058] T 606fac5c6efe48d1877104b01c23ba5b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:22.058444 1890 raft_consensus.cc:491] T 72ba689f9230411995dc96e2b45601df P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:22.058604 1890 raft_consensus.cc:513] T 72ba689f9230411995dc96e2b45601df P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.059173 1890 leader_election.cc:290] T 72ba689f9230411995dc96e2b45601df P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:22.060570 858 leader_election.cc:336] T 49aaaec611784e6bac131efd58dbca51 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:22.060668 1881 raft_consensus.cc:513] T 606fac5c6efe48d1877104b01c23ba5b P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.061180 1881 leader_election.cc:290] T 606fac5c6efe48d1877104b01c23ba5b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:22.061532 859 leader_election.cc:343] T 49aaaec611784e6bac131efd58dbca51 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:22.061656 859 leader_election.cc:304] T 49aaaec611784e6bac131efd58dbca51 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:22.062383 992 leader_election.cc:304] T f62d96c37cca4143a4b098b7c69979e6 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:22.062361 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "72ba689f9230411995dc96e2b45601df" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:22.062562 919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "606fac5c6efe48d1877104b01c23ba5b" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:22.062708 919 raft_consensus.cc:3058] T 606fac5c6efe48d1877104b01c23ba5b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:22.065488 919 raft_consensus.cc:2466] T 606fac5c6efe48d1877104b01c23ba5b P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
W20250901 01:52:22.065706 859 leader_election.cc:343] T 72ba689f9230411995dc96e2b45601df P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:22.065946 1890 raft_consensus.cc:2747] T 49aaaec611784e6bac131efd58dbca51 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:22.065955 1881 raft_consensus.cc:2802] T f62d96c37cca4143a4b098b7c69979e6 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:22.066038 1881 raft_consensus.cc:491] T f62d96c37cca4143a4b098b7c69979e6 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:22.066123 1881 raft_consensus.cc:3058] T f62d96c37cca4143a4b098b7c69979e6 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
W20250901 01:52:22.066179 858 leader_election.cc:336] T 72ba689f9230411995dc96e2b45601df P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:22.066262 858 leader_election.cc:304] T 72ba689f9230411995dc96e2b45601df P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:22.066666 1890 raft_consensus.cc:2747] T 72ba689f9230411995dc96e2b45601df P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:22.068738 1881 raft_consensus.cc:513] T f62d96c37cca4143a4b098b7c69979e6 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.069283 1881 leader_election.cc:290] T f62d96c37cca4143a4b098b7c69979e6 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:22.069723 992 leader_election.cc:304] T 606fac5c6efe48d1877104b01c23ba5b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:22.070017 919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f62d96c37cca4143a4b098b7c69979e6" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:22.070196 919 raft_consensus.cc:3058] T f62d96c37cca4143a4b098b7c69979e6 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
W20250901 01:52:22.070202 992 leader_election.cc:336] T 606fac5c6efe48d1877104b01c23ba5b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:22.070928 992 leader_election.cc:336] T f62d96c37cca4143a4b098b7c69979e6 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:22.071902 1881 raft_consensus.cc:2802] T 606fac5c6efe48d1877104b01c23ba5b P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:22.072922 1881 raft_consensus.cc:695] T 606fac5c6efe48d1877104b01c23ba5b P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:22.073103 919 raft_consensus.cc:2466] T f62d96c37cca4143a4b098b7c69979e6 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:22.073205 1881 consensus_queue.cc:237] T 606fac5c6efe48d1877104b01c23ba5b P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.074191 992 leader_election.cc:304] T f62d96c37cca4143a4b098b7c69979e6 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:22.074453 1761 raft_consensus.cc:2802] T f62d96c37cca4143a4b098b7c69979e6 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:22.074613 1761 raft_consensus.cc:695] T f62d96c37cca4143a4b098b7c69979e6 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:22.074842 1761 consensus_queue.cc:237] T f62d96c37cca4143a4b098b7c69979e6 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.077200 1150 tablet_bootstrap.cc:492] T e9b179dba2cc49428ba8090c962d6136 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:22.077356 1150 ts_tablet_manager.cc:1397] T e9b179dba2cc49428ba8090c962d6136 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.034s user 0.007s sys 0.000s
I20250901 01:52:22.078095 1150 raft_consensus.cc:357] T e9b179dba2cc49428ba8090c962d6136 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.078279 1150 raft_consensus.cc:383] T e9b179dba2cc49428ba8090c962d6136 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:22.078358 1150 raft_consensus.cc:738] T e9b179dba2cc49428ba8090c962d6136 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:22.078596 1150 consensus_queue.cc:260] T e9b179dba2cc49428ba8090c962d6136 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.081647 1151 tablet_bootstrap.cc:492] T f8a5886ddb644a86b76df44aab309057 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:22.081830 1151 ts_tablet_manager.cc:1397] T f8a5886ddb644a86b76df44aab309057 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.029s user 0.001s sys 0.005s
I20250901 01:52:22.082569 1881 raft_consensus.cc:491] T ddbc15685fca4c708329420109ad0c57 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:22.082706 1881 raft_consensus.cc:513] T ddbc15685fca4c708329420109ad0c57 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.083206 1881 leader_election.cc:290] T ddbc15685fca4c708329420109ad0c57 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:22.084435 1890 raft_consensus.cc:491] T 3c1fd0820d8a4995abe027f685da7f7b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:22.084563 1890 raft_consensus.cc:513] T 3c1fd0820d8a4995abe027f685da7f7b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:22.084710 992 leader_election.cc:336] T ddbc15685fca4c708329420109ad0c57 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:22.085024 1890 leader_election.cc:290] T 3c1fd0820d8a4995abe027f685da7f7b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:22.085198 1151 raft_consensus.cc:357] T f8a5886ddb644a86b76df44aab309057 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.085403 1151 raft_consensus.cc:383] T f8a5886ddb644a86b76df44aab309057 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:22.085484 1151 raft_consensus.cc:738] T f8a5886ddb644a86b76df44aab309057 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:22.085703 1151 consensus_queue.cc:260] T f8a5886ddb644a86b76df44aab309057 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.086187 1151 ts_tablet_manager.cc:1428] T f8a5886ddb644a86b76df44aab309057 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250901 01:52:22.086215 1890 raft_consensus.cc:491] T 92f0ad59fbbb435ea872c669a52f854a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:22.086314 1890 raft_consensus.cc:513] T 92f0ad59fbbb435ea872c669a52f854a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.086390 1711 raft_consensus.cc:491] T 3c12d6a70ac24dce82e851782e6fe5aa P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:22.086509 1151 tablet_bootstrap.cc:492] T 670d90a37b934579b353bb4e4820127b P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:22.086488 1711 raft_consensus.cc:513] T 3c12d6a70ac24dce82e851782e6fe5aa P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.086812 1890 leader_election.cc:290] T 92f0ad59fbbb435ea872c669a52f854a P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:22.086901 1711 leader_election.cc:290] T 3c12d6a70ac24dce82e851782e6fe5aa P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:22.086997 919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ddbc15685fca4c708329420109ad0c57" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:22.087136 919 raft_consensus.cc:2466] T ddbc15685fca4c708329420109ad0c57 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:22.087469 1761 raft_consensus.cc:491] T 82d163681fda47f38d79863bd28f576d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:22.087600 1761 raft_consensus.cc:513] T 82d163681fda47f38d79863bd28f576d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.088116 1761 leader_election.cc:290] T 82d163681fda47f38d79863bd28f576d P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:22.088168 1151 tablet_bootstrap.cc:654] T 670d90a37b934579b353bb4e4820127b P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:22.088277 858 leader_election.cc:336] T 3c1fd0820d8a4995abe027f685da7f7b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:22.089253 858 leader_election.cc:336] T 92f0ad59fbbb435ea872c669a52f854a P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:22.089371 858 leader_election.cc:336] T 3c12d6a70ac24dce82e851782e6fe5aa P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:22.090441 1150 ts_tablet_manager.cc:1428] T e9b179dba2cc49428ba8090c962d6136 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.013s user 0.001s sys 0.000s
I20250901 01:52:22.090749 1150 tablet_bootstrap.cc:492] T 4855448edb4b40c184838aa7f81c112b P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:22.092344 1150 tablet_bootstrap.cc:654] T 4855448edb4b40c184838aa7f81c112b P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:22.099345 1150 tablet_bootstrap.cc:492] T 4855448edb4b40c184838aa7f81c112b P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:22.099392 1151 tablet_bootstrap.cc:492] T 670d90a37b934579b353bb4e4820127b P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:22.099511 1150 ts_tablet_manager.cc:1397] T 4855448edb4b40c184838aa7f81c112b P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.001s
I20250901 01:52:22.099926 1151 ts_tablet_manager.cc:1397] T 670d90a37b934579b353bb4e4820127b P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.003s
I20250901 01:52:22.100575 1150 raft_consensus.cc:357] T 4855448edb4b40c184838aa7f81c112b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:22.100745 1150 raft_consensus.cc:383] T 4855448edb4b40c184838aa7f81c112b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:22.100811 1150 raft_consensus.cc:738] T 4855448edb4b40c184838aa7f81c112b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:22.100996 1151 raft_consensus.cc:357] T 670d90a37b934579b353bb4e4820127b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.101054 1150 consensus_queue.cc:260] T 4855448edb4b40c184838aa7f81c112b P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:22.102530 1150 ts_tablet_manager.cc:1428] T 4855448edb4b40c184838aa7f81c112b P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250901 01:52:22.102588 1761 raft_consensus.cc:491] T ded33e0950b544509af11739353c620d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:22.102783 1761 raft_consensus.cc:513] T ded33e0950b544509af11739353c620d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.103291 1761 leader_election.cc:290] T ded33e0950b544509af11739353c620d P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:22.103611 1150 tablet_bootstrap.cc:492] T 25ce0fe8c9544f828694184d19d13df9 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:22.103608 992 leader_election.cc:304] T ddbc15685fca4c708329420109ad0c57 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:22.103859 1761 raft_consensus.cc:2802] T ddbc15685fca4c708329420109ad0c57 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:22.103931 1761 raft_consensus.cc:491] T ddbc15685fca4c708329420109ad0c57 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:22.104010 1761 raft_consensus.cc:3058] T ddbc15685fca4c708329420109ad0c57 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:22.105298 1150 tablet_bootstrap.cc:654] T 25ce0fe8c9544f828694184d19d13df9 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:22.106802 1761 raft_consensus.cc:513] T ddbc15685fca4c708329420109ad0c57 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.107509 1761 leader_election.cc:290] T ddbc15685fca4c708329420109ad0c57 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:22.107955 992 leader_election.cc:343] T 82d163681fda47f38d79863bd28f576d P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 82d163681fda47f38d79863bd28f576d
I20250901 01:52:22.108750 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92f0ad59fbbb435ea872c669a52f854a" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:22.109328 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3c12d6a70ac24dce82e851782e6fe5aa" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:22.111060 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3c1fd0820d8a4995abe027f685da7f7b" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:22.111344 859 leader_election.cc:343] T 92f0ad59fbbb435ea872c669a52f854a P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:22.111464 859 leader_election.cc:304] T 92f0ad59fbbb435ea872c669a52f854a P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:22.111865 859 leader_election.cc:343] T 3c12d6a70ac24dce82e851782e6fe5aa P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:22.111903 917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ddbc15685fca4c708329420109ad0c57" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:22.112073 917 raft_consensus.cc:3058] T ddbc15685fca4c708329420109ad0c57 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:22.112107 919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ded33e0950b544509af11739353c620d" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:22.112219 919 raft_consensus.cc:2466] T ded33e0950b544509af11739353c620d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:22.111946 859 leader_election.cc:304] T 3c12d6a70ac24dce82e851782e6fe5aa P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:22.112653 1890 raft_consensus.cc:2747] T 92f0ad59fbbb435ea872c669a52f854a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:22.112773 1890 raft_consensus.cc:2747] T 3c12d6a70ac24dce82e851782e6fe5aa P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:22.115243 917 raft_consensus.cc:2466] T ddbc15685fca4c708329420109ad0c57 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:22.118721 1761 raft_consensus.cc:491] T c763d252598745a088344aa0eb5fa3d3 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:22.118883 1761 raft_consensus.cc:513] T c763d252598745a088344aa0eb5fa3d3 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.119333 1711 raft_consensus.cc:491] T fae2779fac164b9f8052c3666db803b8 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:22.119484 1711 raft_consensus.cc:513] T fae2779fac164b9f8052c3666db803b8 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.119647 1761 leader_election.cc:290] T c763d252598745a088344aa0eb5fa3d3 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:22.119925 992 leader_election.cc:336] T 82d163681fda47f38d79863bd28f576d P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:22.120029 992 leader_election.cc:304] T 82d163681fda47f38d79863bd28f576d P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:22.120216 1711 leader_election.cc:290] T fae2779fac164b9f8052c3666db803b8 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:22.120440 1881 raft_consensus.cc:2747] T 82d163681fda47f38d79863bd28f576d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:22.121650 858 leader_election.cc:336] T fae2779fac164b9f8052c3666db803b8 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:22.122602 992 leader_election.cc:304] T ded33e0950b544509af11739353c620d P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:22.122867 992 leader_election.cc:304] T ddbc15685fca4c708329420109ad0c57 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:22.123087 1761 raft_consensus.cc:2802] T ded33e0950b544509af11739353c620d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:22.124547 1873 raft_consensus.cc:2802] T ddbc15685fca4c708329420109ad0c57 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:22.124745 1873 raft_consensus.cc:695] T ddbc15685fca4c708329420109ad0c57 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:22.125123 1873 consensus_queue.cc:237] T ddbc15685fca4c708329420109ad0c57 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:22.125609 859 leader_election.cc:343] T 3c1fd0820d8a4995abe027f685da7f7b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:22.125798 859 leader_election.cc:304] T 3c1fd0820d8a4995abe027f685da7f7b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:22.127139 1711 raft_consensus.cc:2747] T 3c1fd0820d8a4995abe027f685da7f7b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:22.129518 992 leader_election.cc:336] T ddbc15685fca4c708329420109ad0c57 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:22.129741 992 leader_election.cc:336] T ded33e0950b544509af11739353c620d P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:22.130347 917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c763d252598745a088344aa0eb5fa3d3" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:22.131439 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fae2779fac164b9f8052c3666db803b8" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:22.131843 917 raft_consensus.cc:2466] T c763d252598745a088344aa0eb5fa3d3 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
W20250901 01:52:22.135533 992 leader_election.cc:336] T c763d252598745a088344aa0eb5fa3d3 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:22.139251 859 leader_election.cc:343] T fae2779fac164b9f8052c3666db803b8 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:22.140079 859 leader_election.cc:304] T fae2779fac164b9f8052c3666db803b8 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:22.140509 1151 raft_consensus.cc:383] T 670d90a37b934579b353bb4e4820127b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:22.140591 1151 raft_consensus.cc:738] T 670d90a37b934579b353bb4e4820127b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:22.140713 1711 raft_consensus.cc:2747] T fae2779fac164b9f8052c3666db803b8 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:22.140811 1151 consensus_queue.cc:260] T 670d90a37b934579b353bb4e4820127b P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.141525 1151 ts_tablet_manager.cc:1428] T 670d90a37b934579b353bb4e4820127b P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.041s user 0.002s sys 0.000s
I20250901 01:52:22.141826 1151 tablet_bootstrap.cc:492] T 92f0ad59fbbb435ea872c669a52f854a P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:22.143615 1151 tablet_bootstrap.cc:654] T 92f0ad59fbbb435ea872c669a52f854a P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:22.144160 1711 raft_consensus.cc:491] T 9127f9c5239e445ba86012951b7c75d9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:22.144301 1711 raft_consensus.cc:513] T 9127f9c5239e445ba86012951b7c75d9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.145071 1711 leader_election.cc:290] T 9127f9c5239e445ba86012951b7c75d9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:22.146876 992 leader_election.cc:304] T c763d252598745a088344aa0eb5fa3d3 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
W20250901 01:52:22.147200 858 leader_election.cc:336] T 9127f9c5239e445ba86012951b7c75d9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:22.147727 1873 raft_consensus.cc:2802] T c763d252598745a088344aa0eb5fa3d3 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:22.147818 1873 raft_consensus.cc:491] T c763d252598745a088344aa0eb5fa3d3 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:22.147890 1873 raft_consensus.cc:3058] T c763d252598745a088344aa0eb5fa3d3 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:22.151117 1873 raft_consensus.cc:513] T c763d252598745a088344aa0eb5fa3d3 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.151825 1873 leader_election.cc:290] T c763d252598745a088344aa0eb5fa3d3 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:22.158891 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9127f9c5239e445ba86012951b7c75d9" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:22.159377 917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c763d252598745a088344aa0eb5fa3d3" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:22.159665 917 raft_consensus.cc:3058] T c763d252598745a088344aa0eb5fa3d3 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:22.163415 917 raft_consensus.cc:2466] T c763d252598745a088344aa0eb5fa3d3 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:22.165346 1873 raft_consensus.cc:491] T f06a77ecc841462396a8d5026c7327f4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250901 01:52:22.165572 992 leader_election.cc:336] T c763d252598745a088344aa0eb5fa3d3 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:22.165545 1873 raft_consensus.cc:513] T f06a77ecc841462396a8d5026c7327f4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.166945 1873 leader_election.cc:290] T f06a77ecc841462396a8d5026c7327f4 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:22.169867 992 leader_election.cc:336] T f06a77ecc841462396a8d5026c7327f4 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:22.165083 859 leader_election.cc:343] T 9127f9c5239e445ba86012951b7c75d9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:22.169354 919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f06a77ecc841462396a8d5026c7327f4" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:22.170349 859 leader_election.cc:304] T 9127f9c5239e445ba86012951b7c75d9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:22.170405 919 raft_consensus.cc:2466] T f06a77ecc841462396a8d5026c7327f4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:22.170967 992 leader_election.cc:304] T f06a77ecc841462396a8d5026c7327f4 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:22.171097 1711 raft_consensus.cc:2747] T 9127f9c5239e445ba86012951b7c75d9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:22.171429 1873 raft_consensus.cc:2802] T f06a77ecc841462396a8d5026c7327f4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:22.171564 1873 raft_consensus.cc:491] T f06a77ecc841462396a8d5026c7327f4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:22.171720 1873 raft_consensus.cc:3058] T f06a77ecc841462396a8d5026c7327f4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:22.173678 992 leader_election.cc:304] T c763d252598745a088344aa0eb5fa3d3 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:22.173969 1881 raft_consensus.cc:2802] T c763d252598745a088344aa0eb5fa3d3 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:22.174079 1881 raft_consensus.cc:695] T c763d252598745a088344aa0eb5fa3d3 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:22.174314 1881 consensus_queue.cc:237] T c763d252598745a088344aa0eb5fa3d3 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.179157 1873 raft_consensus.cc:513] T f06a77ecc841462396a8d5026c7327f4 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.180080 1873 leader_election.cc:290] T f06a77ecc841462396a8d5026c7327f4 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:22.181157 992 consensus_peers.cc:489] T 061b760d8e6647c6ab0b5d538840a230 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250901 01:52:22.181485 1151 tablet_bootstrap.cc:492] T 92f0ad59fbbb435ea872c669a52f854a P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:22.181602 1151 ts_tablet_manager.cc:1397] T 92f0ad59fbbb435ea872c669a52f854a P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.040s user 0.018s sys 0.008s
I20250901 01:52:22.181876 917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f06a77ecc841462396a8d5026c7327f4" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:22.182087 917 raft_consensus.cc:3058] T f06a77ecc841462396a8d5026c7327f4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:22.182329 1151 raft_consensus.cc:357] T 92f0ad59fbbb435ea872c669a52f854a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.182473 1151 raft_consensus.cc:383] T 92f0ad59fbbb435ea872c669a52f854a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:22.182515 1151 raft_consensus.cc:738] T 92f0ad59fbbb435ea872c669a52f854a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:22.182657 1151 consensus_queue.cc:260] T 92f0ad59fbbb435ea872c669a52f854a P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.183185 1151 ts_tablet_manager.cc:1428] T 92f0ad59fbbb435ea872c669a52f854a P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250901 01:52:22.183624 1151 tablet_bootstrap.cc:492] T 75e7a90815e449c39f386901a504143c P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:22.183928 1711 raft_consensus.cc:491] T eac50cbae691416e903cb2cbc907f96d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:22.184048 1711 raft_consensus.cc:513] T eac50cbae691416e903cb2cbc907f96d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.184698 1711 leader_election.cc:290] T eac50cbae691416e903cb2cbc907f96d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:22.185247 1151 tablet_bootstrap.cc:654] T 75e7a90815e449c39f386901a504143c P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:22.186899 858 leader_election.cc:336] T eac50cbae691416e903cb2cbc907f96d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:22.188506 917 raft_consensus.cc:2466] T f06a77ecc841462396a8d5026c7327f4 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:22.190376 992 leader_election.cc:304] T f06a77ecc841462396a8d5026c7327f4 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:22.190723 1881 raft_consensus.cc:2802] T f06a77ecc841462396a8d5026c7327f4 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:22.190846 1881 raft_consensus.cc:695] T f06a77ecc841462396a8d5026c7327f4 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:22.191152 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eac50cbae691416e903cb2cbc907f96d" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:22.191093 1881 consensus_queue.cc:237] T f06a77ecc841462396a8d5026c7327f4 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:22.191483 992 leader_election.cc:336] T f06a77ecc841462396a8d5026c7327f4 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:22.191949 859 leader_election.cc:343] T eac50cbae691416e903cb2cbc907f96d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:22.192071 859 leader_election.cc:304] T eac50cbae691416e903cb2cbc907f96d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:22.192646 1711 raft_consensus.cc:2747] T eac50cbae691416e903cb2cbc907f96d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:22.201195 1150 tablet_bootstrap.cc:492] T 25ce0fe8c9544f828694184d19d13df9 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:22.201373 1150 ts_tablet_manager.cc:1397] T 25ce0fe8c9544f828694184d19d13df9 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.098s user 0.015s sys 0.030s
I20250901 01:52:22.202131 1150 raft_consensus.cc:357] T 25ce0fe8c9544f828694184d19d13df9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:22.202311 1150 raft_consensus.cc:383] T 25ce0fe8c9544f828694184d19d13df9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:22.202395 1150 raft_consensus.cc:738] T 25ce0fe8c9544f828694184d19d13df9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:22.202630 1150 consensus_queue.cc:260] T 25ce0fe8c9544f828694184d19d13df9 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:22.203147 1150 ts_tablet_manager.cc:1428] T 25ce0fe8c9544f828694184d19d13df9 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250901 01:52:22.205120 1151 tablet_bootstrap.cc:492] T 75e7a90815e449c39f386901a504143c P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:22.205248 1151 ts_tablet_manager.cc:1397] T 75e7a90815e449c39f386901a504143c P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.007s
I20250901 01:52:22.206027 1151 raft_consensus.cc:357] T 75e7a90815e449c39f386901a504143c P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.206182 1151 raft_consensus.cc:383] T 75e7a90815e449c39f386901a504143c P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:22.206244 1151 raft_consensus.cc:738] T 75e7a90815e449c39f386901a504143c P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:22.206467 1151 consensus_queue.cc:260] T 75e7a90815e449c39f386901a504143c P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.207042 1151 ts_tablet_manager.cc:1428] T 75e7a90815e449c39f386901a504143c P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250901 01:52:22.207495 1151 tablet_bootstrap.cc:492] T b2e8b2cd10154de08ddd30c214592a4a P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:22.210767 1150 tablet_bootstrap.cc:492] T 46bfbdff3f904715a8c24907b20fcab0 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:22.212539 1150 tablet_bootstrap.cc:654] T 46bfbdff3f904715a8c24907b20fcab0 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:22.213414 1761 raft_consensus.cc:491] T ded33e0950b544509af11739353c620d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:22.213515 1761 raft_consensus.cc:3058] T ded33e0950b544509af11739353c620d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:22.215945 1761 raft_consensus.cc:513] T ded33e0950b544509af11739353c620d P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.216714 1761 leader_election.cc:290] T ded33e0950b544509af11739353c620d P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:22.217837 917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ded33e0950b544509af11739353c620d" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:22.218032 917 raft_consensus.cc:3058] T ded33e0950b544509af11739353c620d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:22.220903 917 raft_consensus.cc:2466] T ded33e0950b544509af11739353c620d P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:22.223397 992 leader_election.cc:304] T ded33e0950b544509af11739353c620d P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:22.224628 1150 tablet_bootstrap.cc:492] T 46bfbdff3f904715a8c24907b20fcab0 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
W20250901 01:52:22.224663 992 leader_election.cc:336] T ded33e0950b544509af11739353c620d P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:22.224813 1150 ts_tablet_manager.cc:1397] T 46bfbdff3f904715a8c24907b20fcab0 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.004s
I20250901 01:52:22.225010 1761 raft_consensus.cc:2802] T ded33e0950b544509af11739353c620d P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:22.225134 1761 raft_consensus.cc:695] T ded33e0950b544509af11739353c620d P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:22.225401 1761 consensus_queue.cc:237] T ded33e0950b544509af11739353c620d P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.225610 1150 raft_consensus.cc:357] T 46bfbdff3f904715a8c24907b20fcab0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.225845 1150 raft_consensus.cc:383] T 46bfbdff3f904715a8c24907b20fcab0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:22.225924 1150 raft_consensus.cc:738] T 46bfbdff3f904715a8c24907b20fcab0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:22.226172 1150 consensus_queue.cc:260] T 46bfbdff3f904715a8c24907b20fcab0 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.223882 1151 tablet_bootstrap.cc:654] T b2e8b2cd10154de08ddd30c214592a4a P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:22.232692 1150 ts_tablet_manager.cc:1428] T 46bfbdff3f904715a8c24907b20fcab0 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250901 01:52:22.233042 1150 tablet_bootstrap.cc:492] T 6741c56499b44f648fb032c8cf50205b P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:22.238823 1711 raft_consensus.cc:491] T 64fb46b942ff4e4c8749986ecc3dfe65 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:22.238965 1711 raft_consensus.cc:513] T 64fb46b942ff4e4c8749986ecc3dfe65 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.239508 1711 leader_election.cc:290] T 64fb46b942ff4e4c8749986ecc3dfe65 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:22.241115 858 leader_election.cc:336] T 64fb46b942ff4e4c8749986ecc3dfe65 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:22.241605 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "64fb46b942ff4e4c8749986ecc3dfe65" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:22.242084 859 leader_election.cc:343] T 64fb46b942ff4e4c8749986ecc3dfe65 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:22.242228 859 leader_election.cc:304] T 64fb46b942ff4e4c8749986ecc3dfe65 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:22.242626 1711 raft_consensus.cc:2747] T 64fb46b942ff4e4c8749986ecc3dfe65 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:22.242735 1150 tablet_bootstrap.cc:654] T 6741c56499b44f648fb032c8cf50205b P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:22.243851 1151 tablet_bootstrap.cc:492] T b2e8b2cd10154de08ddd30c214592a4a P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:22.244021 1151 ts_tablet_manager.cc:1397] T b2e8b2cd10154de08ddd30c214592a4a P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.037s user 0.004s sys 0.000s
I20250901 01:52:22.244834 1151 raft_consensus.cc:357] T b2e8b2cd10154de08ddd30c214592a4a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.245047 1151 raft_consensus.cc:383] T b2e8b2cd10154de08ddd30c214592a4a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:22.245128 1151 raft_consensus.cc:738] T b2e8b2cd10154de08ddd30c214592a4a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:22.245389 1151 consensus_queue.cc:260] T b2e8b2cd10154de08ddd30c214592a4a P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.247032 1151 ts_tablet_manager.cc:1428] T b2e8b2cd10154de08ddd30c214592a4a P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250901 01:52:22.248085 1151 tablet_bootstrap.cc:492] T bb0cf5a6a27045a08df6845d7473184d P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:22.249711 1761 raft_consensus.cc:491] T a6445b29591548be9f9577db1ba0bfa5 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:22.249820 1761 raft_consensus.cc:513] T a6445b29591548be9f9577db1ba0bfa5 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:22.250277 1761 leader_election.cc:290] T a6445b29591548be9f9577db1ba0bfa5 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:22.251199 917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a6445b29591548be9f9577db1ba0bfa5" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:22.251430 917 raft_consensus.cc:2466] T a6445b29591548be9f9577db1ba0bfa5 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:22.252384 992 leader_election.cc:304] T a6445b29591548be9f9577db1ba0bfa5 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:22.252941 1761 raft_consensus.cc:2802] T a6445b29591548be9f9577db1ba0bfa5 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:22.253008 1761 raft_consensus.cc:491] T a6445b29591548be9f9577db1ba0bfa5 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:22.253060 1761 raft_consensus.cc:3058] T a6445b29591548be9f9577db1ba0bfa5 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:22.255511 1761 raft_consensus.cc:513] T a6445b29591548be9f9577db1ba0bfa5 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:22.255990 1761 leader_election.cc:290] T a6445b29591548be9f9577db1ba0bfa5 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:22.257053 917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a6445b29591548be9f9577db1ba0bfa5" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:22.257254 917 raft_consensus.cc:3058] T a6445b29591548be9f9577db1ba0bfa5 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:22.259860 917 raft_consensus.cc:2466] T a6445b29591548be9f9577db1ba0bfa5 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:22.263201 534 raft_consensus.cc:1273] T 00000000000000000000000000000000 P d2ae2cb6af0f4b3a8f47ab1cdd6567d3 [term 4 FOLLOWER]: Refusing update from remote peer 7ce2288f784044eea010664e3def133f: Log matching property violated. Preceding OpId in replica: term: 2 index: 633. Preceding OpId from leader: term: 4 index: 634. (index mismatch)
I20250901 01:52:22.263885 1904 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2ae2cb6af0f4b3a8f47ab1cdd6567d3" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 634, Last known committed idx: 632, Time since last communication: 0.000s
W20250901 01:52:22.264293 992 leader_election.cc:336] T a6445b29591548be9f9577db1ba0bfa5 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:22.266800 992 leader_election.cc:304] T a6445b29591548be9f9577db1ba0bfa5 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
W20250901 01:52:22.267040 992 leader_election.cc:336] T a6445b29591548be9f9577db1ba0bfa5 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:22.268144 1761 raft_consensus.cc:2802] T a6445b29591548be9f9577db1ba0bfa5 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:22.268273 1761 raft_consensus.cc:695] T a6445b29591548be9f9577db1ba0bfa5 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:22.268543 1761 consensus_queue.cc:237] T a6445b29591548be9f9577db1ba0bfa5 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:22.274454 1151 tablet_bootstrap.cc:654] T bb0cf5a6a27045a08df6845d7473184d P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:22.281814 1151 tablet_bootstrap.cc:492] T bb0cf5a6a27045a08df6845d7473184d P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:22.281967 1151 ts_tablet_manager.cc:1397] T bb0cf5a6a27045a08df6845d7473184d P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.000s
I20250901 01:52:22.288110 1150 tablet_bootstrap.cc:492] T 6741c56499b44f648fb032c8cf50205b P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:22.288164 1151 raft_consensus.cc:357] T bb0cf5a6a27045a08df6845d7473184d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.288300 1150 ts_tablet_manager.cc:1397] T 6741c56499b44f648fb032c8cf50205b P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.055s user 0.004s sys 0.016s
I20250901 01:52:22.288362 1151 raft_consensus.cc:383] T bb0cf5a6a27045a08df6845d7473184d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:22.288479 1151 raft_consensus.cc:738] T bb0cf5a6a27045a08df6845d7473184d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:22.288718 1151 consensus_queue.cc:260] T bb0cf5a6a27045a08df6845d7473184d P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.289166 1150 raft_consensus.cc:357] T 6741c56499b44f648fb032c8cf50205b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.289366 1150 raft_consensus.cc:383] T 6741c56499b44f648fb032c8cf50205b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:22.289466 1150 raft_consensus.cc:738] T 6741c56499b44f648fb032c8cf50205b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:22.289698 1150 consensus_queue.cc:260] T 6741c56499b44f648fb032c8cf50205b P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.290217 1150 ts_tablet_manager.cc:1428] T 6741c56499b44f648fb032c8cf50205b P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250901 01:52:22.289398 1940 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2ae2cb6af0f4b3a8f47ab1cdd6567d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ce2288f784044eea010664e3def133f. Latest consensus state: current_term: 4 leader_uuid: "7ce2288f784044eea010664e3def133f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3add7edb97a40ef9f7c6bc011d13227" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 43871 } } peers { permanent_uuid: "d2ae2cb6af0f4b3a8f47ab1cdd6567d3" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42375 } } peers { permanent_uuid: "7ce2288f784044eea010664e3def133f" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 44231 } } }
I20250901 01:52:22.296213 1940 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2ae2cb6af0f4b3a8f47ab1cdd6567d3 [sys.catalog]: This master's current role is: FOLLOWER
I20250901 01:52:22.298584 1711 raft_consensus.cc:491] T 20f7bfab18134e079e91c52c89468cc3 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:22.298745 1711 raft_consensus.cc:513] T 20f7bfab18134e079e91c52c89468cc3 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.299401 1711 leader_election.cc:290] T 20f7bfab18134e079e91c52c89468cc3 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:22.300756 1150 tablet_bootstrap.cc:492] T f80f13db501c42c7950499df8fcb6fc8 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:22.301542 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20f7bfab18134e079e91c52c89468cc3" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:22.302263 859 leader_election.cc:343] T 20f7bfab18134e079e91c52c89468cc3 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:22.302484 1761 raft_consensus.cc:491] T ef58f477ebeb49639ed7d071e1a62c10 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:22.302520 1150 tablet_bootstrap.cc:654] T f80f13db501c42c7950499df8fcb6fc8 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:22.302598 1761 raft_consensus.cc:513] T ef58f477ebeb49639ed7d071e1a62c10 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.303143 1761 leader_election.cc:290] T ef58f477ebeb49639ed7d071e1a62c10 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:22.304611 992 leader_election.cc:343] T ef58f477ebeb49639ed7d071e1a62c10 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: ef58f477ebeb49639ed7d071e1a62c10
W20250901 01:52:22.305721 992 leader_election.cc:336] T ef58f477ebeb49639ed7d071e1a62c10 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:22.305841 992 leader_election.cc:304] T ef58f477ebeb49639ed7d071e1a62c10 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:22.306190 1761 raft_consensus.cc:2747] T ef58f477ebeb49639ed7d071e1a62c10 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:22.308861 1151 ts_tablet_manager.cc:1428] T bb0cf5a6a27045a08df6845d7473184d P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.027s user 0.002s sys 0.000s
I20250901 01:52:22.309293 1761 raft_consensus.cc:491] T 3a8231590ea643b0a1f83eec4d9a8191 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:22.309423 1761 raft_consensus.cc:513] T 3a8231590ea643b0a1f83eec4d9a8191 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.309999 1761 leader_election.cc:290] T 3a8231590ea643b0a1f83eec4d9a8191 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:22.310899 917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3a8231590ea643b0a1f83eec4d9a8191" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:22.311198 917 raft_consensus.cc:2466] T 3a8231590ea643b0a1f83eec4d9a8191 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:22.311635 1711 raft_consensus.cc:491] T f4fd117c395e41c2ab679c684908c5b7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:22.311846 1711 raft_consensus.cc:513] T f4fd117c395e41c2ab679c684908c5b7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.312659 1711 leader_election.cc:290] T f4fd117c395e41c2ab679c684908c5b7 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:22.313668 992 leader_election.cc:336] T 3a8231590ea643b0a1f83eec4d9a8191 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:22.314244 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f4fd117c395e41c2ab679c684908c5b7" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:22.314700 992 leader_election.cc:304] T 3a8231590ea643b0a1f83eec4d9a8191 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
W20250901 01:52:22.315349 859 leader_election.cc:343] T f4fd117c395e41c2ab679c684908c5b7 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:22.315651 1761 raft_consensus.cc:2802] T 3a8231590ea643b0a1f83eec4d9a8191 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:22.315781 1761 raft_consensus.cc:491] T 3a8231590ea643b0a1f83eec4d9a8191 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:22.315910 1761 raft_consensus.cc:3058] T 3a8231590ea643b0a1f83eec4d9a8191 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:22.318812 1761 raft_consensus.cc:513] T 3a8231590ea643b0a1f83eec4d9a8191 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.319157 1881 raft_consensus.cc:491] T 670d90a37b934579b353bb4e4820127b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:22.319293 1881 raft_consensus.cc:513] T 670d90a37b934579b353bb4e4820127b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.319888 1881 leader_election.cc:290] T 670d90a37b934579b353bb4e4820127b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:22.321218 917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "670d90a37b934579b353bb4e4820127b" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:22.321616 917 raft_consensus.cc:2466] T 670d90a37b934579b353bb4e4820127b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
W20250901 01:52:22.324888 858 leader_election.cc:336] T 20f7bfab18134e079e91c52c89468cc3 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:22.325030 858 leader_election.cc:304] T 20f7bfab18134e079e91c52c89468cc3 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:22.327418 1897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "7ce2288f784044eea010664e3def133f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3add7edb97a40ef9f7c6bc011d13227" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 43871 } } peers { permanent_uuid: "d2ae2cb6af0f4b3a8f47ab1cdd6567d3" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42375 } } peers { permanent_uuid: "7ce2288f784044eea010664e3def133f" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 44231 } } }
I20250901 01:52:22.327700 1897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f [sys.catalog]: This master's current role is: LEADER
I20250901 01:52:22.328573 1711 raft_consensus.cc:2747] T 20f7bfab18134e079e91c52c89468cc3 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:22.328902 1881 raft_consensus.cc:491] T c95798f0daf4437db18a291c4464522e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:22.329057 1881 raft_consensus.cc:513] T c95798f0daf4437db18a291c4464522e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.329787 1881 leader_election.cc:290] T c95798f0daf4437db18a291c4464522e P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:22.331733 858 leader_election.cc:336] T f4fd117c395e41c2ab679c684908c5b7 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:22.334177 1472 consensus_peers.cc:489] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f -> Peer e3add7edb97a40ef9f7c6bc011d13227 (127.0.104.62:43871): Couldn't send request to peer e3add7edb97a40ef9f7c6bc011d13227. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.62:43871: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:22.335474 1150 tablet_bootstrap.cc:492] T f80f13db501c42c7950499df8fcb6fc8 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:22.335631 1150 ts_tablet_manager.cc:1397] T f80f13db501c42c7950499df8fcb6fc8 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.035s user 0.014s sys 0.013s
I20250901 01:52:22.336627 1881 raft_consensus.cc:491] T 831e42de95854541a49ae6b3e039d3e9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:22.336645 1150 raft_consensus.cc:357] T f80f13db501c42c7950499df8fcb6fc8 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.336757 1881 raft_consensus.cc:513] T 831e42de95854541a49ae6b3e039d3e9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.336851 1150 raft_consensus.cc:383] T f80f13db501c42c7950499df8fcb6fc8 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:22.336915 1150 raft_consensus.cc:738] T f80f13db501c42c7950499df8fcb6fc8 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:22.337296 1881 leader_election.cc:290] T 831e42de95854541a49ae6b3e039d3e9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:22.337263 1150 consensus_queue.cc:260] T f80f13db501c42c7950499df8fcb6fc8 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.337777 1150 ts_tablet_manager.cc:1428] T f80f13db501c42c7950499df8fcb6fc8 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250901 01:52:22.338052 1150 tablet_bootstrap.cc:492] T a289a66b2dfa44dc8c8547ff83c1716e P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:22.339782 1150 tablet_bootstrap.cc:654] T a289a66b2dfa44dc8c8547ff83c1716e P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:22.341197 917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "831e42de95854541a49ae6b3e039d3e9" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:22.341356 917 raft_consensus.cc:2466] T 831e42de95854541a49ae6b3e039d3e9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:22.341775 992 leader_election.cc:304] T 670d90a37b934579b353bb4e4820127b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:22.342051 1881 raft_consensus.cc:2802] T 670d90a37b934579b353bb4e4820127b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:22.342110 1881 raft_consensus.cc:491] T 670d90a37b934579b353bb4e4820127b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:22.342160 1881 raft_consensus.cc:3058] T 670d90a37b934579b353bb4e4820127b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:22.344950 1881 raft_consensus.cc:513] T 670d90a37b934579b353bb4e4820127b P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.345264 1711 raft_consensus.cc:491] T 0cb3262973934c8689b851adad60afe0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:22.345391 1711 raft_consensus.cc:513] T 0cb3262973934c8689b851adad60afe0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.345921 1711 leader_election.cc:290] T 0cb3262973934c8689b851adad60afe0 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:22.346710 1881 leader_election.cc:290] T 670d90a37b934579b353bb4e4820127b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:22.347105 992 leader_election.cc:343] T c95798f0daf4437db18a291c4464522e P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: c95798f0daf4437db18a291c4464522e
I20250901 01:52:22.347303 992 leader_election.cc:304] T 831e42de95854541a49ae6b3e039d3e9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
W20250901 01:52:22.349494 992 leader_election.cc:336] T 670d90a37b934579b353bb4e4820127b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:22.350862 917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "670d90a37b934579b353bb4e4820127b" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:22.351042 917 raft_consensus.cc:3058] T 670d90a37b934579b353bb4e4820127b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:22.351317 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cb3262973934c8689b851adad60afe0" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:22.351802 992 leader_election.cc:336] T 831e42de95854541a49ae6b3e039d3e9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:22.351979 992 leader_election.cc:336] T c95798f0daf4437db18a291c4464522e P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:22.352043 992 leader_election.cc:304] T c95798f0daf4437db18a291c4464522e P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:22.352356 1873 raft_consensus.cc:2802] T 831e42de95854541a49ae6b3e039d3e9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:22.352435 1873 raft_consensus.cc:491] T 831e42de95854541a49ae6b3e039d3e9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:22.352502 1873 raft_consensus.cc:3058] T 831e42de95854541a49ae6b3e039d3e9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:22.353708 917 raft_consensus.cc:2466] T 670d90a37b934579b353bb4e4820127b P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:22.354007 919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3a8231590ea643b0a1f83eec4d9a8191" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:22.354171 919 raft_consensus.cc:3058] T 3a8231590ea643b0a1f83eec4d9a8191 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:22.355072 1873 raft_consensus.cc:513] T 831e42de95854541a49ae6b3e039d3e9 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.355609 1873 leader_election.cc:290] T 831e42de95854541a49ae6b3e039d3e9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:22.355748 1873 raft_consensus.cc:2747] T c95798f0daf4437db18a291c4464522e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:22.348953 1761 leader_election.cc:290] T 3a8231590ea643b0a1f83eec4d9a8191 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:22.356685 919 raft_consensus.cc:2466] T 3a8231590ea643b0a1f83eec4d9a8191 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:22.357582 1150 tablet_bootstrap.cc:492] T a289a66b2dfa44dc8c8547ff83c1716e P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:22.357740 1150 ts_tablet_manager.cc:1397] T a289a66b2dfa44dc8c8547ff83c1716e P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.014s
I20250901 01:52:22.358711 1150 raft_consensus.cc:357] T a289a66b2dfa44dc8c8547ff83c1716e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.358901 1150 raft_consensus.cc:383] T a289a66b2dfa44dc8c8547ff83c1716e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:22.358973 1150 raft_consensus.cc:738] T a289a66b2dfa44dc8c8547ff83c1716e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:22.359210 1150 consensus_queue.cc:260] T a289a66b2dfa44dc8c8547ff83c1716e P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.359798 1150 ts_tablet_manager.cc:1428] T a289a66b2dfa44dc8c8547ff83c1716e P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250901 01:52:22.360335 1150 tablet_bootstrap.cc:492] T 3618a3eb4f174c43805ce7dc0e5315cb P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:22.362277 1150 tablet_bootstrap.cc:654] T 3618a3eb4f174c43805ce7dc0e5315cb P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:22.364096 1150 tablet_bootstrap.cc:492] T 3618a3eb4f174c43805ce7dc0e5315cb P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:22.364213 1150 ts_tablet_manager.cc:1397] T 3618a3eb4f174c43805ce7dc0e5315cb P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
W20250901 01:52:22.364297 859 leader_election.cc:343] T 0cb3262973934c8689b851adad60afe0 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:22.365100 1150 raft_consensus.cc:357] T 3618a3eb4f174c43805ce7dc0e5315cb P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.365267 1150 raft_consensus.cc:383] T 3618a3eb4f174c43805ce7dc0e5315cb P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:22.365347 1150 raft_consensus.cc:738] T 3618a3eb4f174c43805ce7dc0e5315cb P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:22.365603 1150 consensus_queue.cc:260] T 3618a3eb4f174c43805ce7dc0e5315cb P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.366061 1150 ts_tablet_manager.cc:1428] T 3618a3eb4f174c43805ce7dc0e5315cb P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250901 01:52:22.366317 1150 tablet_bootstrap.cc:492] T ad45c90ba0da43c9860745e534664a98 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:22.366433 1151 tablet_bootstrap.cc:492] T a54af99bc10045e8b0b6868e5aed98b4 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:22.367337 919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "831e42de95854541a49ae6b3e039d3e9" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:22.367506 919 raft_consensus.cc:3058] T 831e42de95854541a49ae6b3e039d3e9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:22.368516 1150 tablet_bootstrap.cc:654] T ad45c90ba0da43c9860745e534664a98 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:22.370433 919 raft_consensus.cc:2466] T 831e42de95854541a49ae6b3e039d3e9 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:22.371106 1940 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2ae2cb6af0f4b3a8f47ab1cdd6567d3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "7ce2288f784044eea010664e3def133f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3add7edb97a40ef9f7c6bc011d13227" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 43871 } } peers { permanent_uuid: "d2ae2cb6af0f4b3a8f47ab1cdd6567d3" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42375 } } peers { permanent_uuid: "7ce2288f784044eea010664e3def133f" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 44231 } } }
I20250901 01:52:22.371276 1940 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2ae2cb6af0f4b3a8f47ab1cdd6567d3 [sys.catalog]: This master's current role is: FOLLOWER
I20250901 01:52:22.371901 992 leader_election.cc:304] T 670d90a37b934579b353bb4e4820127b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:22.372656 1761 raft_consensus.cc:2802] T 670d90a37b934579b353bb4e4820127b P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:22.372769 1761 raft_consensus.cc:695] T 670d90a37b934579b353bb4e4820127b P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:22.373004 1761 consensus_queue.cc:237] T 670d90a37b934579b353bb4e4820127b P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.374045 992 leader_election.cc:304] T 3a8231590ea643b0a1f83eec4d9a8191 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:22.378026 1150 tablet_bootstrap.cc:492] T ad45c90ba0da43c9860745e534664a98 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:22.378170 1150 ts_tablet_manager.cc:1397] T ad45c90ba0da43c9860745e534664a98 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.005s
I20250901 01:52:22.378877 1150 raft_consensus.cc:357] T ad45c90ba0da43c9860745e534664a98 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.379033 1150 raft_consensus.cc:383] T ad45c90ba0da43c9860745e534664a98 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:22.379077 1150 raft_consensus.cc:738] T ad45c90ba0da43c9860745e534664a98 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:22.379243 1150 consensus_queue.cc:260] T ad45c90ba0da43c9860745e534664a98 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.379726 1150 ts_tablet_manager.cc:1428] T ad45c90ba0da43c9860745e534664a98 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250901 01:52:22.379980 1150 tablet_bootstrap.cc:492] T c754129977904f978f87fd89d7092e36 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:22.381633 1150 tablet_bootstrap.cc:654] T c754129977904f978f87fd89d7092e36 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:22.382120 1881 raft_consensus.cc:2802] T 3a8231590ea643b0a1f83eec4d9a8191 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:22.382256 1881 raft_consensus.cc:695] T 3a8231590ea643b0a1f83eec4d9a8191 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:22.382486 1881 consensus_queue.cc:237] T 3a8231590ea643b0a1f83eec4d9a8191 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.384958 1150 tablet_bootstrap.cc:492] T c754129977904f978f87fd89d7092e36 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:22.385118 1150 ts_tablet_manager.cc:1397] T c754129977904f978f87fd89d7092e36 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250901 01:52:22.386133 1150 raft_consensus.cc:357] T c754129977904f978f87fd89d7092e36 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.386336 1150 raft_consensus.cc:383] T c754129977904f978f87fd89d7092e36 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:22.386410 1150 raft_consensus.cc:738] T c754129977904f978f87fd89d7092e36 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:22.386652 1150 consensus_queue.cc:260] T c754129977904f978f87fd89d7092e36 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.387252 1150 ts_tablet_manager.cc:1428] T c754129977904f978f87fd89d7092e36 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250901 01:52:22.387602 1150 tablet_bootstrap.cc:492] T 17e8340dbd1a406cb782dad3fe88cc7e P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:22.389793 1150 tablet_bootstrap.cc:654] T 17e8340dbd1a406cb782dad3fe88cc7e P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:22.387627 992 leader_election.cc:304] T 831e42de95854541a49ae6b3e039d3e9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:22.390874 1873 raft_consensus.cc:2802] T 831e42de95854541a49ae6b3e039d3e9 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:22.391014 1873 raft_consensus.cc:695] T 831e42de95854541a49ae6b3e039d3e9 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:22.391242 1873 consensus_queue.cc:237] T 831e42de95854541a49ae6b3e039d3e9 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.392741 1151 tablet_bootstrap.cc:654] T a54af99bc10045e8b0b6868e5aed98b4 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:22.398984 992 leader_election.cc:336] T 3a8231590ea643b0a1f83eec4d9a8191 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:22.399266 1150 tablet_bootstrap.cc:492] T 17e8340dbd1a406cb782dad3fe88cc7e P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:22.399405 1150 ts_tablet_manager.cc:1397] T 17e8340dbd1a406cb782dad3fe88cc7e P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
W20250901 01:52:22.399564 992 leader_election.cc:336] T 670d90a37b934579b353bb4e4820127b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:22.400321 1150 raft_consensus.cc:357] T 17e8340dbd1a406cb782dad3fe88cc7e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.400516 1150 raft_consensus.cc:383] T 17e8340dbd1a406cb782dad3fe88cc7e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:22.400713 1150 raft_consensus.cc:738] T 17e8340dbd1a406cb782dad3fe88cc7e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:22.400933 1150 consensus_queue.cc:260] T 17e8340dbd1a406cb782dad3fe88cc7e P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.401392 1150 ts_tablet_manager.cc:1428] T 17e8340dbd1a406cb782dad3fe88cc7e P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250901 01:52:22.401749 1150 tablet_bootstrap.cc:492] T 6cf09e8f299548dba2cce16d7b565bad P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:22.403360 1150 tablet_bootstrap.cc:654] T 6cf09e8f299548dba2cce16d7b565bad P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:22.409818 1910 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250901 01:52:22.411370 1150 tablet_bootstrap.cc:492] T 6cf09e8f299548dba2cce16d7b565bad P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:22.411509 1150 ts_tablet_manager.cc:1397] T 6cf09e8f299548dba2cce16d7b565bad P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20250901 01:52:22.411639 858 leader_election.cc:304] T f4fd117c395e41c2ab679c684908c5b7 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:22.415853 992 leader_election.cc:336] T 831e42de95854541a49ae6b3e039d3e9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:22.420890 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_c2ed44ae561c4b88b22e8975a61b77aa [id=004b578e7e5d47ed9dd12d89076b0f28]
I20250901 01:52:22.422025 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_cb4904990e284a1e8137d11272036154 [id=00e7f7fe6dc44f2787b9dd8e3cbf4a51]
I20250901 01:52:22.422654 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_657603f2bb3c43bc85028218860da978 [id=013cf6104d1a47e592f90f6ba89653aa]
I20250901 01:52:22.423693 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_93afd37289414655980b22934695bd55 [id=01c391f6c76943c9af685a74348eaf41]
I20250901 01:52:22.423904 1711 raft_consensus.cc:2747] T f4fd117c395e41c2ab679c684908c5b7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:22.424232 1150 raft_consensus.cc:357] T 6cf09e8f299548dba2cce16d7b565bad P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:22.424487 1150 raft_consensus.cc:383] T 6cf09e8f299548dba2cce16d7b565bad P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250901 01:52:22.424525 858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:40731, user_credentials={real_user=slave}} blocked reactor thread for 92834.8us
I20250901 01:52:22.424556 1150 raft_consensus.cc:738] T 6cf09e8f299548dba2cce16d7b565bad P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:22.424945 1150 consensus_queue.cc:260] T 6cf09e8f299548dba2cce16d7b565bad P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:22.425351 1711 raft_consensus.cc:491] T e9b179dba2cc49428ba8090c962d6136 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:22.425459 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_1b5729d9768546fe99eaafc4e2604f42 [id=02d0056abe2146ac9af81ebf4447b99d]
I20250901 01:52:22.425526 1150 ts_tablet_manager.cc:1428] T 6cf09e8f299548dba2cce16d7b565bad P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20250901 01:52:22.425487 1711 raft_consensus.cc:513] T e9b179dba2cc49428ba8090c962d6136 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.425822 1150 tablet_bootstrap.cc:492] T e249474dcbf642189560c91ec3a99fdf P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:22.425954 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_fd17a2ad225546aa821489ce94a788a2 [id=033c76e0598342c495ce2be42a0df5dd]
I20250901 01:52:22.426040 1711 leader_election.cc:290] T e9b179dba2cc49428ba8090c962d6136 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:22.426349 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_2227d0e5eae04a08853aa24bfe533ee5 [id=06560cb0c30e40bd876ec65931ab75bf]
I20250901 01:52:22.426800 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_16ca44aa872f426f858aa71d8491b750 [id=088a30537e2a4ac882796e3b5f1e412c]
I20250901 01:52:22.427416 1150 tablet_bootstrap.cc:654] T e249474dcbf642189560c91ec3a99fdf P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:22.427445 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_139f6d174d794a6e9a618e90587d0c0d [id=0baa4993b72e4a8d94ee3059c9d2db6c]
I20250901 01:52:22.427851 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_d61fc9783e7b45e49afb36280adcbd86 [id=0bc0d862df03433dacaf6af87a8390cd]
I20250901 01:52:22.428308 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_0e2cd82b00f1466eaef7f287402e644d [id=0d9b514272444aa1914c4ff033b8f611]
I20250901 01:52:22.428879 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_dfb6e260190a40df8db77152b962cb8e [id=0e10c4b90e884aaeb0a75dbcf3dfd1bc]
I20250901 01:52:22.429383 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_fc0336a00d084f6e98e1d8d32142e66f [id=0e69044796ea4292b1dc100db2f26cee]
I20250901 01:52:22.429924 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_dc9c5d18c7ec478592a76dbd76424b88 [id=126b298fef6c4653952d672adff1db7e]
I20250901 01:52:22.430400 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_0b769136b0664331945b30cfe0482fc6 [id=135c19f9795c4195bb211d0be5fba6a5]
I20250901 01:52:22.432906 1711 raft_consensus.cc:491] T 15e4d972e8254cf899c3e0b71b6e2bd9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:22.433034 1711 raft_consensus.cc:513] T 15e4d972e8254cf899c3e0b71b6e2bd9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:22.433676 1711 leader_election.cc:290] T 15e4d972e8254cf899c3e0b71b6e2bd9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:22.439334 1150 tablet_bootstrap.cc:492] T e249474dcbf642189560c91ec3a99fdf P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:22.439591 1150 ts_tablet_manager.cc:1397] T e249474dcbf642189560c91ec3a99fdf P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.006s
I20250901 01:52:22.441042 1150 raft_consensus.cc:357] T e249474dcbf642189560c91ec3a99fdf P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.441306 1150 raft_consensus.cc:383] T e249474dcbf642189560c91ec3a99fdf P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:22.441390 1150 raft_consensus.cc:738] T e249474dcbf642189560c91ec3a99fdf P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:22.441747 1150 consensus_queue.cc:260] T e249474dcbf642189560c91ec3a99fdf P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.442512 1150 ts_tablet_manager.cc:1428] T e249474dcbf642189560c91ec3a99fdf P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250901 01:52:22.442507 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9b179dba2cc49428ba8090c962d6136" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:22.442659 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e4d972e8254cf899c3e0b71b6e2bd9" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:22.442974 859 leader_election.cc:343] T e9b179dba2cc49428ba8090c962d6136 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
W20250901 01:52:22.443334 859 leader_election.cc:343] T 15e4d972e8254cf899c3e0b71b6e2bd9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:22.443398 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_95515169b000454aa95025573560278f [id=1442b92adbbe43a6b8580cce1bbfe389]
I20250901 01:52:22.444074 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_c7bfa05d564246a6be270afb7a3f0bde [id=16030cb64ac64ff68762651bd095fed1]
I20250901 01:52:22.444562 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_4401d83f75f8464da49d5b811d7dd89d [id=16143c8e6f484f9a873437e709d8466d]
I20250901 01:52:22.444938 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_c1959bda12f14bc9995d832476abf670 [id=17355a61629849dd9d8228a469ab744f]
I20250901 01:52:22.445402 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_2eba037c989c4095bc355667be2fb109 [id=18aecbf3fe4643c4b589ee47706161bd]
I20250901 01:52:22.446437 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_5a72e7206f604795915f3a577d37f874 [id=1d8f34cc14cb4e90a562ff4424b8577a]
I20250901 01:52:22.446935 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_bfb1749ab3454806b139ff43f672583a [id=1eca61e34b5640d7868d6eacd854a307]
W20250901 01:52:22.447213 858 leader_election.cc:336] T 0cb3262973934c8689b851adad60afe0 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:22.447187 1150 tablet_bootstrap.cc:492] T 81c36693199945d98465579b509852e2 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:22.447356 858 leader_election.cc:304] T 0cb3262973934c8689b851adad60afe0 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:22.447561 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_0877ac8202a24e9da650a589bc6dbfcc [id=1f65b3e09c9748f189ecb2b56c5d1132]
I20250901 01:52:22.448021 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_583918346a424cb293d2c35bfa4a5b5f [id=1f677c66803b4cb1bf585cb621a4df6e]
I20250901 01:52:22.448482 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_59039012009944fdbe0179002ea44633 [id=218947d08fb24732b5d0375d4af973e9]
I20250901 01:52:22.448499 1711 raft_consensus.cc:2747] T 0cb3262973934c8689b851adad60afe0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:22.448956 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_3552791ff8104f71a25c92ce0842e349 [id=221faafa4d8f43a2909981853b251810]
I20250901 01:52:22.449055 1151 tablet_bootstrap.cc:492] T a54af99bc10045e8b0b6868e5aed98b4 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
W20250901 01:52:22.449160 858 leader_election.cc:336] T e9b179dba2cc49428ba8090c962d6136 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:22.449203 1151 ts_tablet_manager.cc:1397] T a54af99bc10045e8b0b6868e5aed98b4 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.083s user 0.005s sys 0.002s
I20250901 01:52:22.449259 858 leader_election.cc:304] T e9b179dba2cc49428ba8090c962d6136 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:22.449849 1150 tablet_bootstrap.cc:654] T 81c36693199945d98465579b509852e2 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:22.449959 1151 raft_consensus.cc:357] T a54af99bc10045e8b0b6868e5aed98b4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:22.450132 1151 raft_consensus.cc:383] T a54af99bc10045e8b0b6868e5aed98b4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:22.450186 1151 raft_consensus.cc:738] T a54af99bc10045e8b0b6868e5aed98b4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:22.450392 1711 raft_consensus.cc:2747] T e9b179dba2cc49428ba8090c962d6136 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:22.450378 1151 consensus_queue.cc:260] T a54af99bc10045e8b0b6868e5aed98b4 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:22.451023 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_749fcd4bf97d439c9fd49b138a52a3fd [id=24fffc7ebf5a47bdb37211910301acf7]
W20250901 01:52:22.456486 858 leader_election.cc:336] T 15e4d972e8254cf899c3e0b71b6e2bd9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:22.456606 858 leader_election.cc:304] T 15e4d972e8254cf899c3e0b71b6e2bd9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:22.456818 1711 raft_consensus.cc:2747] T 15e4d972e8254cf899c3e0b71b6e2bd9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:22.457634 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_4185b4f73a2743e08f24a26878cf73ca [id=264cc469a73841e79538843479a165ab]
I20250901 01:52:22.458160 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_83cf2d86a1284544ba2e7aa9ebb66bb7 [id=264d1a223e7d43c8a7a661db6722cb17]
I20250901 01:52:22.458637 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_d2049d0c20bb4949ae6c249b690e7f86 [id=26bba0e60b6b43eb87d084be403c556b]
I20250901 01:52:22.459322 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_942175dadf4b496195b2fa53f39f6ddc [id=2bd332b452f84f25af4225af8e6c6740]
I20250901 01:52:22.459391 1150 tablet_bootstrap.cc:492] T 81c36693199945d98465579b509852e2 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:22.459638 1150 ts_tablet_manager.cc:1397] T 81c36693199945d98465579b509852e2 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.003s
I20250901 01:52:22.459755 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_9d8c1fb24bc24631a2ba2b34a395f24a [id=2d12d6d6275e499ba3928e76bfd3cfc7]
I20250901 01:52:22.460609 1151 ts_tablet_manager.cc:1428] T a54af99bc10045e8b0b6868e5aed98b4 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20250901 01:52:22.460713 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_ed6fd284be6f4ae3a008e21e7ce17930 [id=303e76c918bc4c39aa47a4d24849a358]
I20250901 01:52:22.461225 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_5e32028e03b04431b57a5d3a653afda4 [id=305490ecfddb4d46927f7bb128c7f878]
I20250901 01:52:22.461069 1150 raft_consensus.cc:357] T 81c36693199945d98465579b509852e2 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.461381 1150 raft_consensus.cc:383] T 81c36693199945d98465579b509852e2 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:22.461503 1150 raft_consensus.cc:738] T 81c36693199945d98465579b509852e2 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:22.461867 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_97c44ea002094df8a6af3dccd0099c39 [id=305a9d97cb394d61821c13807ebebc17]
I20250901 01:52:22.461858 1150 consensus_queue.cc:260] T 81c36693199945d98465579b509852e2 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.463153 1150 ts_tablet_manager.cc:1428] T 81c36693199945d98465579b509852e2 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250901 01:52:22.463569 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_83f2c0acedbc4c6982caf5313646e90b [id=31a2e50c32644321825da40176127015]
I20250901 01:52:22.464044 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_c4148158ed7b48a3946559ba978e72fd [id=327c5a63904540508142ab600c308757]
I20250901 01:52:22.464936 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_604f2c5c58d74154bacf2a065a467f32 [id=351a05134f4948039c920cafbe050914]
I20250901 01:52:22.468173 1150 tablet_bootstrap.cc:492] T a8483e7212d94daba7e5401256c33aef P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:22.468734 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_1d63a9f86fe4488b92d5d607b102328c [id=36915a3e843340ba8ef3578ac35a9af4]
W20250901 01:52:22.467229 858 consensus_peers.cc:489] T b009cc95c819431ea9827a3e453b8da3 P 00c282f779c1477d920f24dc0dc16628 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:22.469825 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_f027ccc695a74d1fb4595d9614f30381 [id=3ac80ef18f1044c0b22018ae09997fd3]
I20250901 01:52:22.470300 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_37dedd79eacd4c56ac546fe8b1902259 [id=3c4e9916282d46589939d5fe10592959]
I20250901 01:52:22.470808 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_a5c783cbdb694af69047b31d99ff8def [id=3debd2adcd05421ea96dc426f739f8bd]
I20250901 01:52:22.471330 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_b4fbc34da806475c912052d0fab3fc53 [id=3f1b378fb3444fd08dc947740ac0fa01]
I20250901 01:52:22.471462 1150 tablet_bootstrap.cc:654] T a8483e7212d94daba7e5401256c33aef P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:22.473281 1151 tablet_bootstrap.cc:492] T 9127f9c5239e445ba86012951b7c75d9 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:22.473516 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_82d91a1ba2eb4265896824028dec8d3a [id=3f7624d2901445e39155c12fa68e34e9]
I20250901 01:52:22.474982 1151 tablet_bootstrap.cc:654] T 9127f9c5239e445ba86012951b7c75d9 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:22.476145 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_b5d681f51fc84395bbabe1e5b4cfb358 [id=40d05d90b0354334840ad17764f5b6bb]
I20250901 01:52:22.476750 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_4f2be95a1b824f06ba16e56abcdebc8a [id=434cfb105b894499b15bf360a8cffda4]
W20250901 01:52:22.478372 992 consensus_peers.cc:489] T 4fd0e4f9d86c4362af96ce9964b72328 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250901 01:52:22.479106 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_f840789a58984799bd8057128f6ea439 [id=4453959e273c473fadd737f56f7db3dd]
I20250901 01:52:22.479669 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_8ac222c93e36436c9a3ecb2c489f6d74 [id=44b75816b8b24af0a9530d4a12c2416e]
I20250901 01:52:22.480201 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_52733c9a363b4ed9b6ff11ad8571e421 [id=453153553bea48319e132857f628d48a]
I20250901 01:52:22.481875 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_3940656768d5406fb0e10a71f8f77003 [id=4630ed8abaea4b5195902aa51645d30f]
I20250901 01:52:22.482069 1150 tablet_bootstrap.cc:492] T a8483e7212d94daba7e5401256c33aef P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:22.482295 1150 ts_tablet_manager.cc:1397] T a8483e7212d94daba7e5401256c33aef P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.003s
I20250901 01:52:22.482708 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_99afdac28a4e4ac9ae93b4fbbe704a4d [id=4639e91c14d8427d86c89d3c2b0918ad]
I20250901 01:52:22.483640 1150 raft_consensus.cc:357] T a8483e7212d94daba7e5401256c33aef P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:22.483951 1150 raft_consensus.cc:383] T a8483e7212d94daba7e5401256c33aef P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:22.484061 1150 raft_consensus.cc:738] T a8483e7212d94daba7e5401256c33aef P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:22.484519 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_0f91070385df441b96af4d5406a94724 [id=4747cdd47a954e0ab19609c9526b688d]
I20250901 01:52:22.484436 1150 consensus_queue.cc:260] T a8483e7212d94daba7e5401256c33aef P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:22.485081 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_0e73f80424cd48408f18c2eff1411706 [id=47c13af47c944d0fa79dee115033265d]
I20250901 01:52:22.485591 1150 ts_tablet_manager.cc:1428] T a8483e7212d94daba7e5401256c33aef P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250901 01:52:22.485642 1151 tablet_bootstrap.cc:492] T 9127f9c5239e445ba86012951b7c75d9 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:22.485872 1151 ts_tablet_manager.cc:1397] T 9127f9c5239e445ba86012951b7c75d9 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.001s
I20250901 01:52:22.486243 1150 tablet_bootstrap.cc:492] T 2599f98abfdc4871b573edfc2ad97d22 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:22.486860 1151 raft_consensus.cc:357] T 9127f9c5239e445ba86012951b7c75d9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.487016 1151 raft_consensus.cc:383] T 9127f9c5239e445ba86012951b7c75d9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:22.487066 1151 raft_consensus.cc:738] T 9127f9c5239e445ba86012951b7c75d9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:22.487244 1151 consensus_queue.cc:260] T 9127f9c5239e445ba86012951b7c75d9 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.487684 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_6e461d9cb0ae4dd095975bbc21533794 [id=49673fa73f8c436e9b8026efc4f3cbbc]
I20250901 01:52:22.491853 1150 tablet_bootstrap.cc:654] T 2599f98abfdc4871b573edfc2ad97d22 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:22.492425 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_b300f31bdab14e24ad40f265ac4de311 [id=4a54bc023da7481fbb0ce573be49f4c3]
I20250901 01:52:22.492919 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_df96f1aa51644aed967bdfd4f75588f6 [id=4aabc328f6374ec5b1dea26f778208b2]
I20250901 01:52:22.493386 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_ca5bada2f51949d790e9a38e25fa1bac [id=4ae8785dc52c4e89a1a97a1490356898]
I20250901 01:52:22.493997 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_c714bf77d32b42aeb225bbc5b9a6f43b [id=4c36dd7be4ae4312a19164ef7ef3c6e5]
I20250901 01:52:22.494536 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_1a9de416bd864f7ba7ed302579ebda5a [id=4d54f43f814d48f79124714c187ea383]
I20250901 01:52:22.495114 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_1c8baaaa506548bd85b9f6a7483e5c7a [id=4d9ea39016ca4bf4859f4f80ea4e9ed6]
I20250901 01:52:22.496667 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_8b1086c9a4c44239851c586ea7267e3c [id=523de440fbab4bf687c0f6168e48513c]
I20250901 01:52:22.496925 1151 ts_tablet_manager.cc:1428] T 9127f9c5239e445ba86012951b7c75d9 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20250901 01:52:22.502612 1150 tablet_bootstrap.cc:492] T 2599f98abfdc4871b573edfc2ad97d22 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:22.502858 1150 ts_tablet_manager.cc:1397] T 2599f98abfdc4871b573edfc2ad97d22 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.009s
I20250901 01:52:22.503542 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_21b93b5fe2ce49c6bc637d8cab41fc06 [id=5419e36671ec470a98dff53acdbd4760]
I20250901 01:52:22.504065 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_5ea919f20c6b4a4e91e599357a17a53a [id=543f2074598d4c29972e564be1123fca]
I20250901 01:52:22.504226 1150 raft_consensus.cc:357] T 2599f98abfdc4871b573edfc2ad97d22 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.504594 1150 raft_consensus.cc:383] T 2599f98abfdc4871b573edfc2ad97d22 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:22.504671 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_ce16a1b94f714c2699ee6cd916bb0215 [id=551e2f9616d84026b96b6c22507f29de]
I20250901 01:52:22.504722 1150 raft_consensus.cc:738] T 2599f98abfdc4871b573edfc2ad97d22 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:22.505227 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_2ece91972d494680a8ad00273c10dee4 [id=557afe29976d40f6bfb6ae4ca348f887]
I20250901 01:52:22.505136 1150 consensus_queue.cc:260] T 2599f98abfdc4871b573edfc2ad97d22 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.505825 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_8770c960d4bd4c49b043ffe889b4b87b [id=55c31cd42fac43e6b8eddc048f565d0a]
I20250901 01:52:22.506312 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_faf7ef12324f454396e5ae64d64ebe51 [id=5812902df9ea406ba59ff0ff23f130b6]
I20250901 01:52:22.506526 1151 tablet_bootstrap.cc:492] T d58f69fad01f403eab74a47d3695ed83 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:22.506595 1150 ts_tablet_manager.cc:1428] T 2599f98abfdc4871b573edfc2ad97d22 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20250901 01:52:22.506942 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_50ae3908a7944f70949c54be8fbf1564 [id=58ff38d356d74951b663d8ef7530f1c8]
I20250901 01:52:22.507050 1150 tablet_bootstrap.cc:492] T 8ca5cf4e99014c0d84a3acad4ef713af P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:22.507416 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_4aa01c0e60364804a5e99d552ca3956c [id=59306e9184ca4715b1cd675609807eca]
I20250901 01:52:22.508016 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_6295dbf840ff483a91edb67301e9b439 [id=597ca217b3f945d88f32297e970378a1]
I20250901 01:52:22.508215 1151 tablet_bootstrap.cc:654] T d58f69fad01f403eab74a47d3695ed83 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:22.508858 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_d28d7742171c4465a8b438d2e337b5f7 [id=5bb33f9da2dc4a9cbf17b31cd14fdd23]
I20250901 01:52:22.509583 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_589c302e02474fc3bc5d3a290a8bd54a [id=5d14706774d345c48ca3575feabb17b0]
I20250901 01:52:22.511389 1150 tablet_bootstrap.cc:654] T 8ca5cf4e99014c0d84a3acad4ef713af P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:22.511973 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_a96f01d4ce3c4694a635575ffaa4f02d [id=5dc1ab36f2974f07866b14fedbdabd57]
I20250901 01:52:22.512537 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_5a32379f33204b9e9b2dfe82995134dc [id=5e18d8cdb4fd4399b1a9f8518676f08f]
I20250901 01:52:22.513139 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_0451d0afa26f4d5f8d4214a946eea04f [id=5ee13d46e06a4fa59b1070547573e523]
I20250901 01:52:22.516279 1711 raft_consensus.cc:491] T 7f64ebd080944190be2e64d02d70a8a7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:22.516465 1711 raft_consensus.cc:513] T 7f64ebd080944190be2e64d02d70a8a7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.517069 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_23fb8cbcf1bd43fa8346f3c26c8b4c65 [id=5f9a8c2c0e264e84bff79c8b41a27b4b]
I20250901 01:52:22.517189 1711 leader_election.cc:290] T 7f64ebd080944190be2e64d02d70a8a7 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:22.517944 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f64ebd080944190be2e64d02d70a8a7" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:22.518463 859 leader_election.cc:343] T 7f64ebd080944190be2e64d02d70a8a7 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:22.518996 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_53e30205c5fd488eb3b48c685a159d04 [id=60837b865ee7415ea42a85a953d837a5]
I20250901 01:52:22.519503 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_8f7db2ae75a24dfab71c2a95c6a0a404 [id=61f1e8a842844ed899e5913c76f362ff]
I20250901 01:52:22.520284 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_3dde393d3a564e84b8ef08c0d25265ce [id=649558231b1542b5bc2d90699a644b2a]
W20250901 01:52:22.520304 858 leader_election.cc:336] T 7f64ebd080944190be2e64d02d70a8a7 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:22.520424 858 leader_election.cc:304] T 7f64ebd080944190be2e64d02d70a8a7 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:22.520867 1711 raft_consensus.cc:2747] T 7f64ebd080944190be2e64d02d70a8a7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:22.521466 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_edd1006860a84e209149771e23bb8834 [id=662be9c7921d422f93307d06187b1cae]
I20250901 01:52:22.521889 1761 consensus_queue.cc:1035] T e6c10426f60f43e8bde69d9b1b9c8eb6 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:22.523994 1711 raft_consensus.cc:491] T 42c1156e8dd84c6fa1396b46de13eb6f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:22.524153 1711 raft_consensus.cc:513] T 42c1156e8dd84c6fa1396b46de13eb6f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:22.524892 1711 leader_election.cc:290] T 42c1156e8dd84c6fa1396b46de13eb6f P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:22.525475 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "42c1156e8dd84c6fa1396b46de13eb6f" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:22.525874 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_fea398de1f1948809aebf95c387f4b17 [id=6975ac6754db4664a168e25ad61db164]
W20250901 01:52:22.525929 859 leader_election.cc:343] T 42c1156e8dd84c6fa1396b46de13eb6f P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
W20250901 01:52:22.526392 858 leader_election.cc:336] T 42c1156e8dd84c6fa1396b46de13eb6f P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:22.526530 858 leader_election.cc:304] T 42c1156e8dd84c6fa1396b46de13eb6f P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:22.527065 1711 raft_consensus.cc:2747] T 42c1156e8dd84c6fa1396b46de13eb6f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:22.526476 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_142ccc8cc7414621bc0c31326d5aa0e9 [id=6a01b6c6977546be9c6128ca5f200a92]
I20250901 01:52:22.527681 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_aa1bfa4e8b59486f96c587bc9cf84d73 [id=6b36921a6764455aa19284d36c3f8d32]
I20250901 01:52:22.533748 1151 tablet_bootstrap.cc:492] T d58f69fad01f403eab74a47d3695ed83 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:22.533896 1151 ts_tablet_manager.cc:1397] T d58f69fad01f403eab74a47d3695ed83 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.027s user 0.000s sys 0.007s
I20250901 01:52:22.534509 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_ea8e4576448a476b8bf399421a378e2c [id=6be613d73c4a4044afc7750d2367b4d2]
I20250901 01:52:22.534611 1151 raft_consensus.cc:357] T d58f69fad01f403eab74a47d3695ed83 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.534782 1151 raft_consensus.cc:383] T d58f69fad01f403eab74a47d3695ed83 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:22.534829 1151 raft_consensus.cc:738] T d58f69fad01f403eab74a47d3695ed83 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:22.535017 1151 consensus_queue.cc:260] T d58f69fad01f403eab74a47d3695ed83 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.535346 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_03fb48eaab9b42858ab7cefb722e43a8 [id=6c0576410d7340d6aa493a747e95565b]
I20250901 01:52:22.535985 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_409dad98d7eb455a8e16edfb4acedb34 [id=6f29cdc9974049368af2ae10beadb772]
I20250901 01:52:22.536566 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_1cd3f3fb6b244b3f8796bfca9f7622c8 [id=6fd739b0e725424ca844dbb652ea8028]
I20250901 01:52:22.537156 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_cbeebbc633854c008e07b7d39d2024ee [id=6fdd3d817dc74551923021bd716ba58b]
I20250901 01:52:22.538825 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_02e911ce389e4ad581c0a2f976f99c4e [id=70a5ce71d16b47d4bc6dae97be4548e7]
I20250901 01:52:22.539431 1150 tablet_bootstrap.cc:492] T 8ca5cf4e99014c0d84a3acad4ef713af P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:22.539649 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_7736ed4b5a0d4b1a816500c525e68ad3 [id=7191ff77d44c4c86aa44473814570d09]
I20250901 01:52:22.539685 1150 ts_tablet_manager.cc:1397] T 8ca5cf4e99014c0d84a3acad4ef713af P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.033s user 0.001s sys 0.020s
I20250901 01:52:22.540464 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_202b4c5ccc56474a8b2e7b9d7c4a28d6 [id=746845bd84c6444b8a1e5683eaba4a68]
I20250901 01:52:22.540963 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_fbba493de02249499d26467bf15c3d5f [id=748daa63137c48a5b151bd34dc5b13fe]
I20250901 01:52:22.541190 1150 raft_consensus.cc:357] T 8ca5cf4e99014c0d84a3acad4ef713af P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.541515 1150 raft_consensus.cc:383] T 8ca5cf4e99014c0d84a3acad4ef713af P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:22.541595 1150 raft_consensus.cc:738] T 8ca5cf4e99014c0d84a3acad4ef713af P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:22.541447 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_1186b4165684445da50e713ed6a227f9 [id=74d89178d7594613ba29d2bb01358e80]
I20250901 01:52:22.542196 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_aba07c28a3e248e4b102730106d6a2ba [id=76928d8caa2f4134a014656cf2cc871a]
I20250901 01:52:22.542034 1150 consensus_queue.cc:260] T 8ca5cf4e99014c0d84a3acad4ef713af P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.543128 1150 ts_tablet_manager.cc:1428] T 8ca5cf4e99014c0d84a3acad4ef713af P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250901 01:52:22.543654 1150 tablet_bootstrap.cc:492] T 641a83a3e50d451189c33c7b7b4ca070 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:22.543897 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_14e7f1e7dd314cdcaee09ac2fdf28993 [id=76e521ff309947a58840a7135e783477]
I20250901 01:52:22.547711 1150 tablet_bootstrap.cc:654] T 641a83a3e50d451189c33c7b7b4ca070 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:22.548285 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_c931e9dfbef04fd3b49aa4f43db3d981 [id=77ae3376281349e087a3b3cee41f2a09]
I20250901 01:52:22.548880 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_5610feff8a344dc9be4bc1f0ad8ddd51 [id=78b00856a9b747c19f390b4fdae8ddcf]
I20250901 01:52:22.549343 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_ae42ba019ee14c69b1a16d1177f3642c [id=78cd71321e9a480ca2d7d88327f157ee]
I20250901 01:52:22.550997 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_7dd48000e9b24337b75ceb4c53b6be25 [id=8051adf4041f4a2686a0b8e3850ef29c]
I20250901 01:52:22.551581 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_22d8d38d0b624912a0d8968fb764420e [id=808a8849c99e485b8be7cd17397874ec]
I20250901 01:52:22.552152 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_efc44c841f0b4ae2a168be904751b65c [id=8106fbf3fbe244dabcc6f134e512a9d4]
I20250901 01:52:22.556641 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_0e63d21acfcb4c028c9ea6406b6a04e4 [id=81323e6fbe424655b8e185f52657fadc]
I20250901 01:52:22.557264 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_da76b13a22bc47eea908959d5c1720df [id=815cc5de9497417ea88a3f72e3e729ed]
I20250901 01:52:22.557874 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_9e7ca61fcc324a8984bd1e470162c61f [id=8199118617cb4707bc3ae9b1363750f7]
I20250901 01:52:22.558421 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_77f4810bc83c4d948531da2a45a3792e [id=827adadb52964c88b06dfb995250bda9]
I20250901 01:52:22.559276 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_2b74c001cb4f4d82ab314557f4f09422 [id=85b64046d2fa4f1d871ab43e5c800fb8]
I20250901 01:52:22.561157 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_2f266fe78a3c41c2ab134faad48a377d [id=86e2bc0201e04503a1f60452605eaf27]
I20250901 01:52:22.561754 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_5facf725e0fc44cda7d77f8511030fa5 [id=892696e8040d4f2db8620d1d4ac90e0f]
I20250901 01:52:22.568948 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_1da54cdf1dfd4c0fb5f2538e26f17ad9 [id=8a1a77329ab740dda2d6a5d78bbff906]
I20250901 01:52:22.569865 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_0c9bfebabc22494489b1c1251a412c81 [id=8b74044a78e0493dbaf1835f33334212]
I20250901 01:52:22.570492 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_0b4ac0a1caa5472d909448e41318e679 [id=8c1b263824ec4c15a5ec424364c78895]
I20250901 01:52:22.571799 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_4dacb3b1d2e94e42ac8937eb6eb58375 [id=8ee0aa1ba24042dd8ded004644913b9c]
I20250901 01:52:22.572068 1873 raft_consensus.cc:491] T bb0cf5a6a27045a08df6845d7473184d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:22.572218 1873 raft_consensus.cc:513] T bb0cf5a6a27045a08df6845d7473184d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.572269 1150 tablet_bootstrap.cc:492] T 641a83a3e50d451189c33c7b7b4ca070 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:22.572582 1150 ts_tablet_manager.cc:1397] T 641a83a3e50d451189c33c7b7b4ca070 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.029s user 0.001s sys 0.021s
I20250901 01:52:22.572785 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_7c12d66048a6470e8c227dc1d9f45fa4 [id=91472f220fe44297b79d3eec8b9b80e8]
I20250901 01:52:22.572839 1873 leader_election.cc:290] T bb0cf5a6a27045a08df6845d7473184d P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:22.573673 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_dc856980ff4c4aafb92cf1fd4f29e694 [id=917ac6e5eaf54d019896a1d1ae2f6622]
I20250901 01:52:22.574020 917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb0cf5a6a27045a08df6845d7473184d" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:22.574185 917 raft_consensus.cc:2466] T bb0cf5a6a27045a08df6845d7473184d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:22.574057 1150 raft_consensus.cc:357] T 641a83a3e50d451189c33c7b7b4ca070 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.574375 1150 raft_consensus.cc:383] T 641a83a3e50d451189c33c7b7b4ca070 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:22.574514 1150 raft_consensus.cc:738] T 641a83a3e50d451189c33c7b7b4ca070 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:22.574714 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_8cd9c1e6a8e74232b32d36451269562a [id=946ec3e53dcd4de4b8501b3e87e99072]
I20250901 01:52:22.574853 992 leader_election.cc:304] T bb0cf5a6a27045a08df6845d7473184d P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:22.575122 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_91214af783ff4aa3b6c3cbdb0358307e [id=94d314a62cf24de19d6e1fff9b3794ce]
I20250901 01:52:22.574929 1150 consensus_queue.cc:260] T 641a83a3e50d451189c33c7b7b4ca070 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.575384 1873 raft_consensus.cc:2802] T bb0cf5a6a27045a08df6845d7473184d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:22.575457 1873 raft_consensus.cc:491] T bb0cf5a6a27045a08df6845d7473184d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:22.575510 1873 raft_consensus.cc:3058] T bb0cf5a6a27045a08df6845d7473184d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:22.575512 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_97a5ec9ec34c4a89b0f917704b43e52b [id=956e796e0f5747f6ad5b4e4163639c35]
I20250901 01:52:22.575930 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_d51ef24ec0f846c7ad212360eb478cd7 [id=958e51a6bbee47638e504e1a44094737]
I20250901 01:52:22.576129 1150 ts_tablet_manager.cc:1428] T 641a83a3e50d451189c33c7b7b4ca070 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250901 01:52:22.576653 1150 tablet_bootstrap.cc:492] T eaa521c52a2f4408b59770bd2a05907c P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:22.578162 1873 raft_consensus.cc:513] T bb0cf5a6a27045a08df6845d7473184d P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.578637 1873 leader_election.cc:290] T bb0cf5a6a27045a08df6845d7473184d P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:22.580053 917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb0cf5a6a27045a08df6845d7473184d" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:22.580199 917 raft_consensus.cc:3058] T bb0cf5a6a27045a08df6845d7473184d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:22.580727 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_8825dd28afbe464193eb1b668cc9326a [id=9aa62cb634314a0a849612c54c1220d9]
I20250901 01:52:22.580802 1150 tablet_bootstrap.cc:654] T eaa521c52a2f4408b59770bd2a05907c P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:22.581444 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_7f273a7efc7c496d8d8f9a49871026f4 [id=9b67513e73dd441096d0b0086b0e6a6b]
I20250901 01:52:22.581851 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_5e84ae9a9c2a4084a1f9c5ffb4594de2 [id=9bebb2e486ab48888e6b523281ac1e88]
I20250901 01:52:22.582316 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_a89cc8e48563460baabc34c821ceae35 [id=9c3930e9fcfe4b49ba4bea67a2b0ad07]
I20250901 01:52:22.582752 917 raft_consensus.cc:2466] T bb0cf5a6a27045a08df6845d7473184d P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:22.582834 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_e533a2595cc9420f9ab7a36bda6dc4a3 [id=9ce26dcf307149b595a43b6ac4562fba]
W20250901 01:52:22.583240 992 leader_election.cc:336] T bb0cf5a6a27045a08df6845d7473184d P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:22.583556 992 leader_election.cc:304] T bb0cf5a6a27045a08df6845d7473184d P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:22.583989 1881 raft_consensus.cc:2802] T bb0cf5a6a27045a08df6845d7473184d P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:22.584088 1881 raft_consensus.cc:695] T bb0cf5a6a27045a08df6845d7473184d P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:22.584098 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_3c7c628f35fb4d02bb79a34995f12938 [id=a39e8b171a874df1890d072732e55df9]
I20250901 01:52:22.584292 1881 consensus_queue.cc:237] T bb0cf5a6a27045a08df6845d7473184d P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.585408 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_364cf1adeb2c4102aad338f2e228fb0b [id=a4d0af97df5949cca940a81a41b8e454]
I20250901 01:52:22.585935 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_ac189f6a9fcf457e9fa0d77badc81594 [id=a541b55be0dc420a8353ef4793b925b5]
I20250901 01:52:22.587702 1150 tablet_bootstrap.cc:492] T eaa521c52a2f4408b59770bd2a05907c P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:22.588063 1150 ts_tablet_manager.cc:1397] T eaa521c52a2f4408b59770bd2a05907c P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.002s
I20250901 01:52:22.589483 1150 raft_consensus.cc:357] T eaa521c52a2f4408b59770bd2a05907c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:22.589762 1150 raft_consensus.cc:383] T eaa521c52a2f4408b59770bd2a05907c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:22.589903 1150 raft_consensus.cc:738] T eaa521c52a2f4408b59770bd2a05907c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:22.590485 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_75ba3ba70ca24a94920e7bce96190ff2 [id=a72f676761b045e29dc0daa74687333d]
I20250901 01:52:22.590327 1150 consensus_queue.cc:260] T eaa521c52a2f4408b59770bd2a05907c P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:22.591581 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_219f0092f43743039ae1a61fedb3e4d1 [id=a7b1d2a8ace14ef8bcce0a3bed41e85d]
I20250901 01:52:22.592144 1150 ts_tablet_manager.cc:1428] T eaa521c52a2f4408b59770bd2a05907c P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250901 01:52:22.592697 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_2f09538b6fbe401f909105decb56e1cb [id=a90d1f10531042c3bb7a196b8be61c5f]
I20250901 01:52:22.593147 1150 tablet_bootstrap.cc:492] T 081e2630e284498aa819f85bf8326315 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:22.593870 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_9163155cb611406a8ce0d5ae600a15c8 [id=aa9c7768ab244d60bedd5ea94cc5e94e]
I20250901 01:52:22.596181 1150 tablet_bootstrap.cc:654] T 081e2630e284498aa819f85bf8326315 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:22.596792 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_db61e055dbd7450898e82c4c0c4763be [id=aabca67f6f754820977ac9c4c20f3e4f]
I20250901 01:52:22.597347 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_9d0d7a93ff15424092012d4d681b8ae0 [id=aad143eea5c2439299aad87723399c58]
I20250901 01:52:22.597822 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_da8cc5e7d1e24421b170ecca2fc2147b [id=ab670bb02c6e4ed491e7831ad73cdfc6]
I20250901 01:52:22.598408 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_bec114a616504798bede9a9066e57192 [id=ae79f2c40fd7483ea400a36cf3c48888]
I20250901 01:52:22.607054 1150 tablet_bootstrap.cc:492] T 081e2630e284498aa819f85bf8326315 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:22.607337 1150 ts_tablet_manager.cc:1397] T 081e2630e284498aa819f85bf8326315 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.015s user 0.011s sys 0.000s
I20250901 01:52:22.607822 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_259a716264db43c49671b6fef6665f0e [id=aebce875b7dd4718a3611eec4ca56917]
I20250901 01:52:22.608333 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_37e451883f5642f094a03b5ca48a0b25 [id=aed8d4b4fa6a4fbf850cf5718610108f]
I20250901 01:52:22.608811 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_442e11b9211c4b209c68f653d7bfc313 [id=af349c3f534e4e8e99fdae0344187c44]
I20250901 01:52:22.608989 1150 raft_consensus.cc:357] T 081e2630e284498aa819f85bf8326315 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.609264 1150 raft_consensus.cc:383] T 081e2630e284498aa819f85bf8326315 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:22.609398 1150 raft_consensus.cc:738] T 081e2630e284498aa819f85bf8326315 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:22.609797 1150 consensus_queue.cc:260] T 081e2630e284498aa819f85bf8326315 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.610258 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_bcd502e1360b48b1953fa97eadce7006 [id=b3914f33e856466e8479d0a2263ec56f]
I20250901 01:52:22.610754 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_820e1c5a64cd4128b7ae1b70bd951964 [id=b43a92ed46164d199b445287ef039bc6]
I20250901 01:52:22.611466 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_da6f34d486984dedbc970080cf14482b [id=b572fb895e3248978514cf759bc531d4]
I20250901 01:52:22.611665 1150 ts_tablet_manager.cc:1428] T 081e2630e284498aa819f85bf8326315 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20250901 01:52:22.612015 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_8d0db1143f1547c2a7d41775b4d9ef09 [id=b67bfe518a094e0781ef931524b55990]
I20250901 01:52:22.612538 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_8ad4807386b64e4195236b6fe222ad8e [id=b7e4716d607141b785fc30c9376fe11a]
I20250901 01:52:22.612736 1150 tablet_bootstrap.cc:492] T 8ed6616ce54548ba83ba299c4ecf3c38 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:22.613075 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_b8f8ee74648c406eaf589d3ab80934db [id=b8f69313439543be9c11eea88dfa69c5]
I20250901 01:52:22.613552 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_c298ecfa989347b0b7a4d7227611539f [id=b94e5ce799ee4046982550aace3a08d6]
I20250901 01:52:22.614058 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_5e65c5f068b84995a2d7a76fa65dda0f [id=b9d81ddd38fe47838877b010be2b40d0]
I20250901 01:52:22.615029 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_36f5f8854ed14d1ea48eb18865d00659 [id=bdf8a102ae7d43228173d7b82507d80d]
I20250901 01:52:22.615921 1150 tablet_bootstrap.cc:654] T 8ed6616ce54548ba83ba299c4ecf3c38 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:22.616070 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_bb44f1b7c92140309a212038f8bf1286 [id=bfd87b9b708449d3b16e3e0639561a49]
I20250901 01:52:22.616626 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_d5a6ac03c9ff4969ada556d1318b64f0 [id=c1483e9d0db64d468b12344500f9cac8]
I20250901 01:52:22.617331 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_da2c70a8f2fe49a0a8184e88c0a9e0ea [id=c1b1e9e3876244f68d31c63ce7bdebf1]
I20250901 01:52:22.622885 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_f0e56dba1e8a4ed49a760e204c1ff96e [id=c3865e058daa4cea9a89f5046fca712a]
I20250901 01:52:22.623924 1711 raft_consensus.cc:491] T 46bfbdff3f904715a8c24907b20fcab0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:22.624058 1711 raft_consensus.cc:513] T 46bfbdff3f904715a8c24907b20fcab0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.624660 1711 leader_election.cc:290] T 46bfbdff3f904715a8c24907b20fcab0 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:22.624846 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_d56ff20d129b440cbab7b15c4af6fa46 [id=c6acca4c20ba4dfaa385fc3c124507a3]
I20250901 01:52:22.625694 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_85b9e2f3f8f7400aa833201daae93dc3 [id=cac94430e07342dfa9b266df6472b6e3]
W20250901 01:52:22.626039 858 leader_election.cc:336] T 46bfbdff3f904715a8c24907b20fcab0 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:22.627913 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_203547c9458e43b3b48ea051d56892c6 [id=cc26c89db2484a6fad919231116285f2]
I20250901 01:52:22.628108 1150 tablet_bootstrap.cc:492] T 8ed6616ce54548ba83ba299c4ecf3c38 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:22.628506 1150 ts_tablet_manager.cc:1397] T 8ed6616ce54548ba83ba299c4ecf3c38 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.007s
I20250901 01:52:22.629315 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_d6b752d82c2b4be88b762e67bac0d5a1 [id=ce82be9787f14aa6bcdcb070228abc51]
I20250901 01:52:22.630028 1150 raft_consensus.cc:357] T 8ed6616ce54548ba83ba299c4ecf3c38 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.630347 1150 raft_consensus.cc:383] T 8ed6616ce54548ba83ba299c4ecf3c38 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:22.630461 1150 raft_consensus.cc:738] T 8ed6616ce54548ba83ba299c4ecf3c38 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:22.630905 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_0f959237963c4e33932e193285c0eacb [id=cecdd8d6ca6d445c9fd82decd2a95366]
I20250901 01:52:22.630895 1150 consensus_queue.cc:260] T 8ed6616ce54548ba83ba299c4ecf3c38 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.631471 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_b0e916c76a7a4766aa0887cc8691488f [id=cf61137a852c48748a29f833f1860da7]
I20250901 01:52:22.632591 1150 ts_tablet_manager.cc:1428] T 8ed6616ce54548ba83ba299c4ecf3c38 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250901 01:52:22.633116 1150 tablet_bootstrap.cc:492] T 7a2f6e2c7ee2402ea310c058e046416c P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:22.633435 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_66ad08ce0b6840d4848bcddb304cf88b [id=d1cd5288671540c0a95f7a037363014e]
I20250901 01:52:22.633944 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_420b0979d6fd42da9b320456f4ccc9ce [id=d2004563c4374790954327f05c305cc6]
I20250901 01:52:22.636859 1150 tablet_bootstrap.cc:654] T 7a2f6e2c7ee2402ea310c058e046416c P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:22.637486 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_df46d9604f6a4a19b4d7c3424f48d90a [id=d28ebc8fa6d745919bf025a5a9f6ebca]
I20250901 01:52:22.638283 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_eecfab7239cc40c5a2f1fb6ac6d35dc9 [id=d3dbeb4258424133844e20b181ee44e4]
I20250901 01:52:22.639398 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_9e04f976a93f4d30baccf90fa3622dfe [id=d68960dbe1154bd8a8016f9a5718e5ad]
I20250901 01:52:22.640277 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_5e8920128d214deb96ef83f369a8b3f1 [id=d7615918ae0f48d394fbe509b82127e7]
I20250901 01:52:22.640342 1872 raft_consensus.cc:491] T b2e8b2cd10154de08ddd30c214592a4a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:22.640535 1872 raft_consensus.cc:513] T b2e8b2cd10154de08ddd30c214592a4a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.641304 1872 leader_election.cc:290] T b2e8b2cd10154de08ddd30c214592a4a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:22.641635 1151 ts_tablet_manager.cc:1428] T d58f69fad01f403eab74a47d3695ed83 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.108s user 0.000s sys 0.002s
I20250901 01:52:22.641983 1151 tablet_bootstrap.cc:492] T fae2779fac164b9f8052c3666db803b8 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:22.643630 1151 tablet_bootstrap.cc:654] T fae2779fac164b9f8052c3666db803b8 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:22.644399 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_046289f393ff432aae0b033bdec1fb17 [id=db57b18a9ce24e068b362f5fbe940531]
I20250901 01:52:22.645011 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_f611c1794e564a0496a761da46171d0e [id=dc22dac261644596b2d65e804791462d]
I20250901 01:52:22.646179 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_95e1338a49a04f22b679e363e253262e [id=e24b91b4ac904ef0a8c4725f70bced2f]
I20250901 01:52:22.652946 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_2d7b072b55354009b9cce1aef9408e35 [id=e42970b06935463186a7d4a23b9e74c2]
I20250901 01:52:22.653532 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_d9e28b97a2e94ca8bf0ee8a01acd76e8 [id=e4a6d6fe44f947e59089e1d7855b785c]
I20250901 01:52:22.654085 1150 tablet_bootstrap.cc:492] T 7a2f6e2c7ee2402ea310c058e046416c P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:22.654287 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_122585be3bdc45fb9498d02564fb8567 [id=e7b9f3937d0241b291db9ccbaa523da2]
I20250901 01:52:22.654338 1150 ts_tablet_manager.cc:1397] T 7a2f6e2c7ee2402ea310c058e046416c P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.008s
I20250901 01:52:22.654827 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_15c9f4f02eb24474936a43d705f90f7b [id=ea928501fe954cd0b81d38add4ef7128]
I20250901 01:52:22.655337 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_09ffc2a7b4b14b25adbc5471363c02af [id=eaa8142db4844b908cf36991898aca3c]
I20250901 01:52:22.656054 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_e37cc89ad7ee4d96852b121d67b06248 [id=edbc04078c904f29a16a1bbb354dad92]
I20250901 01:52:22.655910 1150 raft_consensus.cc:357] T 7a2f6e2c7ee2402ea310c058e046416c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:22.656265 1150 raft_consensus.cc:383] T 7a2f6e2c7ee2402ea310c058e046416c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:22.656379 1150 raft_consensus.cc:738] T 7a2f6e2c7ee2402ea310c058e046416c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:22.656791 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_54422fcb2b624e7fa8d42cba3558a327 [id=ee0eb3570f614c5099121396d0e52ca3]
I20250901 01:52:22.656841 1150 consensus_queue.cc:260] T 7a2f6e2c7ee2402ea310c058e046416c P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:22.657338 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_e113aaaa38db48fdb16d32569b81b930 [id=f09f20dc4db844a8abc825e9086093ba]
I20250901 01:52:22.657845 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_3f9422b08b0349b9a5466218f0fe1b06 [id=f0dfb3811e744bd4ba6d59195cebde6f]
I20250901 01:52:22.658288 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_b362441111a241e19cc1a09bdeefea49 [id=f1d75be759db40ebb72ff32560f1d038]
I20250901 01:52:22.658638 1150 ts_tablet_manager.cc:1428] T 7a2f6e2c7ee2402ea310c058e046416c P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20250901 01:52:22.659989 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_583e53c2e86544e9be68eb27ab6068a3 [id=f376eaf65bb549429aa71b03fca81647]
I20250901 01:52:22.660327 1150 tablet_bootstrap.cc:492] T c58edaa76ebf47dfb1d9d5fab6de7dfa P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:22.662815 1872 raft_consensus.cc:491] T 23af154a6f1c4e2e9352fb17c9024f10 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:22.662954 1872 raft_consensus.cc:513] T 23af154a6f1c4e2e9352fb17c9024f10 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.663486 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_2d3d2227e6a24bd2a86f457e553e698b [id=f48fd53522234290b6c0373fd7b02570]
I20250901 01:52:22.663568 1872 leader_election.cc:290] T 23af154a6f1c4e2e9352fb17c9024f10 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:22.664093 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_b3ec3b285fac447f8c5c4a9ac4196628 [id=f49c96ff2eaa44cc9b754225cb9e6fdf]
I20250901 01:52:22.665055 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_277b91103c4d41d2a9f9424f99082b29 [id=f9dcf4937b1143248bfa9d88053fce17]
I20250901 01:52:22.667025 1150 tablet_bootstrap.cc:654] T c58edaa76ebf47dfb1d9d5fab6de7dfa P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:22.667760 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_3b46a0a1f77a4cea842e19b3c89de475 [id=fac15e527f974e228cf2e53026fba40e]
I20250901 01:52:22.668277 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_ce1fd26ac65a4ed085d023e10a849e5c [id=fb088d31bba64f04871fbfd355d6263b]
I20250901 01:52:22.669029 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_a4ebd8ab87c842f082da5cd04d1bddac [id=fd6d8e92d6e64a89a7d38aa761376955]
I20250901 01:52:22.669601 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_4a412a59b6db421eb0187ffd02ecf541 [id=fd72e46c84634b6c9ea74970480ecc62]
I20250901 01:52:22.672986 1910 catalog_manager.cc:671] Loaded metadata for table default.Table_ce86974fd8974de9a5a737954fac2ad7 [id=ff772d20b287476f94716379b86d2a71]
W20250901 01:52:22.684772 992 leader_election.cc:336] T bb0cf5a6a27045a08df6845d7473184d P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:22.686435 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46bfbdff3f904715a8c24907b20fcab0" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:22.687145 918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2e8b2cd10154de08ddd30c214592a4a" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:22.687314 918 raft_consensus.cc:2466] T b2e8b2cd10154de08ddd30c214592a4a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
W20250901 01:52:22.687460 859 leader_election.cc:343] T 46bfbdff3f904715a8c24907b20fcab0 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:22.687574 859 leader_election.cc:304] T 46bfbdff3f904715a8c24907b20fcab0 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:22.687808 1711 raft_consensus.cc:2747] T 46bfbdff3f904715a8c24907b20fcab0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:22.687672 1150 tablet_bootstrap.cc:492] T c58edaa76ebf47dfb1d9d5fab6de7dfa P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:22.688277 1150 ts_tablet_manager.cc:1397] T c58edaa76ebf47dfb1d9d5fab6de7dfa P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.028s user 0.005s sys 0.012s
W20250901 01:52:22.688611 992 leader_election.cc:343] T 23af154a6f1c4e2e9352fb17c9024f10 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 23af154a6f1c4e2e9352fb17c9024f10
I20250901 01:52:22.689944 1150 raft_consensus.cc:357] T c58edaa76ebf47dfb1d9d5fab6de7dfa P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.690106 1150 raft_consensus.cc:383] T c58edaa76ebf47dfb1d9d5fab6de7dfa P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:22.690157 1150 raft_consensus.cc:738] T c58edaa76ebf47dfb1d9d5fab6de7dfa P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:22.690384 1150 consensus_queue.cc:260] T c58edaa76ebf47dfb1d9d5fab6de7dfa P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.691241 1150 ts_tablet_manager.cc:1428] T c58edaa76ebf47dfb1d9d5fab6de7dfa P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250901 01:52:22.692662 1151 tablet_bootstrap.cc:492] T fae2779fac164b9f8052c3666db803b8 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:22.692788 1151 ts_tablet_manager.cc:1397] T fae2779fac164b9f8052c3666db803b8 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.051s user 0.002s sys 0.001s
W20250901 01:52:22.694624 858 consensus_peers.cc:489] T 629db80a8ab444e582628578155638fd P 00c282f779c1477d920f24dc0dc16628 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:22.694862 1873 consensus_queue.cc:1035] T 1286c0ab85704837a99b6e5a62184d77 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:22.695107 1151 raft_consensus.cc:357] T fae2779fac164b9f8052c3666db803b8 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.695245 1151 raft_consensus.cc:383] T fae2779fac164b9f8052c3666db803b8 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:22.695312 1151 raft_consensus.cc:738] T fae2779fac164b9f8052c3666db803b8 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:22.695521 1151 consensus_queue.cc:260] T fae2779fac164b9f8052c3666db803b8 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.695622 1873 consensus_queue.cc:1035] T 2ff914352c7c43de800f3d54ecf63072 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:22.696143 1873 raft_consensus.cc:491] T 40d74d7b5a3b4d1dae09eebd9e07a0c4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:22.696245 1873 raft_consensus.cc:513] T 40d74d7b5a3b4d1dae09eebd9e07a0c4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.696730 1873 leader_election.cc:290] T 40d74d7b5a3b4d1dae09eebd9e07a0c4 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:22.696985 992 leader_election.cc:304] T b2e8b2cd10154de08ddd30c214592a4a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:22.697475 1873 raft_consensus.cc:2802] T b2e8b2cd10154de08ddd30c214592a4a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:22.697554 1873 raft_consensus.cc:491] T b2e8b2cd10154de08ddd30c214592a4a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:22.697630 1873 raft_consensus.cc:3058] T b2e8b2cd10154de08ddd30c214592a4a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:22.698005 1150 tablet_bootstrap.cc:492] T ea846f88b3534beebd0001b7f17b5ca9 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:22.699695 1910 tablet_loader.cc:96] loaded metadata for tablet 008d23f236c348b6be9aee718f8698af (table default.Table_0e2cd82b00f1466eaef7f287402e644d [id=0d9b514272444aa1914c4ff033b8f611])
I20250901 01:52:22.700343 1873 raft_consensus.cc:513] T b2e8b2cd10154de08ddd30c214592a4a P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.701058 1150 tablet_bootstrap.cc:654] T ea846f88b3534beebd0001b7f17b5ca9 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:22.701218 1873 leader_election.cc:290] T b2e8b2cd10154de08ddd30c214592a4a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:22.701551 992 leader_election.cc:336] T b2e8b2cd10154de08ddd30c214592a4a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:22.701767 992 consensus_peers.cc:489] T e6c10426f60f43e8bde69d9b1b9c8eb6 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250901 01:52:22.701858 992 leader_election.cc:336] T 23af154a6f1c4e2e9352fb17c9024f10 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:22.701946 992 leader_election.cc:304] T 23af154a6f1c4e2e9352fb17c9024f10 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:22.702718 1910 tablet_loader.cc:96] loaded metadata for tablet 061b760d8e6647c6ab0b5d538840a230 (table default.Table_fbba493de02249499d26467bf15c3d5f [id=748daa63137c48a5b151bd34dc5b13fe])
I20250901 01:52:22.705426 1873 raft_consensus.cc:2747] T 23af154a6f1c4e2e9352fb17c9024f10 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:22.705718 992 consensus_peers.cc:489] T 1286c0ab85704837a99b6e5a62184d77 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:22.706132 1910 tablet_loader.cc:96] loaded metadata for tablet 06d5eeb70e664f73bf643922d445bc7a (table default.Table_2f266fe78a3c41c2ab134faad48a377d [id=86e2bc0201e04503a1f60452605eaf27])
I20250901 01:52:22.707305 924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "40d74d7b5a3b4d1dae09eebd9e07a0c4" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:22.707490 924 raft_consensus.cc:2466] T 40d74d7b5a3b4d1dae09eebd9e07a0c4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:22.707626 921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2e8b2cd10154de08ddd30c214592a4a" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:22.707777 921 raft_consensus.cc:3058] T b2e8b2cd10154de08ddd30c214592a4a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:22.708842 992 leader_election.cc:304] T 40d74d7b5a3b4d1dae09eebd9e07a0c4 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:22.709913 1873 raft_consensus.cc:2802] T 40d74d7b5a3b4d1dae09eebd9e07a0c4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:22.709982 1873 raft_consensus.cc:491] T 40d74d7b5a3b4d1dae09eebd9e07a0c4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:22.710037 1873 raft_consensus.cc:3058] T 40d74d7b5a3b4d1dae09eebd9e07a0c4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:22.711184 921 raft_consensus.cc:2466] T b2e8b2cd10154de08ddd30c214592a4a P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:22.712683 1873 raft_consensus.cc:513] T 40d74d7b5a3b4d1dae09eebd9e07a0c4 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.713310 1873 leader_election.cc:290] T 40d74d7b5a3b4d1dae09eebd9e07a0c4 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:22.695996 1151 ts_tablet_manager.cc:1428] T fae2779fac164b9f8052c3666db803b8 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250901 01:52:22.713804 1151 tablet_bootstrap.cc:492] T 822382a1781b4304bce29355ce83730e P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:22.714023 1910 tablet_loader.cc:96] loaded metadata for tablet 081e2630e284498aa819f85bf8326315 (table default.Table_9e7ca61fcc324a8984bd1e470162c61f [id=8199118617cb4707bc3ae9b1363750f7])
I20250901 01:52:22.714881 1910 tablet_loader.cc:96] loaded metadata for tablet 088ef4861dc14f81ae470c3aaee5b4e1 (table default.Table_9e04f976a93f4d30baccf90fa3622dfe [id=d68960dbe1154bd8a8016f9a5718e5ad])
I20250901 01:52:22.715596 1151 tablet_bootstrap.cc:654] T 822382a1781b4304bce29355ce83730e P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:22.716101 1910 tablet_loader.cc:96] loaded metadata for tablet 0958b0a4068344abb0f9db601d7dd2ad (table default.Table_b300f31bdab14e24ad40f265ac4de311 [id=4a54bc023da7481fbb0ce573be49f4c3])
I20250901 01:52:22.718732 1910 tablet_loader.cc:96] loaded metadata for tablet 0cba5329ab31440999faec599d704633 (table default.Table_a5c783cbdb694af69047b31d99ff8def [id=3debd2adcd05421ea96dc426f739f8bd])
I20250901 01:52:22.720798 1910 tablet_loader.cc:96] loaded metadata for tablet 0e4009963cfe42d6ba164fc828897f09 (table default.Table_749fcd4bf97d439c9fd49b138a52a3fd [id=24fffc7ebf5a47bdb37211910301acf7])
I20250901 01:52:22.725035 919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "40d74d7b5a3b4d1dae09eebd9e07a0c4" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:22.725482 919 raft_consensus.cc:3058] T 40d74d7b5a3b4d1dae09eebd9e07a0c4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:22.727072 1981 raft_consensus.cc:491] T 7a2f6e2c7ee2402ea310c058e046416c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:22.727315 1981 raft_consensus.cc:513] T 7a2f6e2c7ee2402ea310c058e046416c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:22.728166 1981 leader_election.cc:290] T 7a2f6e2c7ee2402ea310c058e046416c P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:22.729135 858 leader_election.cc:336] T 7a2f6e2c7ee2402ea310c058e046416c P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:22.729565 1151 tablet_bootstrap.cc:492] T 822382a1781b4304bce29355ce83730e P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:22.729709 1151 ts_tablet_manager.cc:1397] T 822382a1781b4304bce29355ce83730e P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20250901 01:52:22.730511 1151 raft_consensus.cc:357] T 822382a1781b4304bce29355ce83730e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.730686 1151 raft_consensus.cc:383] T 822382a1781b4304bce29355ce83730e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:22.730743 1151 raft_consensus.cc:738] T 822382a1781b4304bce29355ce83730e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:22.730911 919 raft_consensus.cc:2466] T 40d74d7b5a3b4d1dae09eebd9e07a0c4 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:22.730926 1151 consensus_queue.cc:260] T 822382a1781b4304bce29355ce83730e P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.731429 1151 ts_tablet_manager.cc:1428] T 822382a1781b4304bce29355ce83730e P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250901 01:52:22.731703 1151 tablet_bootstrap.cc:492] T 121803e79ab64ec0b948b448646b9d0a P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:22.733774 1151 tablet_bootstrap.cc:654] T 121803e79ab64ec0b948b448646b9d0a P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:22.735750 1151 tablet_bootstrap.cc:492] T 121803e79ab64ec0b948b448646b9d0a P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:22.735865 1151 ts_tablet_manager.cc:1397] T 121803e79ab64ec0b948b448646b9d0a P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20250901 01:52:22.737264 1910 tablet_loader.cc:96] loaded metadata for tablet 0f37a24e693847f98b574950379081f3 (table default.Table_16ca44aa872f426f858aa71d8491b750 [id=088a30537e2a4ac882796e3b5f1e412c])
I20250901 01:52:22.738806 1910 tablet_loader.cc:96] loaded metadata for tablet 1495de70f0844f82a0eea99c059f076b (table default.Table_02e911ce389e4ad581c0a2f976f99c4e [id=70a5ce71d16b47d4bc6dae97be4548e7])
I20250901 01:52:22.739495 1910 tablet_loader.cc:96] loaded metadata for tablet 14b466b2a5a249aa841f1b8b06301edc (table default.Table_ae42ba019ee14c69b1a16d1177f3642c [id=78cd71321e9a480ca2d7d88327f157ee])
I20250901 01:52:22.739619 992 leader_election.cc:304] T b2e8b2cd10154de08ddd30c214592a4a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:22.740012 992 leader_election.cc:304] T 40d74d7b5a3b4d1dae09eebd9e07a0c4 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:22.739988 1873 raft_consensus.cc:2802] T b2e8b2cd10154de08ddd30c214592a4a P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:22.740224 1873 raft_consensus.cc:695] T b2e8b2cd10154de08ddd30c214592a4a P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:22.740491 1150 tablet_bootstrap.cc:492] T ea846f88b3534beebd0001b7f17b5ca9 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:22.740613 1150 ts_tablet_manager.cc:1397] T ea846f88b3534beebd0001b7f17b5ca9 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.009s
I20250901 01:52:22.740556 1873 consensus_queue.cc:237] T b2e8b2cd10154de08ddd30c214592a4a P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.740566 1761 raft_consensus.cc:2802] T 40d74d7b5a3b4d1dae09eebd9e07a0c4 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:22.740962 1761 raft_consensus.cc:695] T 40d74d7b5a3b4d1dae09eebd9e07a0c4 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:22.741159 1761 consensus_queue.cc:237] T 40d74d7b5a3b4d1dae09eebd9e07a0c4 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.741281 1150 raft_consensus.cc:357] T ea846f88b3534beebd0001b7f17b5ca9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.741410 1150 raft_consensus.cc:383] T ea846f88b3534beebd0001b7f17b5ca9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:22.741469 1150 raft_consensus.cc:738] T ea846f88b3534beebd0001b7f17b5ca9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:22.741645 1150 consensus_queue.cc:260] T ea846f88b3534beebd0001b7f17b5ca9 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.742180 1150 ts_tablet_manager.cc:1428] T ea846f88b3534beebd0001b7f17b5ca9 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250901 01:52:22.742506 1150 tablet_bootstrap.cc:492] T aec951f2cc84499b9162b9fa24d5440e P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:22.744545 1150 tablet_bootstrap.cc:654] T aec951f2cc84499b9162b9fa24d5440e P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:22.746199 1910 tablet_loader.cc:96] loaded metadata for tablet 15a4f99ae43745b9a3f60052d916a7bd (table default.Table_23fb8cbcf1bd43fa8346f3c26c8b4c65 [id=5f9a8c2c0e264e84bff79c8b41a27b4b])
I20250901 01:52:22.747869 1910 tablet_loader.cc:96] loaded metadata for tablet 15e4d972e8254cf899c3e0b71b6e2bd9 (table default.Table_ce16a1b94f714c2699ee6cd916bb0215 [id=551e2f9616d84026b96b6c22507f29de])
I20250901 01:52:22.751636 1910 tablet_loader.cc:96] loaded metadata for tablet 17e8340dbd1a406cb782dad3fe88cc7e (table default.Table_50ae3908a7944f70949c54be8fbf1564 [id=58ff38d356d74951b663d8ef7530f1c8])
I20250901 01:52:22.754809 1910 tablet_loader.cc:96] loaded metadata for tablet 18498c9cb09141d08c1c5c0854b63ebd (table default.Table_77f4810bc83c4d948531da2a45a3792e [id=827adadb52964c88b06dfb995250bda9])
I20250901 01:52:22.757139 1151 raft_consensus.cc:357] T 121803e79ab64ec0b948b448646b9d0a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:22.757413 1151 raft_consensus.cc:383] T 121803e79ab64ec0b948b448646b9d0a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:22.757467 1151 raft_consensus.cc:738] T 121803e79ab64ec0b948b448646b9d0a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:22.757697 1151 consensus_queue.cc:260] T 121803e79ab64ec0b948b448646b9d0a P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:22.758247 1910 tablet_loader.cc:96] loaded metadata for tablet 186ec810d02e4748aad15228b5866ac1 (table default.Table_9d0d7a93ff15424092012d4d681b8ae0 [id=aad143eea5c2439299aad87723399c58])
I20250901 01:52:22.763967 1992 raft_consensus.cc:491] T f8a5886ddb644a86b76df44aab309057 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:22.766132 1992 raft_consensus.cc:513] T f8a5886ddb644a86b76df44aab309057 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.768004 1992 leader_election.cc:290] T f8a5886ddb644a86b76df44aab309057 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:22.770437 1981 raft_consensus.cc:491] T d0c3c363dda444519a17dca63ad9caa7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:22.770639 1981 raft_consensus.cc:513] T d0c3c363dda444519a17dca63ad9caa7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:22.768956 1151 ts_tablet_manager.cc:1428] T 121803e79ab64ec0b948b448646b9d0a P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.033s user 0.003s sys 0.001s
I20250901 01:52:22.771353 1981 leader_election.cc:290] T d0c3c363dda444519a17dca63ad9caa7 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:22.771732 1151 tablet_bootstrap.cc:492] T d77aadd7291647a0bae7c421f244c252 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
W20250901 01:52:22.772683 858 leader_election.cc:336] T d0c3c363dda444519a17dca63ad9caa7 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:22.773365 1910 tablet_loader.cc:96] loaded metadata for tablet 219fe3882d884ee49b08671b547cac85 (table default.Table_3552791ff8104f71a25c92ce0842e349 [id=221faafa4d8f43a2909981853b251810])
I20250901 01:52:22.775616 1150 tablet_bootstrap.cc:492] T aec951f2cc84499b9162b9fa24d5440e P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:22.775776 1150 ts_tablet_manager.cc:1397] T aec951f2cc84499b9162b9fa24d5440e P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.011s
I20250901 01:52:22.776746 1150 raft_consensus.cc:357] T aec951f2cc84499b9162b9fa24d5440e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.776952 1150 raft_consensus.cc:383] T aec951f2cc84499b9162b9fa24d5440e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:22.777009 1150 raft_consensus.cc:738] T aec951f2cc84499b9162b9fa24d5440e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:22.777856 1150 consensus_queue.cc:260] T aec951f2cc84499b9162b9fa24d5440e P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.779107 1150 ts_tablet_manager.cc:1428] T aec951f2cc84499b9162b9fa24d5440e P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250901 01:52:22.780385 1150 tablet_bootstrap.cc:492] T 89d40f0331114490b51de3ba6d773571 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:22.782671 1150 tablet_bootstrap.cc:654] T 89d40f0331114490b51de3ba6d773571 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:22.787699 1981 raft_consensus.cc:491] T 8ed6616ce54548ba83ba299c4ecf3c38 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250901 01:52:22.787851 992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:40561, user_credentials={real_user=slave}} blocked reactor thread for 47559.6us
I20250901 01:52:22.787896 1981 raft_consensus.cc:513] T 8ed6616ce54548ba83ba299c4ecf3c38 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.788679 1981 leader_election.cc:290] T 8ed6616ce54548ba83ba299c4ecf3c38 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:22.789855 1992 raft_consensus.cc:491] T 5d794471dbed4477b9d52fd411fa4d10 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:22.789991 1992 raft_consensus.cc:513] T 5d794471dbed4477b9d52fd411fa4d10 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.790102 919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f8a5886ddb644a86b76df44aab309057" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:22.790277 919 raft_consensus.cc:2466] T f8a5886ddb644a86b76df44aab309057 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:22.790520 1992 leader_election.cc:290] T 5d794471dbed4477b9d52fd411fa4d10 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:22.790547 1055 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8ed6616ce54548ba83ba299c4ecf3c38" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:22.790776 1054 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0c3c363dda444519a17dca63ad9caa7" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:22.791034 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a2f6e2c7ee2402ea310c058e046416c" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:22.790031 1910 tablet_loader.cc:96] loaded metadata for tablet 246b33338de647689d7404c70676e76f (table default.Table_219f0092f43743039ae1a61fedb3e4d1 [id=a7b1d2a8ace14ef8bcce0a3bed41e85d])
W20250901 01:52:22.791234 992 leader_election.cc:336] T 40d74d7b5a3b4d1dae09eebd9e07a0c4 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:22.791685 992 leader_election.cc:336] T b2e8b2cd10154de08ddd30c214592a4a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:22.791764 1910 tablet_loader.cc:96] loaded metadata for tablet 24ddc13d1bb04db4a7850acbff091577 (table default.Table_2eba037c989c4095bc355667be2fb109 [id=18aecbf3fe4643c4b589ee47706161bd])
I20250901 01:52:22.792575 1910 tablet_loader.cc:96] loaded metadata for tablet 2599f98abfdc4871b573edfc2ad97d22 (table default.Table_0e63d21acfcb4c028c9ea6406b6a04e4 [id=81323e6fbe424655b8e185f52657fadc])
W20250901 01:52:22.792670 858 leader_election.cc:336] T 8ed6616ce54548ba83ba299c4ecf3c38 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:22.792976 1711 raft_consensus.cc:491] T 4855448edb4b40c184838aa7f81c112b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250901 01:52:22.793025 859 leader_election.cc:343] T 8ed6616ce54548ba83ba299c4ecf3c38 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:22.793082 1711 raft_consensus.cc:513] T 4855448edb4b40c184838aa7f81c112b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:22.793243 1981 consensus_queue.cc:1035] T 629db80a8ab444e582628578155638fd P 00c282f779c1477d920f24dc0dc16628 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:22.793310 917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d794471dbed4477b9d52fd411fa4d10" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:22.793448 917 raft_consensus.cc:2466] T 5d794471dbed4477b9d52fd411fa4d10 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:22.793586 1711 leader_election.cc:290] T 4855448edb4b40c184838aa7f81c112b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:22.793121 859 leader_election.cc:304] T 8ed6616ce54548ba83ba299c4ecf3c38 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:22.794301 1711 raft_consensus.cc:2747] T 8ed6616ce54548ba83ba299c4ecf3c38 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:22.795029 858 leader_election.cc:336] T 4855448edb4b40c184838aa7f81c112b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:22.795341 1910 tablet_loader.cc:96] loaded metadata for tablet 268439ff8d3a4f7a89f3d6395f2079c5 (table default.Table_b4fbc34da806475c912052d0fab3fc53 [id=3f1b378fb3444fd08dc947740ac0fa01])
I20250901 01:52:22.796442 1940 raft_consensus.cc:491] T 00000000000000000000000000000000 P d2ae2cb6af0f4b3a8f47ab1cdd6567d3 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7ce2288f784044eea010664e3def133f)
I20250901 01:52:22.796641 1940 raft_consensus.cc:513] T 00000000000000000000000000000000 P d2ae2cb6af0f4b3a8f47ab1cdd6567d3 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3add7edb97a40ef9f7c6bc011d13227" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 43871 } } peers { permanent_uuid: "d2ae2cb6af0f4b3a8f47ab1cdd6567d3" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42375 } } peers { permanent_uuid: "7ce2288f784044eea010664e3def133f" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 44231 } }
I20250901 01:52:22.797302 1940 leader_election.cc:290] T 00000000000000000000000000000000 P d2ae2cb6af0f4b3a8f47ab1cdd6567d3 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e3add7edb97a40ef9f7c6bc011d13227 (127.0.104.62:43871), 7ce2288f784044eea010664e3def133f (127.0.104.60:44231)
I20250901 01:52:22.798249 1151 tablet_bootstrap.cc:654] T d77aadd7291647a0bae7c421f244c252 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:22.804221 1910 tablet_loader.cc:96] loaded metadata for tablet 26c8ed083cea4dc0acca80d00b11b29b (table default.Table_5a32379f33204b9e9b2dfe82995134dc [id=5e18d8cdb4fd4399b1a9f8518676f08f])
I20250901 01:52:22.805063 1910 tablet_loader.cc:96] loaded metadata for tablet 26e3ad68393547e5b6e3dd6b9ec1dbbe (table default.Table_7f273a7efc7c496d8d8f9a49871026f4 [id=9b67513e73dd441096d0b0086b0e6a6b])
I20250901 01:52:22.806875 1910 tablet_loader.cc:96] loaded metadata for tablet 2936fdd7f303487db7b03dccff71867d (table default.Table_8ad4807386b64e4195236b6fe222ad8e [id=b7e4716d607141b785fc30c9376fe11a])
I20250901 01:52:22.807428 1151 tablet_bootstrap.cc:492] T d77aadd7291647a0bae7c421f244c252 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:22.807582 1151 ts_tablet_manager.cc:1397] T d77aadd7291647a0bae7c421f244c252 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.036s user 0.005s sys 0.004s
I20250901 01:52:22.807969 1910 tablet_loader.cc:96] loaded metadata for tablet 2973b59bb1df41d18d9a5b6a29ae336f (table default.Table_202b4c5ccc56474a8b2e7b9d7c4a28d6 [id=746845bd84c6444b8a1e5683eaba4a68])
I20250901 01:52:22.808588 1151 raft_consensus.cc:357] T d77aadd7291647a0bae7c421f244c252 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.808784 1151 raft_consensus.cc:383] T d77aadd7291647a0bae7c421f244c252 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:22.808837 1151 raft_consensus.cc:738] T d77aadd7291647a0bae7c421f244c252 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:22.808924 1150 tablet_bootstrap.cc:492] T 89d40f0331114490b51de3ba6d773571 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:22.809032 1151 consensus_queue.cc:260] T d77aadd7291647a0bae7c421f244c252 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.809502 1910 tablet_loader.cc:96] loaded metadata for tablet 2dacb608897d47d28512df87dc320927 (table default.Table_91214af783ff4aa3b6c3cbdb0358307e [id=94d314a62cf24de19d6e1fff9b3794ce])
I20250901 01:52:22.809545 1151 ts_tablet_manager.cc:1428] T d77aadd7291647a0bae7c421f244c252 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250901 01:52:22.809486 1150 ts_tablet_manager.cc:1397] T 89d40f0331114490b51de3ba6d773571 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.004s
I20250901 01:52:22.809808 1151 tablet_bootstrap.cc:492] T 3c1fd0820d8a4995abe027f685da7f7b P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:22.810647 1910 tablet_loader.cc:96] loaded metadata for tablet 2df2240631844defa433270a3829d65b (table default.Table_6295dbf840ff483a91edb67301e9b439 [id=597ca217b3f945d88f32297e970378a1])
I20250901 01:52:22.811615 1151 tablet_bootstrap.cc:654] T 3c1fd0820d8a4995abe027f685da7f7b P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:22.812937 1150 raft_consensus.cc:357] T 89d40f0331114490b51de3ba6d773571 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.813491 1150 raft_consensus.cc:383] T 89d40f0331114490b51de3ba6d773571 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:22.814018 1711 raft_consensus.cc:491] T 641a83a3e50d451189c33c7b7b4ca070 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:22.814136 1711 raft_consensus.cc:513] T 641a83a3e50d451189c33c7b7b4ca070 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.814697 1711 leader_election.cc:290] T 641a83a3e50d451189c33c7b7b4ca070 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:22.813778 1150 raft_consensus.cc:738] T 89d40f0331114490b51de3ba6d773571 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
W20250901 01:52:22.815752 859 leader_election.cc:343] T d0c3c363dda444519a17dca63ad9caa7 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:22.815869 859 leader_election.cc:304] T d0c3c363dda444519a17dca63ad9caa7 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:22.815902 992 leader_election.cc:336] T f8a5886ddb644a86b76df44aab309057 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:22.816219 859 leader_election.cc:343] T 7a2f6e2c7ee2402ea310c058e046416c P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:22.816305 859 leader_election.cc:304] T 7a2f6e2c7ee2402ea310c058e046416c P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:22.816499 858 leader_election.cc:336] T 641a83a3e50d451189c33c7b7b4ca070 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:22.816615 992 leader_election.cc:336] T 40d74d7b5a3b4d1dae09eebd9e07a0c4 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:22.816685 1981 raft_consensus.cc:2747] T d0c3c363dda444519a17dca63ad9caa7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:22.816836 1981 raft_consensus.cc:2747] T 7a2f6e2c7ee2402ea310c058e046416c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:22.818082 1054 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4855448edb4b40c184838aa7f81c112b" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:22.818298 1055 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "641a83a3e50d451189c33c7b7b4ca070" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:22.818908 992 leader_election.cc:304] T f8a5886ddb644a86b76df44aab309057 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:22.819499 1761 raft_consensus.cc:2802] T f8a5886ddb644a86b76df44aab309057 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:22.819588 1761 raft_consensus.cc:491] T f8a5886ddb644a86b76df44aab309057 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:22.819671 1761 raft_consensus.cc:3058] T f8a5886ddb644a86b76df44aab309057 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:22.823477 1761 raft_consensus.cc:513] T f8a5886ddb644a86b76df44aab309057 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.823347 1150 consensus_queue.cc:260] T 89d40f0331114490b51de3ba6d773571 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.824179 1761 leader_election.cc:290] T f8a5886ddb644a86b76df44aab309057 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:22.827252 1150 ts_tablet_manager.cc:1428] T 89d40f0331114490b51de3ba6d773571 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.017s user 0.003s sys 0.000s
I20250901 01:52:22.830060 1150 tablet_bootstrap.cc:492] T f13b972e25214f168421439fe3943e8a P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:22.826416 1151 tablet_bootstrap.cc:492] T 3c1fd0820d8a4995abe027f685da7f7b P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:22.832459 1151 ts_tablet_manager.cc:1397] T 3c1fd0820d8a4995abe027f685da7f7b P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.001s
I20250901 01:52:22.833199 1151 raft_consensus.cc:357] T 3c1fd0820d8a4995abe027f685da7f7b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.833359 1151 raft_consensus.cc:383] T 3c1fd0820d8a4995abe027f685da7f7b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:22.833412 1151 raft_consensus.cc:738] T 3c1fd0820d8a4995abe027f685da7f7b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:22.833647 1151 consensus_queue.cc:260] T 3c1fd0820d8a4995abe027f685da7f7b P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.834113 1151 ts_tablet_manager.cc:1428] T 3c1fd0820d8a4995abe027f685da7f7b P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250901 01:52:22.834368 1151 tablet_bootstrap.cc:492] T 72ba689f9230411995dc96e2b45601df P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:22.835566 1999 raft_consensus.cc:491] T 8ca5cf4e99014c0d84a3acad4ef713af P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:22.835832 1999 raft_consensus.cc:513] T 8ca5cf4e99014c0d84a3acad4ef713af P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.836716 1150 tablet_bootstrap.cc:654] T f13b972e25214f168421439fe3943e8a P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:22.837085 858 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111) [suppressed 87 similar messages]
I20250901 01:52:22.837376 1999 leader_election.cc:290] T 8ca5cf4e99014c0d84a3acad4ef713af P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:22.837921 858 leader_election.cc:336] T 8ca5cf4e99014c0d84a3acad4ef713af P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:22.839763 1151 tablet_bootstrap.cc:654] T 72ba689f9230411995dc96e2b45601df P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:22.824718 992 leader_election.cc:304] T 5d794471dbed4477b9d52fd411fa4d10 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:22.841156 1990 raft_consensus.cc:2802] T 5d794471dbed4477b9d52fd411fa4d10 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:22.841253 1990 raft_consensus.cc:491] T 5d794471dbed4477b9d52fd411fa4d10 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:22.841317 1990 raft_consensus.cc:3058] T 5d794471dbed4477b9d52fd411fa4d10 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:22.844595 1990 raft_consensus.cc:513] T 5d794471dbed4477b9d52fd411fa4d10 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.845213 1990 leader_election.cc:290] T 5d794471dbed4477b9d52fd411fa4d10 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:22.849710 859 leader_election.cc:343] T 4855448edb4b40c184838aa7f81c112b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:22.849843 859 leader_election.cc:304] T 4855448edb4b40c184838aa7f81c112b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:22.849994 1910 tablet_loader.cc:96] loaded metadata for tablet 2ff914352c7c43de800f3d54ecf63072 (table default.Table_1186b4165684445da50e713ed6a227f9 [id=74d89178d7594613ba29d2bb01358e80])
I20250901 01:52:22.850157 1999 raft_consensus.cc:2747] T 4855448edb4b40c184838aa7f81c112b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:22.850533 1999 consensus_queue.cc:1035] T b009cc95c819431ea9827a3e453b8da3 P 00c282f779c1477d920f24dc0dc16628 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:22.850653 1910 tablet_loader.cc:96] loaded metadata for tablet 315c15cf0afc44d4a79cb29294281d60 (table default.Table_37e451883f5642f094a03b5ca48a0b25 [id=aed8d4b4fa6a4fbf850cf5718610108f])
W20250901 01:52:22.850654 859 leader_election.cc:343] T 641a83a3e50d451189c33c7b7b4ca070 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:22.850734 859 leader_election.cc:304] T 641a83a3e50d451189c33c7b7b4ca070 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:22.850962 1999 raft_consensus.cc:2747] T 641a83a3e50d451189c33c7b7b4ca070 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:22.851317 1910 tablet_loader.cc:96] loaded metadata for tablet 3239e58531d445fa80165cf678bc277c (table default.Table_9163155cb611406a8ce0d5ae600a15c8 [id=aa9c7768ab244d60bedd5ea94cc5e94e])
I20250901 01:52:22.851573 917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f8a5886ddb644a86b76df44aab309057" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:22.851742 917 raft_consensus.cc:3058] T f8a5886ddb644a86b76df44aab309057 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:22.854797 917 raft_consensus.cc:2466] T f8a5886ddb644a86b76df44aab309057 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:22.855649 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8ca5cf4e99014c0d84a3acad4ef713af" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:22.859007 1151 tablet_bootstrap.cc:492] T 72ba689f9230411995dc96e2b45601df P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:22.859145 1151 ts_tablet_manager.cc:1397] T 72ba689f9230411995dc96e2b45601df P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.025s user 0.001s sys 0.001s
I20250901 01:52:22.860175 1151 raft_consensus.cc:357] T 72ba689f9230411995dc96e2b45601df P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.860349 1151 raft_consensus.cc:383] T 72ba689f9230411995dc96e2b45601df P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:22.860452 1151 raft_consensus.cc:738] T 72ba689f9230411995dc96e2b45601df P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:22.860754 1151 consensus_queue.cc:260] T 72ba689f9230411995dc96e2b45601df P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:22.862732 992 leader_election.cc:336] T 5d794471dbed4477b9d52fd411fa4d10 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:22.863014 992 consensus_peers.cc:489] T 2ff914352c7c43de800f3d54ecf63072 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:22.864167 1151 ts_tablet_manager.cc:1428] T 72ba689f9230411995dc96e2b45601df P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20250901 01:52:22.864706 917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d794471dbed4477b9d52fd411fa4d10" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:22.864895 917 raft_consensus.cc:3058] T 5d794471dbed4477b9d52fd411fa4d10 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
W20250901 01:52:22.864974 859 leader_election.cc:343] T 8ca5cf4e99014c0d84a3acad4ef713af P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:22.865088 859 leader_election.cc:304] T 8ca5cf4e99014c0d84a3acad4ef713af P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:22.865473 1151 tablet_bootstrap.cc:492] T b6daeff0070a4db08e90020752d21c29 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:22.866406 1711 raft_consensus.cc:2747] T 8ca5cf4e99014c0d84a3acad4ef713af P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:22.867472 992 leader_election.cc:304] T f8a5886ddb644a86b76df44aab309057 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:22.867839 917 raft_consensus.cc:2466] T 5d794471dbed4477b9d52fd411fa4d10 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:22.868898 1910 tablet_loader.cc:96] loaded metadata for tablet 3363fd67b1ad44849ecb7b4090b61a72 (table default.Table_09ffc2a7b4b14b25adbc5471363c02af [id=eaa8142db4844b908cf36991898aca3c])
I20250901 01:52:22.869122 1151 tablet_bootstrap.cc:654] T b6daeff0070a4db08e90020752d21c29 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:22.869278 1910 tablet_loader.cc:96] loaded metadata for tablet 33f637f6a8134215bb1995ec38440073 (table default.Table_2ece91972d494680a8ad00273c10dee4 [id=557afe29976d40f6bfb6ae4ca348f887])
I20250901 01:52:22.869628 1910 tablet_loader.cc:96] loaded metadata for tablet 34042ea5c1b645969c1de41fe587c343 (table default.Table_e37cc89ad7ee4d96852b121d67b06248 [id=edbc04078c904f29a16a1bbb354dad92])
I20250901 01:52:22.873812 1910 tablet_loader.cc:96] loaded metadata for tablet 34ebb441c27747ec860ff2e25f9a0adc (table default.Table_da6f34d486984dedbc970080cf14482b [id=b572fb895e3248978514cf759bc531d4])
W20250901 01:52:22.874801 511 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.62:43871, user_credentials={real_user=slave}} blocked reactor thread for 68609.3us
I20250901 01:52:22.876168 1910 tablet_loader.cc:96] loaded metadata for tablet 3618a3eb4f174c43805ce7dc0e5315cb (table default.Table_83f2c0acedbc4c6982caf5313646e90b [id=31a2e50c32644321825da40176127015])
W20250901 01:52:22.878515 511 leader_election.cc:336] T 00000000000000000000000000000000 P d2ae2cb6af0f4b3a8f47ab1cdd6567d3 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer e3add7edb97a40ef9f7c6bc011d13227 (127.0.104.62:43871): Network error: Client connection negotiation failed: client connection to 127.0.104.62:43871: connect: Connection refused (error 111)
I20250901 01:52:22.878995 1990 raft_consensus.cc:2802] T f8a5886ddb644a86b76df44aab309057 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:22.879189 1990 raft_consensus.cc:695] T f8a5886ddb644a86b76df44aab309057 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:22.879519 1990 consensus_queue.cc:237] T f8a5886ddb644a86b76df44aab309057 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.880882 1761 raft_consensus.cc:491] T 75e7a90815e449c39f386901a504143c P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:22.881029 1761 raft_consensus.cc:513] T 75e7a90815e449c39f386901a504143c P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.881721 1761 leader_election.cc:290] T 75e7a90815e449c39f386901a504143c P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:22.890259 1910 tablet_loader.cc:96] loaded metadata for tablet 3697821cd29446bb908b020599df0db9 (table default.Table_efc44c841f0b4ae2a168be904751b65c [id=8106fbf3fbe244dabcc6f134e512a9d4])
I20250901 01:52:22.890328 992 leader_election.cc:304] T 5d794471dbed4477b9d52fd411fa4d10 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:22.891934 1761 raft_consensus.cc:2802] T 5d794471dbed4477b9d52fd411fa4d10 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:22.892441 1761 raft_consensus.cc:695] T 5d794471dbed4477b9d52fd411fa4d10 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:22.893023 1910 tablet_loader.cc:96] loaded metadata for tablet 36e10328c34242ebb4dbd8f294162696 (table default.Table_d51ef24ec0f846c7ad212360eb478cd7 [id=958e51a6bbee47638e504e1a44094737])
I20250901 01:52:22.893081 1761 consensus_queue.cc:237] T 5d794471dbed4477b9d52fd411fa4d10 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.893841 1910 tablet_loader.cc:96] loaded metadata for tablet 3a2bc5ad1cd243daa3c92fceca8ba414 (table default.Table_122585be3bdc45fb9498d02564fb8567 [id=e7b9f3937d0241b291db9ccbaa523da2])
I20250901 01:52:22.894723 1910 tablet_loader.cc:96] loaded metadata for tablet 3a8231590ea643b0a1f83eec4d9a8191 (table default.Table_a4ebd8ab87c842f082da5cd04d1bddac [id=fd6d8e92d6e64a89a7d38aa761376955])
I20250901 01:52:22.898597 1987 raft_consensus.cc:491] T 3c1fd0820d8a4995abe027f685da7f7b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:22.898793 1987 raft_consensus.cc:513] T 3c1fd0820d8a4995abe027f685da7f7b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.899561 1987 leader_election.cc:290] T 3c1fd0820d8a4995abe027f685da7f7b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:22.899544 1999 raft_consensus.cc:491] T aec951f2cc84499b9162b9fa24d5440e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:22.899720 1999 raft_consensus.cc:513] T aec951f2cc84499b9162b9fa24d5440e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.900389 1910 tablet_loader.cc:96] loaded metadata for tablet 3c12d6a70ac24dce82e851782e6fe5aa (table default.Table_d2049d0c20bb4949ae6c249b690e7f86 [id=26bba0e60b6b43eb87d084be403c556b])
I20250901 01:52:22.900538 1999 leader_election.cc:290] T aec951f2cc84499b9162b9fa24d5440e P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:22.900485 992 leader_election.cc:336] T 5d794471dbed4477b9d52fd411fa4d10 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:22.900796 1910 tablet_loader.cc:96] loaded metadata for tablet 3c1fd0820d8a4995abe027f685da7f7b (table default.Table_bcd502e1360b48b1953fa97eadce7006 [id=b3914f33e856466e8479d0a2263ec56f])
W20250901 01:52:22.900998 992 leader_election.cc:336] T f8a5886ddb644a86b76df44aab309057 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:22.902108 858 leader_election.cc:336] T aec951f2cc84499b9162b9fa24d5440e P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:22.903661 1910 tablet_loader.cc:96] loaded metadata for tablet 3d90c6ef06d1436d8e2bb1333e7d9985 (table default.Table_5facf725e0fc44cda7d77f8511030fa5 [id=892696e8040d4f2db8620d1d4ac90e0f])
I20250901 01:52:22.904567 1150 tablet_bootstrap.cc:492] T f13b972e25214f168421439fe3943e8a P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:22.904770 1150 ts_tablet_manager.cc:1397] T f13b972e25214f168421439fe3943e8a P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.076s user 0.002s sys 0.008s
I20250901 01:52:22.905908 1150 raft_consensus.cc:357] T f13b972e25214f168421439fe3943e8a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.906296 1150 raft_consensus.cc:383] T f13b972e25214f168421439fe3943e8a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:22.906402 1150 raft_consensus.cc:738] T f13b972e25214f168421439fe3943e8a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:22.906708 1150 consensus_queue.cc:260] T f13b972e25214f168421439fe3943e8a P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.907989 1150 ts_tablet_manager.cc:1428] T f13b972e25214f168421439fe3943e8a P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250901 01:52:22.909962 917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75e7a90815e449c39f386901a504143c" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:22.910149 917 raft_consensus.cc:2466] T 75e7a90815e449c39f386901a504143c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:22.910296 1150 tablet_bootstrap.cc:492] T 500e3c33598b45a4afa983da7cb0845a P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:22.910352 919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3c1fd0820d8a4995abe027f685da7f7b" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:22.910494 919 raft_consensus.cc:2466] T 3c1fd0820d8a4995abe027f685da7f7b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:22.911353 1055 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aec951f2cc84499b9162b9fa24d5440e" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:22.912523 1150 tablet_bootstrap.cc:654] T 500e3c33598b45a4afa983da7cb0845a P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:22.912992 992 leader_election.cc:336] T 75e7a90815e449c39f386901a504143c P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:22.915071 1910 tablet_loader.cc:96] loaded metadata for tablet 3dae00aa38204ee3ac77325eb334be42 (table default.Table_faf7ef12324f454396e5ae64d64ebe51 [id=5812902df9ea406ba59ff0ff23f130b6])
W20250901 01:52:22.914847 992 leader_election.cc:336] T 3c1fd0820d8a4995abe027f685da7f7b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:22.916288 859 leader_election.cc:343] T aec951f2cc84499b9162b9fa24d5440e P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:22.916433 859 leader_election.cc:304] T aec951f2cc84499b9162b9fa24d5440e P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:22.916663 1999 raft_consensus.cc:2747] T aec951f2cc84499b9162b9fa24d5440e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:22.918159 1910 tablet_loader.cc:96] loaded metadata for tablet 3eba54e5ddd14c518f4cbd539268ae37 (table default.Table_df46d9604f6a4a19b4d7c3424f48d90a [id=d28ebc8fa6d745919bf025a5a9f6ebca])
I20250901 01:52:22.917861 992 leader_election.cc:304] T 75e7a90815e449c39f386901a504143c P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:22.919013 1990 raft_consensus.cc:2802] T 75e7a90815e449c39f386901a504143c P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:22.919286 1990 raft_consensus.cc:491] T 75e7a90815e449c39f386901a504143c P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:22.919549 1990 raft_consensus.cc:3058] T 75e7a90815e449c39f386901a504143c P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:22.919145 1910 tablet_loader.cc:96] loaded metadata for tablet 410d44e5f1974ed69e5dee65e98856a4 (table default.Table_bec114a616504798bede9a9066e57192 [id=ae79f2c40fd7483ea400a36cf3c48888])
I20250901 01:52:22.920146 1910 tablet_loader.cc:96] loaded metadata for tablet 42c1156e8dd84c6fa1396b46de13eb6f (table default.Table_b5d681f51fc84395bbabe1e5b4cfb358 [id=40d05d90b0354334840ad17764f5b6bb])
I20250901 01:52:22.921528 1910 tablet_loader.cc:96] loaded metadata for tablet 435b1cca5a7d4d49959a310cb250b2a0 (table default.Table_dc9c5d18c7ec478592a76dbd76424b88 [id=126b298fef6c4653952d672adff1db7e])
I20250901 01:52:22.922441 1150 tablet_bootstrap.cc:492] T 500e3c33598b45a4afa983da7cb0845a P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:22.922657 1150 ts_tablet_manager.cc:1397] T 500e3c33598b45a4afa983da7cb0845a P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.003s
I20250901 01:52:22.922178 992 leader_election.cc:304] T 3c1fd0820d8a4995abe027f685da7f7b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:22.923569 1761 raft_consensus.cc:2802] T 3c1fd0820d8a4995abe027f685da7f7b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:22.923782 1761 raft_consensus.cc:491] T 3c1fd0820d8a4995abe027f685da7f7b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:22.923996 1761 raft_consensus.cc:3058] T 3c1fd0820d8a4995abe027f685da7f7b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:22.923640 1150 raft_consensus.cc:357] T 500e3c33598b45a4afa983da7cb0845a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.925061 1150 raft_consensus.cc:383] T 500e3c33598b45a4afa983da7cb0845a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:22.925179 1150 raft_consensus.cc:738] T 500e3c33598b45a4afa983da7cb0845a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:22.925469 1150 consensus_queue.cc:260] T 500e3c33598b45a4afa983da7cb0845a P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.932267 1987 consensus_queue.cc:1035] T ddbc15685fca4c708329420109ad0c57 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:22.933888 1150 ts_tablet_manager.cc:1428] T 500e3c33598b45a4afa983da7cb0845a P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20250901 01:52:22.934747 1150 tablet_bootstrap.cc:492] T e87d1281f35648b0b115b67c95e244e7 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:22.936825 1150 tablet_bootstrap.cc:654] T e87d1281f35648b0b115b67c95e244e7 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:22.937367 1151 tablet_bootstrap.cc:492] T b6daeff0070a4db08e90020752d21c29 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:22.937389 1981 raft_consensus.cc:491] T 81c36693199945d98465579b509852e2 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:22.937561 1151 ts_tablet_manager.cc:1397] T b6daeff0070a4db08e90020752d21c29 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.073s user 0.003s sys 0.022s
I20250901 01:52:22.937772 1981 raft_consensus.cc:513] T 81c36693199945d98465579b509852e2 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.938144 1910 tablet_loader.cc:96] loaded metadata for tablet 438ae684af234c868a419fbab43b0b98 (table default.Table_95515169b000454aa95025573560278f [id=1442b92adbbe43a6b8580cce1bbfe389])
W20250901 01:52:22.939596 858 leader_election.cc:336] T 81c36693199945d98465579b509852e2 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:22.939785 1981 leader_election.cc:290] T 81c36693199945d98465579b509852e2 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:22.943079 1922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250901 01:52:22.943477 1922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250901 01:52:22.943612 1922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250901 01:52:22.944011 1910 tablet_loader.cc:96] loaded metadata for tablet 4410da9e95454ecf95faf80221b0a71c (table default.Table_420b0979d6fd42da9b320456f4ccc9ce [id=d2004563c4374790954327f05c305cc6])
I20250901 01:52:22.944885 1910 tablet_loader.cc:96] loaded metadata for tablet 44af4764d4f64e44aec84ccb84be6f5f (table default.Table_ac189f6a9fcf457e9fa0d77badc81594 [id=a541b55be0dc420a8353ef4793b925b5])
I20250901 01:52:22.945732 1055 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "81c36693199945d98465579b509852e2" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:22.946105 1910 tablet_loader.cc:96] loaded metadata for tablet 45f6e8ba99b74898a2922b36768c6c36 (table default.Table_e533a2595cc9420f9ab7a36bda6dc4a3 [id=9ce26dcf307149b595a43b6ac4562fba])
I20250901 01:52:22.946254 1761 raft_consensus.cc:513] T 3c1fd0820d8a4995abe027f685da7f7b P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.947209 1981 raft_consensus.cc:491] T 3618a3eb4f174c43805ce7dc0e5315cb P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:22.948104 1761 leader_election.cc:290] T 3c1fd0820d8a4995abe027f685da7f7b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:22.948469 1910 tablet_loader.cc:96] loaded metadata for tablet 482a9a2cb81f448bb2823892e35920d4 (table default.Table_c298ecfa989347b0b7a4d7227611539f [id=b94e5ce799ee4046982550aace3a08d6])
I20250901 01:52:22.948745 1761 raft_consensus.cc:491] T 72ba689f9230411995dc96e2b45601df P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:22.949018 1992 raft_consensus.cc:491] T e25923b2daf9429683b2d44a75a56189 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:22.951125 1992 raft_consensus.cc:513] T e25923b2daf9429683b2d44a75a56189 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.951745 1992 leader_election.cc:290] T e25923b2daf9429683b2d44a75a56189 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:22.948920 1761 raft_consensus.cc:513] T 72ba689f9230411995dc96e2b45601df P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.953298 1761 leader_election.cc:290] T 72ba689f9230411995dc96e2b45601df P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:22.955186 992 consensus_peers.cc:489] T ddbc15685fca4c708329420109ad0c57 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:22.947335 1981 raft_consensus.cc:513] T 3618a3eb4f174c43805ce7dc0e5315cb P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:22.948734 859 leader_election.cc:343] T 81c36693199945d98465579b509852e2 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:22.955811 859 leader_election.cc:304] T 81c36693199945d98465579b509852e2 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:22.956146 1999 raft_consensus.cc:2747] T 81c36693199945d98465579b509852e2 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:22.956164 1981 leader_election.cc:290] T 3618a3eb4f174c43805ce7dc0e5315cb P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:22.954120 917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e25923b2daf9429683b2d44a75a56189" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:22.952859 919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3c1fd0820d8a4995abe027f685da7f7b" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:22.956439 917 raft_consensus.cc:2466] T e25923b2daf9429683b2d44a75a56189 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:22.956607 919 raft_consensus.cc:3058] T 3c1fd0820d8a4995abe027f685da7f7b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:22.957064 1990 raft_consensus.cc:513] T 75e7a90815e449c39f386901a504143c P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.957571 1990 leader_election.cc:290] T 75e7a90815e449c39f386901a504143c P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:22.955452 1151 raft_consensus.cc:357] T b6daeff0070a4db08e90020752d21c29 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:22.958029 858 leader_election.cc:336] T 3618a3eb4f174c43805ce7dc0e5315cb P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:22.958040 1151 raft_consensus.cc:383] T b6daeff0070a4db08e90020752d21c29 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:22.958182 1151 raft_consensus.cc:738] T b6daeff0070a4db08e90020752d21c29 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:22.958359 1910 tablet_loader.cc:96] loaded metadata for tablet 49aaaec611784e6bac131efd58dbca51 (table default.Table_95e1338a49a04f22b679e363e253262e [id=e24b91b4ac904ef0a8c4725f70bced2f])
I20250901 01:52:22.958694 1999 raft_consensus.cc:491] T e249474dcbf642189560c91ec3a99fdf P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:22.958612 1151 consensus_queue.cc:260] T b6daeff0070a4db08e90020752d21c29 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.958809 1999 raft_consensus.cc:513] T e249474dcbf642189560c91ec3a99fdf P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.959345 919 raft_consensus.cc:2466] T 3c1fd0820d8a4995abe027f685da7f7b P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:22.959394 1999 leader_election.cc:290] T e249474dcbf642189560c91ec3a99fdf P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:22.959528 1910 tablet_loader.cc:96] loaded metadata for tablet 4b4aa063ffe948e69e47f03102581899 (table default.Table_14e7f1e7dd314cdcaee09ac2fdf28993 [id=76e521ff309947a58840a7135e783477])
I20250901 01:52:22.959810 1151 ts_tablet_manager.cc:1428] T b6daeff0070a4db08e90020752d21c29 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.022s user 0.001s sys 0.002s
W20250901 01:52:22.961241 858 leader_election.cc:336] T e249474dcbf642189560c91ec3a99fdf P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:22.961930 1990 consensus_queue.cc:1035] T f62d96c37cca4143a4b098b7c69979e6 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:22.962119 1151 tablet_bootstrap.cc:492] T 0cb3262973934c8689b851adad60afe0 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:22.963287 992 leader_election.cc:304] T e25923b2daf9429683b2d44a75a56189 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:22.964267 1990 raft_consensus.cc:2802] T e25923b2daf9429683b2d44a75a56189 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:22.964468 1990 raft_consensus.cc:491] T e25923b2daf9429683b2d44a75a56189 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:22.964785 1990 raft_consensus.cc:3058] T e25923b2daf9429683b2d44a75a56189 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:22.968031 992 leader_election.cc:304] T 3c1fd0820d8a4995abe027f685da7f7b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:22.972380 919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "72ba689f9230411995dc96e2b45601df" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:22.972609 919 raft_consensus.cc:2466] T 72ba689f9230411995dc96e2b45601df P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:22.972609 1761 raft_consensus.cc:2802] T 3c1fd0820d8a4995abe027f685da7f7b P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:22.972779 1761 raft_consensus.cc:695] T 3c1fd0820d8a4995abe027f685da7f7b P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:22.973080 1761 consensus_queue.cc:237] T 3c1fd0820d8a4995abe027f685da7f7b P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:22.974355 1922 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250901 01:52:22.974526 1922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250901 01:52:22.974650 1922 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250901 01:52:22.974769 1922 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250901 01:52:22.975299 919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75e7a90815e449c39f386901a504143c" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:22.975481 919 raft_consensus.cc:3058] T 75e7a90815e449c39f386901a504143c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:22.978232 919 raft_consensus.cc:2466] T 75e7a90815e449c39f386901a504143c P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:22.978791 1992 raft_consensus.cc:491] T b6daeff0070a4db08e90020752d21c29 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:22.978968 1992 raft_consensus.cc:513] T b6daeff0070a4db08e90020752d21c29 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.979501 1992 leader_election.cc:290] T b6daeff0070a4db08e90020752d21c29 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:22.980005 1055 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3618a3eb4f174c43805ce7dc0e5315cb" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:22.980746 1055 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e249474dcbf642189560c91ec3a99fdf" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:22.980082 1990 raft_consensus.cc:513] T e25923b2daf9429683b2d44a75a56189 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:22.981567 992 leader_election.cc:336] T 3c1fd0820d8a4995abe027f685da7f7b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:22.983630 1151 tablet_bootstrap.cc:654] T 0cb3262973934c8689b851adad60afe0 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:22.984076 992 leader_election.cc:336] T e25923b2daf9429683b2d44a75a56189 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:22.984802 1990 leader_election.cc:290] T e25923b2daf9429683b2d44a75a56189 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:22.960265 1910 tablet_loader.cc:96] loaded metadata for tablet 4cde5f4f02044b7bbd7d43b49aac4ae8 (table default.Table_c931e9dfbef04fd3b49aa4f43db3d981 [id=77ae3376281349e087a3b3cee41f2a09])
I20250901 01:52:22.986939 1150 tablet_bootstrap.cc:492] T e87d1281f35648b0b115b67c95e244e7 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:22.987064 1150 ts_tablet_manager.cc:1397] T e87d1281f35648b0b115b67c95e244e7 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.052s user 0.003s sys 0.000s
I20250901 01:52:22.987892 1910 tablet_loader.cc:96] loaded metadata for tablet 4f4be080f20d41c798835b7111a5318d (table default.Table_1a83c4b1e2734d038d0309401d0ab08e [id=c1902e6ac21b4a8a8af96d647f85e288])
I20250901 01:52:22.987936 1150 raft_consensus.cc:357] T e87d1281f35648b0b115b67c95e244e7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.988134 1150 raft_consensus.cc:383] T e87d1281f35648b0b115b67c95e244e7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:22.988210 1150 raft_consensus.cc:738] T e87d1281f35648b0b115b67c95e244e7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:22.988444 1150 consensus_queue.cc:260] T e87d1281f35648b0b115b67c95e244e7 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:22.989387 1150 ts_tablet_manager.cc:1428] T e87d1281f35648b0b115b67c95e244e7 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250901 01:52:22.989831 1910 tablet_loader.cc:96] loaded metadata for tablet 4fd0e4f9d86c4362af96ce9964b72328 (table default.Table_2d3d2227e6a24bd2a86f457e553e698b [id=f48fd53522234290b6c0373fd7b02570])
I20250901 01:52:22.990250 1150 tablet_bootstrap.cc:492] T f9b293bf6a17453b9ea9892a6970d0b0 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:22.991832 1150 tablet_bootstrap.cc:654] T f9b293bf6a17453b9ea9892a6970d0b0 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:22.995647 859 leader_election.cc:343] T 3618a3eb4f174c43805ce7dc0e5315cb P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:22.995777 859 leader_election.cc:304] T 3618a3eb4f174c43805ce7dc0e5315cb P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:22.996007 1910 tablet_loader.cc:96] loaded metadata for tablet 500e3c33598b45a4afa983da7cb0845a (table default.Table_8ac222c93e36436c9a3ecb2c489f6d74 [id=44b75816b8b24af0a9530d4a12c2416e])
I20250901 01:52:22.996170 1999 raft_consensus.cc:2747] T 3618a3eb4f174c43805ce7dc0e5315cb P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:22.996243 917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b6daeff0070a4db08e90020752d21c29" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:22.996613 917 raft_consensus.cc:2466] T b6daeff0070a4db08e90020752d21c29 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:22.997165 992 leader_election.cc:304] T 72ba689f9230411995dc96e2b45601df P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:22.997615 1910 tablet_loader.cc:96] loaded metadata for tablet 508406601c994e42a5c5b1f665e74957 (table default.Table_df96f1aa51644aed967bdfd4f75588f6 [id=4aabc328f6374ec5b1dea26f778208b2])
I20250901 01:52:22.998178 1990 raft_consensus.cc:2802] T 72ba689f9230411995dc96e2b45601df P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:22.998401 1990 raft_consensus.cc:491] T 72ba689f9230411995dc96e2b45601df P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:22.998602 1990 raft_consensus.cc:3058] T 72ba689f9230411995dc96e2b45601df P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:22.998868 1910 tablet_loader.cc:96] loaded metadata for tablet 55bd2daa07294358999956cb553d973f (table default.Table_c714bf77d32b42aeb225bbc5b9a6f43b [id=4c36dd7be4ae4312a19164ef7ef3c6e5])
I20250901 01:52:22.999240 1910 tablet_loader.cc:96] loaded metadata for tablet 56475ba491a24301a534304811f30140 (table default.Table_b0e916c76a7a4766aa0887cc8691488f [id=cf61137a852c48748a29f833f1860da7])
I20250901 01:52:23.000895 1910 tablet_loader.cc:96] loaded metadata for tablet 5adf6b492640417ca071c94702b48cb9 (table default.Table_589c302e02474fc3bc5d3a290a8bd54a [id=5d14706774d345c48ca3575feabb17b0])
I20250901 01:52:23.001327 1910 tablet_loader.cc:96] loaded metadata for tablet 5afae316ff164413be0d25d4bc0f58dc (table default.Table_1da54cdf1dfd4c0fb5f2538e26f17ad9 [id=8a1a77329ab740dda2d6a5d78bbff906])
I20250901 01:52:23.001998 1150 tablet_bootstrap.cc:492] T f9b293bf6a17453b9ea9892a6970d0b0 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:23.002108 1910 tablet_loader.cc:96] loaded metadata for tablet 5d68ea97c71942c7b55b967ba870e0d7 (table default.Table_1d63a9f86fe4488b92d5d607b102328c [id=36915a3e843340ba8ef3578ac35a9af4])
I20250901 01:52:23.002261 1150 ts_tablet_manager.cc:1397] T f9b293bf6a17453b9ea9892a6970d0b0 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.003s
I20250901 01:52:23.002485 1910 tablet_loader.cc:96] loaded metadata for tablet 5d794471dbed4477b9d52fd411fa4d10 (table default.Table_dc856980ff4c4aafb92cf1fd4f29e694 [id=917ac6e5eaf54d019896a1d1ae2f6622])
I20250901 01:52:23.002888 1910 tablet_loader.cc:96] loaded metadata for tablet 5dd0c6b806574caf9126dad94833fcf7 (table default.Table_3dde393d3a564e84b8ef08c0d25265ce [id=649558231b1542b5bc2d90699a644b2a])
I20250901 01:52:23.003014 1761 raft_consensus.cc:491] T d77aadd7291647a0bae7c421f244c252 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:23.003273 1910 tablet_loader.cc:96] loaded metadata for tablet 5ddbc7e6820247a987f4d54757d1191d (table default.Table_ca5bada2f51949d790e9a38e25fa1bac [id=4ae8785dc52c4e89a1a97a1490356898])
I20250901 01:52:23.003656 1910 tablet_loader.cc:96] loaded metadata for tablet 5e2472ee5d6e4d3bbf1f24027ba531ca (table default.Table_52733c9a363b4ed9b6ff11ad8571e421 [id=453153553bea48319e132857f628d48a])
I20250901 01:52:23.004036 1910 tablet_loader.cc:96] loaded metadata for tablet 5f52698fb56d472d9e91fb10cb10c69a (table default.Table_83cf2d86a1284544ba2e7aa9ebb66bb7 [id=264d1a223e7d43c8a7a661db6722cb17])
I20250901 01:52:23.004487 1910 tablet_loader.cc:96] loaded metadata for tablet 5f5396045979450193d461ba1334a5c9 (table default.Table_a89cc8e48563460baabc34c821ceae35 [id=9c3930e9fcfe4b49ba4bea67a2b0ad07])
I20250901 01:52:23.004431 1761 raft_consensus.cc:513] T d77aadd7291647a0bae7c421f244c252 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.005048 1761 leader_election.cc:290] T d77aadd7291647a0bae7c421f244c252 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:23.005795 992 leader_election.cc:304] T 75e7a90815e449c39f386901a504143c P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:23.006711 1761 raft_consensus.cc:2802] T 75e7a90815e449c39f386901a504143c P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:23.007213 1761 raft_consensus.cc:695] T 75e7a90815e449c39f386901a504143c P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:23.007527 1761 consensus_queue.cc:237] T 75e7a90815e449c39f386901a504143c P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.009150 1151 tablet_bootstrap.cc:492] T 0cb3262973934c8689b851adad60afe0 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:23.009487 1151 ts_tablet_manager.cc:1397] T 0cb3262973934c8689b851adad60afe0 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.048s user 0.001s sys 0.013s
I20250901 01:52:23.008489 1150 raft_consensus.cc:357] T f9b293bf6a17453b9ea9892a6970d0b0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.010016 1150 raft_consensus.cc:383] T f9b293bf6a17453b9ea9892a6970d0b0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:23.010067 992 leader_election.cc:304] T b6daeff0070a4db08e90020752d21c29 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:23.010963 1151 raft_consensus.cc:357] T 0cb3262973934c8689b851adad60afe0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.010591 1150 raft_consensus.cc:738] T f9b293bf6a17453b9ea9892a6970d0b0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:23.011359 1992 raft_consensus.cc:2802] T b6daeff0070a4db08e90020752d21c29 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:23.011767 1992 raft_consensus.cc:491] T b6daeff0070a4db08e90020752d21c29 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:23.011888 1992 raft_consensus.cc:3058] T b6daeff0070a4db08e90020752d21c29 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
W20250901 01:52:23.012506 859 leader_election.cc:343] T e249474dcbf642189560c91ec3a99fdf P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:23.012643 859 leader_election.cc:304] T e249474dcbf642189560c91ec3a99fdf P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:23.012987 1999 raft_consensus.cc:491] T 500e3c33598b45a4afa983da7cb0845a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:23.013178 1981 raft_consensus.cc:2747] T e249474dcbf642189560c91ec3a99fdf P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:23.013126 1999 raft_consensus.cc:513] T 500e3c33598b45a4afa983da7cb0845a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.014864 1922 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:46333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/master-0/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:43871,127.0.104.61:42375,127.0.104.60:44231
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:43871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=34679
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type FASTDEBUG
built by None at 01 Sep 2025 01:43:20 UTC on 24a791456cd2
build id 7840
ASAN enabled
W20250901 01:52:23.015537 858 leader_election.cc:336] T 500e3c33598b45a4afa983da7cb0845a P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:23.013764 1150 consensus_queue.cc:260] T f9b293bf6a17453b9ea9892a6970d0b0 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.017495 1922 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 01:52:23.017645 1910 tablet_loader.cc:96] loaded metadata for tablet 606fac5c6efe48d1877104b01c23ba5b (table default.Table_7c12d66048a6470e8c227dc1d9f45fa4 [id=91472f220fe44297b79d3eec8b9b80e8])
I20250901 01:52:23.018103 917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e25923b2daf9429683b2d44a75a56189" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:23.018275 917 raft_consensus.cc:3058] T e25923b2daf9429683b2d44a75a56189 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:23.019097 1910 tablet_loader.cc:96] loaded metadata for tablet 615b2306fca84003b264da9d5f95ddab (table default.Table_0f959237963c4e33932e193285c0eacb [id=cecdd8d6ca6d445c9fd82decd2a95366])
I20250901 01:52:23.019466 1910 tablet_loader.cc:96] loaded metadata for tablet 616ff34e8d314e0aa0e11ce60be11789 (table default.Table_0451d0afa26f4d5f8d4214a946eea04f [id=5ee13d46e06a4fa59b1070547573e523])
I20250901 01:52:23.019822 1910 tablet_loader.cc:96] loaded metadata for tablet 617cc33d0d23439cb5f1e0c31bf11281 (table default.Table_ce1fd26ac65a4ed085d023e10a849e5c [id=fb088d31bba64f04871fbfd355d6263b])
I20250901 01:52:23.020268 1910 tablet_loader.cc:96] loaded metadata for tablet 61d8f4252767490f842fb28f3edc1a96 (table default.Table_3940656768d5406fb0e10a71f8f77003 [id=4630ed8abaea4b5195902aa51645d30f])
I20250901 01:52:23.020630 1922 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250901 01:52:23.024792 1150 ts_tablet_manager.cc:1428] T f9b293bf6a17453b9ea9892a6970d0b0 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.022s user 0.001s sys 0.002s
W20250901 01:52:23.026309 992 leader_election.cc:336] T 72ba689f9230411995dc96e2b45601df P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:23.027558 1992 raft_consensus.cc:513] T b6daeff0070a4db08e90020752d21c29 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.028391 1150 tablet_bootstrap.cc:492] T f3962b471bdb49aea6e9fa95f1f9dfe2 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:23.029119 917 raft_consensus.cc:2466] T e25923b2daf9429683b2d44a75a56189 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:23.014168 1999 leader_election.cc:290] T 500e3c33598b45a4afa983da7cb0845a P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:23.028375 1992 leader_election.cc:290] T b6daeff0070a4db08e90020752d21c29 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:23.029873 992 leader_election.cc:336] T 75e7a90815e449c39f386901a504143c P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:23.031401 1990 raft_consensus.cc:513] T 72ba689f9230411995dc96e2b45601df P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.037994 1990 leader_election.cc:290] T 72ba689f9230411995dc96e2b45601df P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:23.039227 992 leader_election.cc:336] T b6daeff0070a4db08e90020752d21c29 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:23.041518 1150 tablet_bootstrap.cc:654] T f3962b471bdb49aea6e9fa95f1f9dfe2 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:23.041884 921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b6daeff0070a4db08e90020752d21c29" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:23.042096 921 raft_consensus.cc:3058] T b6daeff0070a4db08e90020752d21c29 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
W20250901 01:52:23.041826 992 leader_election.cc:336] T e25923b2daf9429683b2d44a75a56189 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:23.044426 1910 tablet_loader.cc:96] loaded metadata for tablet 61ddcf86c9f34726939cc116434f0822 (table default.Table_277b91103c4d41d2a9f9424f99082b29 [id=f9dcf4937b1143248bfa9d88053fce17])
I20250901 01:52:23.044621 1055 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "500e3c33598b45a4afa983da7cb0845a" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:23.044860 1910 tablet_loader.cc:96] loaded metadata for tablet 629db80a8ab444e582628578155638fd (table default.Table_f0e56dba1e8a4ed49a760e204c1ff96e [id=c3865e058daa4cea9a89f5046fca712a])
I20250901 01:52:23.045306 1910 tablet_loader.cc:96] loaded metadata for tablet 641a83a3e50d451189c33c7b7b4ca070 (table default.Table_fc0336a00d084f6e98e1d8d32142e66f [id=0e69044796ea4292b1dc100db2f26cee])
I20250901 01:52:23.045193 1990 consensus_queue.cc:1035] T ded33e0950b544509af11739353c620d P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:23.046108 1910 tablet_loader.cc:96] loaded metadata for tablet 669d26c6551c4fc58283b67a501d9bf1 (table default.Table_1a9de416bd864f7ba7ed302579ebda5a [id=4d54f43f814d48f79124714c187ea383])
I20250901 01:52:23.046499 992 leader_election.cc:304] T e25923b2daf9429683b2d44a75a56189 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:23.047442 1910 tablet_loader.cc:96] loaded metadata for tablet 6789505328b1409ea288e8b4eaf94935 (table default.Table_657603f2bb3c43bc85028218860da978 [id=013cf6104d1a47e592f90f6ba89653aa])
I20250901 01:52:23.047585 1990 raft_consensus.cc:2802] T e25923b2daf9429683b2d44a75a56189 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:23.048014 1990 raft_consensus.cc:695] T e25923b2daf9429683b2d44a75a56189 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:23.048228 1910 tablet_loader.cc:96] loaded metadata for tablet 69f61ec283484f9d8b3eebcea27f741b (table default.Table_583e53c2e86544e9be68eb27ab6068a3 [id=f376eaf65bb549429aa71b03fca81647])
I20250901 01:52:23.048533 1999 raft_consensus.cc:491] T c754129977904f978f87fd89d7092e36 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:23.048432 1990 consensus_queue.cc:237] T e25923b2daf9429683b2d44a75a56189 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.048681 1910 tablet_loader.cc:96] loaded metadata for tablet 6ba8241644984394ada95b561e76467e (table default.Table_a96f01d4ce3c4694a635575ffaa4f02d [id=5dc1ab36f2974f07866b14fedbdabd57])
I20250901 01:52:23.048682 1999 raft_consensus.cc:513] T c754129977904f978f87fd89d7092e36 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.049016 1910 tablet_loader.cc:96] loaded metadata for tablet 6ba88af177874544b764cc32420e7ac9 (table default.Table_97c44ea002094df8a6af3dccd0099c39 [id=305a9d97cb394d61821c13807ebebc17])
I20250901 01:52:23.049552 1910 tablet_loader.cc:96] loaded metadata for tablet 6c05eff658924ca4a9a1e7a37f917d67 (table default.Table_c1959bda12f14bc9995d832476abf670 [id=17355a61629849dd9d8228a469ab744f])
I20250901 01:52:23.049901 1999 leader_election.cc:290] T c754129977904f978f87fd89d7092e36 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:23.049997 1910 tablet_loader.cc:96] loaded metadata for tablet 6c0abbfc8d904dce989ce51961e4237d (table default.Table_3f9422b08b0349b9a5466218f0fe1b06 [id=f0dfb3811e744bd4ba6d59195cebde6f])
W20250901 01:52:23.051086 858 leader_election.cc:336] T c754129977904f978f87fd89d7092e36 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:23.051852 1910 tablet_loader.cc:96] loaded metadata for tablet 71941b815d6840478c5acc1cbb187cc9 (table default.Table_f027ccc695a74d1fb4595d9614f30381 [id=3ac80ef18f1044c0b22018ae09997fd3])
I20250901 01:52:23.052249 1910 tablet_loader.cc:96] loaded metadata for tablet 72ba689f9230411995dc96e2b45601df (table default.Table_604f2c5c58d74154bacf2a065a467f32 [id=351a05134f4948039c920cafbe050914])
I20250901 01:52:23.052769 1910 tablet_loader.cc:96] loaded metadata for tablet 7304937048ca48a7bd153f3cdcfaeff0 (table default.Table_7dd48000e9b24337b75ceb4c53b6be25 [id=8051adf4041f4a2686a0b8e3850ef29c])
I20250901 01:52:23.053331 1910 tablet_loader.cc:96] loaded metadata for tablet 7442c6ff699141e1aff208ebdc9d192a (table default.Table_d61fc9783e7b45e49afb36280adcbd86 [id=0bc0d862df03433dacaf6af87a8390cd])
I20250901 01:52:23.053617 1761 consensus_queue.cc:1035] T c763d252598745a088344aa0eb5fa3d3 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250901 01:52:23.053692 2021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 01:52:23.057016 918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "72ba689f9230411995dc96e2b45601df" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:23.057191 918 raft_consensus.cc:3058] T 72ba689f9230411995dc96e2b45601df P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:23.057602 921 raft_consensus.cc:2466] T b6daeff0070a4db08e90020752d21c29 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:23.058017 917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d77aadd7291647a0bae7c421f244c252" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:23.058272 917 raft_consensus.cc:2466] T d77aadd7291647a0bae7c421f244c252 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:23.060042 918 raft_consensus.cc:2466] T 72ba689f9230411995dc96e2b45601df P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
W20250901 01:52:23.060618 859 leader_election.cc:343] T 500e3c33598b45a4afa983da7cb0845a P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:23.060763 859 leader_election.cc:304] T 500e3c33598b45a4afa983da7cb0845a P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:23.061137 1999 raft_consensus.cc:2747] T 500e3c33598b45a4afa983da7cb0845a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:23.066232 1055 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c754129977904f978f87fd89d7092e36" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:23.066790 1151 raft_consensus.cc:383] T 0cb3262973934c8689b851adad60afe0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:23.067399 1151 raft_consensus.cc:738] T 0cb3262973934c8689b851adad60afe0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:23.068158 1151 consensus_queue.cc:260] T 0cb3262973934c8689b851adad60afe0 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:23.068518 2020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 01:52:23.068864 1151 ts_tablet_manager.cc:1428] T 0cb3262973934c8689b851adad60afe0 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.059s user 0.000s sys 0.002s
I20250901 01:52:23.069166 1151 tablet_bootstrap.cc:492] T c49d1143d54b4c319739488906b0e271 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:23.070832 1151 tablet_bootstrap.cc:654] T c49d1143d54b4c319739488906b0e271 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:23.071807 992 leader_election.cc:336] T b6daeff0070a4db08e90020752d21c29 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:23.072274 992 leader_election.cc:336] T d77aadd7291647a0bae7c421f244c252 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:23.072508 992 consensus_peers.cc:489] T f62d96c37cca4143a4b098b7c69979e6 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:23.080163 1151 tablet_bootstrap.cc:492] T c49d1143d54b4c319739488906b0e271 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:23.081003 1151 ts_tablet_manager.cc:1397] T c49d1143d54b4c319739488906b0e271 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.003s
I20250901 01:52:23.080950 992 leader_election.cc:304] T b6daeff0070a4db08e90020752d21c29 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:23.083036 1910 tablet_loader.cc:96] loaded metadata for tablet 756cfe84a517459ca015aecf5cd81571 (table default.Table_b3ec3b285fac447f8c5c4a9ac4196628 [id=f49c96ff2eaa44cc9b754225cb9e6fdf])
I20250901 01:52:23.083051 992 leader_election.cc:304] T d77aadd7291647a0bae7c421f244c252 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:23.083588 1910 tablet_loader.cc:96] loaded metadata for tablet 75a703c8e20a4a2fa0ab6ec58d41ef92 (table default.Table_2f09538b6fbe401f909105decb56e1cb [id=a90d1f10531042c3bb7a196b8be61c5f])
I20250901 01:52:23.083693 1761 raft_consensus.cc:2802] T b6daeff0070a4db08e90020752d21c29 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:23.083845 1761 raft_consensus.cc:695] T b6daeff0070a4db08e90020752d21c29 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:23.084131 1761 consensus_queue.cc:237] T b6daeff0070a4db08e90020752d21c29 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.085209 1872 raft_consensus.cc:2802] T d77aadd7291647a0bae7c421f244c252 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:23.085363 1872 raft_consensus.cc:491] T d77aadd7291647a0bae7c421f244c252 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:23.085460 1872 raft_consensus.cc:3058] T d77aadd7291647a0bae7c421f244c252 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:23.090739 1872 raft_consensus.cc:513] T d77aadd7291647a0bae7c421f244c252 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.092229 1872 leader_election.cc:290] T d77aadd7291647a0bae7c421f244c252 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:23.095736 1999 raft_consensus.cc:491] T 25ce0fe8c9544f828694184d19d13df9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:23.095892 1999 raft_consensus.cc:513] T 25ce0fe8c9544f828694184d19d13df9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:23.096558 1999 leader_election.cc:290] T 25ce0fe8c9544f828694184d19d13df9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:23.096827 1150 tablet_bootstrap.cc:492] T f3962b471bdb49aea6e9fa95f1f9dfe2 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:23.096974 1150 ts_tablet_manager.cc:1397] T f3962b471bdb49aea6e9fa95f1f9dfe2 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.070s user 0.003s sys 0.004s
I20250901 01:52:23.097072 1981 raft_consensus.cc:491] T 6cf09e8f299548dba2cce16d7b565bad P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:23.097213 1981 raft_consensus.cc:513] T 6cf09e8f299548dba2cce16d7b565bad P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:23.097697 1150 raft_consensus.cc:357] T f3962b471bdb49aea6e9fa95f1f9dfe2 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:23.097838 1150 raft_consensus.cc:383] T f3962b471bdb49aea6e9fa95f1f9dfe2 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:23.097898 1150 raft_consensus.cc:738] T f3962b471bdb49aea6e9fa95f1f9dfe2 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:23.098069 1150 consensus_queue.cc:260] T f3962b471bdb49aea6e9fa95f1f9dfe2 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:23.097846 1981 leader_election.cc:290] T 6cf09e8f299548dba2cce16d7b565bad P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:23.098285 1711 raft_consensus.cc:491] T 17e8340dbd1a406cb782dad3fe88cc7e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:23.098515 1150 ts_tablet_manager.cc:1428] T f3962b471bdb49aea6e9fa95f1f9dfe2 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250901 01:52:23.098605 1711 raft_consensus.cc:513] T 17e8340dbd1a406cb782dad3fe88cc7e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.098816 1150 tablet_bootstrap.cc:492] T 7a607fac57994001b2304e5d81c9decc P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:23.099100 1711 leader_election.cc:290] T 17e8340dbd1a406cb782dad3fe88cc7e P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:23.099283 992 leader_election.cc:304] T 72ba689f9230411995dc96e2b45601df P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:23.099355 1151 raft_consensus.cc:357] T c49d1143d54b4c319739488906b0e271 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.100358 1150 tablet_bootstrap.cc:654] T 7a607fac57994001b2304e5d81c9decc P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:23.100468 1872 raft_consensus.cc:2802] T 72ba689f9230411995dc96e2b45601df P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:23.100637 1872 raft_consensus.cc:695] T 72ba689f9230411995dc96e2b45601df P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:23.100980 1872 consensus_queue.cc:237] T 72ba689f9230411995dc96e2b45601df P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:23.102224 858 leader_election.cc:336] T 6cf09e8f299548dba2cce16d7b565bad P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:23.102345 858 leader_election.cc:336] T 17e8340dbd1a406cb782dad3fe88cc7e P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:23.102430 858 leader_election.cc:336] T 25ce0fe8c9544f828694184d19d13df9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:23.102514 2030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 01:52:23.103186 1151 raft_consensus.cc:383] T c49d1143d54b4c319739488906b0e271 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:23.103837 1151 raft_consensus.cc:738] T c49d1143d54b4c319739488906b0e271 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:23.105683 1910 tablet_loader.cc:96] loaded metadata for tablet 764bfeb2a7d14c40815b5ec1b07bdd62 (table default.Table_4401d83f75f8464da49d5b811d7dd89d [id=16143c8e6f484f9a873437e709d8466d])
I20250901 01:52:23.107650 1151 consensus_queue.cc:260] T c49d1143d54b4c319739488906b0e271 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:23.110482 859 leader_election.cc:343] T c754129977904f978f87fd89d7092e36 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:23.110613 859 leader_election.cc:304] T c754129977904f978f87fd89d7092e36 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:23.111003 1999 raft_consensus.cc:2747] T c754129977904f978f87fd89d7092e36 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:23.113512 1910 tablet_loader.cc:96] loaded metadata for tablet 7954393c8c1c4457ab8bb723c03a9c8b (table default.Table_5ea919f20c6b4a4e91e599357a17a53a [id=543f2074598d4c29972e564be1123fca])
I20250901 01:52:23.114253 1151 ts_tablet_manager.cc:1428] T c49d1143d54b4c319739488906b0e271 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.033s user 0.001s sys 0.002s
I20250901 01:52:23.118721 1910 tablet_loader.cc:96] loaded metadata for tablet 7b16ca1daf394a6da65a4af8da82ec33 (table default.Table_9d8c1fb24bc24631a2ba2b34a395f24a [id=2d12d6d6275e499ba3928e76bfd3cfc7])
I20250901 01:52:23.120923 1910 tablet_loader.cc:96] loaded metadata for tablet 7cc1efcb6bff4aba8015d2d3b7f4c923 (table default.Table_5610feff8a344dc9be4bc1f0ad8ddd51 [id=78b00856a9b747c19f390b4fdae8ddcf])
I20250901 01:52:23.122932 1910 tablet_loader.cc:96] loaded metadata for tablet 7f2546a3f6da4b3297ebcbf006eeb3f2 (table default.Table_f8793d5bb6ca480491a9729761f3f61b [id=bc05464fd80c46a5b4ea19d08d7baf49])
I20250901 01:52:23.123366 1910 tablet_loader.cc:96] loaded metadata for tablet 7f46e9ddadaf4ff986d8c991178daa0c (table default.Table_15c9f4f02eb24474936a43d705f90f7b [id=ea928501fe954cd0b81d38add4ef7128])
W20250901 01:52:23.124126 992 leader_election.cc:336] T 72ba689f9230411995dc96e2b45601df P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:23.125425 1151 tablet_bootstrap.cc:492] T 8dd587304e6342edbe96c9897b40e5c9 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
W20250901 01:52:23.126523 992 consensus_peers.cc:489] T c763d252598745a088344aa0eb5fa3d3 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:23.125831 1910 tablet_loader.cc:96] loaded metadata for tablet 8151bad707c04ac687c5bddfea573e3f (table default.Table_bfb1749ab3454806b139ff43f672583a [id=1eca61e34b5640d7868d6eacd854a307])
I20250901 01:52:23.127406 1910 tablet_loader.cc:96] loaded metadata for tablet 81c36693199945d98465579b509852e2 (table default.Table_edd1006860a84e209149771e23bb8834 [id=662be9c7921d422f93307d06187b1cae])
I20250901 01:52:23.131896 917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d77aadd7291647a0bae7c421f244c252" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:23.132097 917 raft_consensus.cc:3058] T d77aadd7291647a0bae7c421f244c252 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:23.134080 1055 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25ce0fe8c9544f828694184d19d13df9" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:23.134717 917 raft_consensus.cc:2466] T d77aadd7291647a0bae7c421f244c252 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:23.134963 1055 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6cf09e8f299548dba2cce16d7b565bad" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:23.135874 1055 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "17e8340dbd1a406cb782dad3fe88cc7e" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:23.136204 1151 tablet_bootstrap.cc:654] T 8dd587304e6342edbe96c9897b40e5c9 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:23.137924 1999 raft_consensus.cc:491] T a8483e7212d94daba7e5401256c33aef P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:23.138051 1999 raft_consensus.cc:513] T a8483e7212d94daba7e5401256c33aef P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:23.138644 1999 leader_election.cc:290] T a8483e7212d94daba7e5401256c33aef P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:23.139724 858 leader_election.cc:336] T a8483e7212d94daba7e5401256c33aef P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:23.140144 1761 raft_consensus.cc:491] T fae2779fac164b9f8052c3666db803b8 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:23.140255 1761 raft_consensus.cc:513] T fae2779fac164b9f8052c3666db803b8 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.140774 1761 leader_election.cc:290] T fae2779fac164b9f8052c3666db803b8 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:23.141714 859 leader_election.cc:343] T 25ce0fe8c9544f828694184d19d13df9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:23.141819 859 leader_election.cc:304] T 25ce0fe8c9544f828694184d19d13df9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:23.142210 1150 tablet_bootstrap.cc:492] T 7a607fac57994001b2304e5d81c9decc P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:23.142289 1999 raft_consensus.cc:2747] T 25ce0fe8c9544f828694184d19d13df9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:23.142232 859 leader_election.cc:343] T 6cf09e8f299548dba2cce16d7b565bad P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:23.142441 1150 ts_tablet_manager.cc:1397] T 7a607fac57994001b2304e5d81c9decc P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.044s user 0.000s sys 0.011s
I20250901 01:52:23.142544 859 leader_election.cc:304] T 6cf09e8f299548dba2cce16d7b565bad P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:23.142900 859 leader_election.cc:343] T 17e8340dbd1a406cb782dad3fe88cc7e P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:23.142992 859 leader_election.cc:304] T 17e8340dbd1a406cb782dad3fe88cc7e P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:23.142901 992 leader_election.cc:304] T d77aadd7291647a0bae7c421f244c252 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:23.143424 1761 raft_consensus.cc:2802] T d77aadd7291647a0bae7c421f244c252 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:23.143445 1711 raft_consensus.cc:2747] T 6cf09e8f299548dba2cce16d7b565bad P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:23.143533 1761 raft_consensus.cc:695] T d77aadd7291647a0bae7c421f244c252 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:23.143575 1711 raft_consensus.cc:2747] T 17e8340dbd1a406cb782dad3fe88cc7e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:23.143572 1150 raft_consensus.cc:357] T 7a607fac57994001b2304e5d81c9decc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.143759 1761 consensus_queue.cc:237] T d77aadd7291647a0bae7c421f244c252 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.143841 1150 raft_consensus.cc:383] T 7a607fac57994001b2304e5d81c9decc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:23.143921 1150 raft_consensus.cc:738] T 7a607fac57994001b2304e5d81c9decc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:23.144969 1922 server_base.cc:1047] running on GCE node
I20250901 01:52:23.146342 1922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 01:52:23.147990 1910 tablet_loader.cc:96] loaded metadata for tablet 81ce448b148447ddbc41875d863e0518 (table default.Table_6e461d9cb0ae4dd095975bbc21533794 [id=49673fa73f8c436e9b8026efc4f3cbbc])
I20250901 01:52:23.149858 1999 raft_consensus.cc:491] T 89d40f0331114490b51de3ba6d773571 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:23.150012 1999 raft_consensus.cc:513] T 89d40f0331114490b51de3ba6d773571 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.150606 1999 leader_election.cc:290] T 89d40f0331114490b51de3ba6d773571 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:23.151367 1872 consensus_queue.cc:1035] T a6445b29591548be9f9577db1ba0bfa5 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:23.144228 1150 consensus_queue.cc:260] T 7a607fac57994001b2304e5d81c9decc P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:23.152558 858 leader_election.cc:336] T 89d40f0331114490b51de3ba6d773571 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:23.153966 1999 raft_consensus.cc:491] T f9b293bf6a17453b9ea9892a6970d0b0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:23.154093 1999 raft_consensus.cc:513] T f9b293bf6a17453b9ea9892a6970d0b0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.154381 1910 tablet_loader.cc:96] loaded metadata for tablet 85322a864d154f299f2d670fdd2a7bfb (table default.Table_5a72e7206f604795915f3a577d37f874 [id=1d8f34cc14cb4e90a562ff4424b8577a])
I20250901 01:52:23.154685 1999 leader_election.cc:290] T f9b293bf6a17453b9ea9892a6970d0b0 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:23.154791 1910 tablet_loader.cc:96] loaded metadata for tablet 858218c4427c41e980b2b5c969a5af40 (table default.Table_4185b4f73a2743e08f24a26878cf73ca [id=264cc469a73841e79538843479a165ab])
I20250901 01:52:23.155114 1910 tablet_loader.cc:96] loaded metadata for tablet 879fa66e55544118aafe504587393c66 (table default.Table_b362441111a241e19cc1a09bdeefea49 [id=f1d75be759db40ebb72ff32560f1d038])
I20250901 01:52:23.155475 1922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 01:52:23.156199 1910 tablet_loader.cc:96] loaded metadata for tablet 87e97a9544a2485897b1053c8929a39a (table default.Table_37dedd79eacd4c56ac546fe8b1902259 [id=3c4e9916282d46589939d5fe10592959])
W20250901 01:52:23.156248 858 leader_election.cc:336] T f9b293bf6a17453b9ea9892a6970d0b0 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:23.156344 1150 ts_tablet_manager.cc:1428] T 7a607fac57994001b2304e5d81c9decc P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.014s user 0.000s sys 0.002s
I20250901 01:52:23.157464 1910 tablet_loader.cc:96] loaded metadata for tablet 88ceba7f48b04d7885cc0164668700a5 (table default.Table_53e30205c5fd488eb3b48c685a159d04 [id=60837b865ee7415ea42a85a953d837a5])
I20250901 01:52:23.157582 1999 raft_consensus.cc:491] T f80f13db501c42c7950499df8fcb6fc8 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:23.157688 1999 raft_consensus.cc:513] T f80f13db501c42c7950499df8fcb6fc8 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.158154 1999 leader_election.cc:290] T f80f13db501c42c7950499df8fcb6fc8 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:23.158725 918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fae2779fac164b9f8052c3666db803b8" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:23.158892 918 raft_consensus.cc:2466] T fae2779fac164b9f8052c3666db803b8 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:23.158933 1150 tablet_bootstrap.cc:492] T 5e2472ee5d6e4d3bbf1f24027ba531ca P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:23.158970 1872 raft_consensus.cc:491] T 92f0ad59fbbb435ea872c669a52f854a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:23.159076 1872 raft_consensus.cc:513] T 92f0ad59fbbb435ea872c669a52f854a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.159031 1910 tablet_loader.cc:96] loaded metadata for tablet 89d40f0331114490b51de3ba6d773571 (table default.Table_4dacb3b1d2e94e42ac8937eb6eb58375 [id=8ee0aa1ba24042dd8ded004644913b9c])
I20250901 01:52:23.159613 1872 leader_election.cc:290] T 92f0ad59fbbb435ea872c669a52f854a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:23.160072 1910 tablet_loader.cc:96] loaded metadata for tablet 8b9d79a25452410891c09fd3f7c4be6d (table default.Table_442e11b9211c4b209c68f653d7bfc313 [id=af349c3f534e4e8e99fdae0344187c44])
I20250901 01:52:23.160756 1055 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8483e7212d94daba7e5401256c33aef" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:23.161852 1055 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "89d40f0331114490b51de3ba6d773571" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:23.162042 858 leader_election.cc:336] T f80f13db501c42c7950499df8fcb6fc8 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:23.162128 1910 tablet_loader.cc:96] loaded metadata for tablet 8ca5cf4e99014c0d84a3acad4ef713af (table default.Table_ed6fd284be6f4ae3a008e21e7ce17930 [id=303e76c918bc4c39aa47a4d24849a358])
I20250901 01:52:23.163262 1055 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9b293bf6a17453b9ea9892a6970d0b0" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:23.164129 1910 tablet_loader.cc:96] loaded metadata for tablet 8e49863da5814e43b442e6081ee911b9 (table default.Table_de8ae42a0de043129ceab6caec04d192 [id=488699b14a44496987bd0d3b7d64fa5b])
I20250901 01:52:23.166569 1150 tablet_bootstrap.cc:654] T 5e2472ee5d6e4d3bbf1f24027ba531ca P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:23.167387 1910 tablet_loader.cc:96] loaded metadata for tablet 8ed6616ce54548ba83ba299c4ecf3c38 (table default.Table_8cd9c1e6a8e74232b32d36451269562a [id=946ec3e53dcd4de4b8501b3e87e99072])
I20250901 01:52:23.168079 1910 tablet_loader.cc:96] loaded metadata for tablet 9127f9c5239e445ba86012951b7c75d9 (table default.Table_da8cc5e7d1e24421b170ecca2fc2147b [id=ab670bb02c6e4ed491e7831ad73cdfc6])
I20250901 01:52:23.168799 1910 tablet_loader.cc:96] loaded metadata for tablet 92f0ad59fbbb435ea872c669a52f854a (table default.Table_4aa01c0e60364804a5e99d552ca3956c [id=59306e9184ca4715b1cd675609807eca])
I20250901 01:52:23.170941 1922 hybrid_clock.cc:648] HybridClock initialized: now 1756691543170824 us; error 84 us; skew 500 ppm
I20250901 01:52:23.171469 1922 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250901 01:52:23.178915 1055 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f80f13db501c42c7950499df8fcb6fc8" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:23.180166 992 leader_election.cc:336] T d77aadd7291647a0bae7c421f244c252 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:23.180958 992 consensus_peers.cc:489] T a6445b29591548be9f9577db1ba0bfa5 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250901 01:52:23.181308 992 consensus_peers.cc:489] T 606fac5c6efe48d1877104b01c23ba5b P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250901 01:52:23.181449 992 consensus_peers.cc:489] T ded33e0950b544509af11739353c620d P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250901 01:52:23.183104 859 leader_election.cc:343] T a8483e7212d94daba7e5401256c33aef P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:23.183229 859 leader_election.cc:304] T a8483e7212d94daba7e5401256c33aef P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:23.183597 859 leader_election.cc:343] T 89d40f0331114490b51de3ba6d773571 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:23.183697 859 leader_election.cc:304] T 89d40f0331114490b51de3ba6d773571 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:23.184032 859 leader_election.cc:343] T f9b293bf6a17453b9ea9892a6970d0b0 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:23.184032 992 leader_election.cc:304] T fae2779fac164b9f8052c3666db803b8 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:23.184129 859 leader_election.cc:304] T f9b293bf6a17453b9ea9892a6970d0b0 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:23.184576 918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92f0ad59fbbb435ea872c669a52f854a" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:23.184778 918 raft_consensus.cc:2466] T 92f0ad59fbbb435ea872c669a52f854a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:23.184834 1761 raft_consensus.cc:2802] T fae2779fac164b9f8052c3666db803b8 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:23.184952 1761 raft_consensus.cc:491] T fae2779fac164b9f8052c3666db803b8 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:23.184998 1999 raft_consensus.cc:2747] T a8483e7212d94daba7e5401256c33aef P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:23.185078 1761 raft_consensus.cc:3058] T fae2779fac164b9f8052c3666db803b8 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:23.185137 1999 raft_consensus.cc:2747] T 89d40f0331114490b51de3ba6d773571 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:23.185241 1999 raft_consensus.cc:2747] T f9b293bf6a17453b9ea9892a6970d0b0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:23.185915 859 leader_election.cc:343] T f80f13db501c42c7950499df8fcb6fc8 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:23.186026 859 leader_election.cc:304] T f80f13db501c42c7950499df8fcb6fc8 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:23.186796 1981 raft_consensus.cc:2747] T f80f13db501c42c7950499df8fcb6fc8 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:23.188515 1910 tablet_loader.cc:96] loaded metadata for tablet 93bcac1fd95d44dc927abfca81692921 (table default.Table_b8f8ee74648c406eaf589d3ab80934db [id=b8f69313439543be9c11eea88dfa69c5])
I20250901 01:52:23.189795 1910 tablet_loader.cc:96] loaded metadata for tablet 98c56eb62ad84f79b04f17a43d44c479 (table default.Table_cb4904990e284a1e8137d11272036154 [id=00e7f7fe6dc44f2787b9dd8e3cbf4a51])
I20250901 01:52:23.190979 1151 tablet_bootstrap.cc:492] T 8dd587304e6342edbe96c9897b40e5c9 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:23.191195 1151 ts_tablet_manager.cc:1397] T 8dd587304e6342edbe96c9897b40e5c9 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.066s user 0.004s sys 0.004s
I20250901 01:52:23.191708 1981 raft_consensus.cc:491] T eaa521c52a2f4408b59770bd2a05907c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:23.191838 1981 raft_consensus.cc:513] T eaa521c52a2f4408b59770bd2a05907c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:23.192220 1761 raft_consensus.cc:513] T fae2779fac164b9f8052c3666db803b8 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.193255 1981 leader_election.cc:290] T eaa521c52a2f4408b59770bd2a05907c P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:23.192860 1151 raft_consensus.cc:357] T 8dd587304e6342edbe96c9897b40e5c9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.193383 1151 raft_consensus.cc:383] T 8dd587304e6342edbe96c9897b40e5c9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:23.193655 1151 raft_consensus.cc:738] T 8dd587304e6342edbe96c9897b40e5c9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
W20250901 01:52:23.194010 858 leader_election.cc:336] T eaa521c52a2f4408b59770bd2a05907c P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:23.195566 1761 leader_election.cc:290] T fae2779fac164b9f8052c3666db803b8 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:23.195777 1151 consensus_queue.cc:260] T 8dd587304e6342edbe96c9897b40e5c9 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.195868 992 leader_election.cc:304] T 92f0ad59fbbb435ea872c669a52f854a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:23.196801 1761 raft_consensus.cc:2802] T 92f0ad59fbbb435ea872c669a52f854a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:23.197021 1761 raft_consensus.cc:491] T 92f0ad59fbbb435ea872c669a52f854a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:23.197476 1761 raft_consensus.cc:3058] T 92f0ad59fbbb435ea872c669a52f854a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:23.202930 1151 ts_tablet_manager.cc:1428] T 8dd587304e6342edbe96c9897b40e5c9 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
W20250901 01:52:23.203083 992 leader_election.cc:336] T fae2779fac164b9f8052c3666db803b8 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:23.203809 1910 tablet_loader.cc:96] loaded metadata for tablet 9fbd44641d9e43c3b592e388e4adf350 (table default.Table_cbeebbc633854c008e07b7d39d2024ee [id=6fdd3d817dc74551923021bd716ba58b])
I20250901 01:52:23.204792 1151 tablet_bootstrap.cc:492] T 7b16ca1daf394a6da65a4af8da82ec33 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:23.205248 1910 tablet_loader.cc:96] loaded metadata for tablet a289a66b2dfa44dc8c8547ff83c1716e (table default.Table_4f2be95a1b824f06ba16e56abcdebc8a [id=434cfb105b894499b15bf360a8cffda4])
I20250901 01:52:23.205780 1910 tablet_loader.cc:96] loaded metadata for tablet a2ca69fabadd4ad69e1670cae7a0c503 (table default.Table_97a5ec9ec34c4a89b0f917704b43e52b [id=956e796e0f5747f6ad5b4e4163639c35])
I20250901 01:52:23.207125 921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fae2779fac164b9f8052c3666db803b8" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:23.207276 921 raft_consensus.cc:3058] T fae2779fac164b9f8052c3666db803b8 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:23.208372 1910 tablet_loader.cc:96] loaded metadata for tablet a4e3a8180fc14dacb1973e04f99c4492 (table default.Table_2227d0e5eae04a08853aa24bfe533ee5 [id=06560cb0c30e40bd876ec65931ab75bf])
I20250901 01:52:23.209843 921 raft_consensus.cc:2466] T fae2779fac164b9f8052c3666db803b8 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:23.210702 1910 tablet_loader.cc:96] loaded metadata for tablet a4ffb0db51e54066a973d29b92cfbcf0 (table default.Table_ea8e4576448a476b8bf399421a378e2c [id=6be613d73c4a4044afc7750d2367b4d2])
I20250901 01:52:23.211401 1761 raft_consensus.cc:513] T 92f0ad59fbbb435ea872c669a52f854a P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.211738 1151 tablet_bootstrap.cc:654] T 7b16ca1daf394a6da65a4af8da82ec33 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:23.211985 1761 leader_election.cc:290] T 92f0ad59fbbb435ea872c669a52f854a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:23.212551 1055 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eaa521c52a2f4408b59770bd2a05907c" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:23.212958 992 leader_election.cc:336] T 92f0ad59fbbb435ea872c669a52f854a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:23.215013 859 leader_election.cc:343] T eaa521c52a2f4408b59770bd2a05907c P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:23.215013 921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92f0ad59fbbb435ea872c669a52f854a" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:23.215175 859 leader_election.cc:304] T eaa521c52a2f4408b59770bd2a05907c P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:23.215236 921 raft_consensus.cc:3058] T 92f0ad59fbbb435ea872c669a52f854a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:23.215561 1981 raft_consensus.cc:2747] T eaa521c52a2f4408b59770bd2a05907c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:23.216610 1910 tablet_loader.cc:96] loaded metadata for tablet a6445b29591548be9f9577db1ba0bfa5 (table default.Table_409dad98d7eb455a8e16edfb4acedb34 [id=6f29cdc9974049368af2ae10beadb772])
I20250901 01:52:23.218138 921 raft_consensus.cc:2466] T 92f0ad59fbbb435ea872c669a52f854a P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:23.218907 1990 raft_consensus.cc:491] T c49d1143d54b4c319739488906b0e271 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:23.219030 1990 raft_consensus.cc:513] T c49d1143d54b4c319739488906b0e271 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.219566 1990 leader_election.cc:290] T c49d1143d54b4c319739488906b0e271 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:23.219695 1990 consensus_queue.cc:1035] T 670d90a37b934579b353bb4e4820127b P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:23.221055 1150 tablet_bootstrap.cc:492] T 5e2472ee5d6e4d3bbf1f24027ba531ca P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:23.221187 1150 ts_tablet_manager.cc:1397] T 5e2472ee5d6e4d3bbf1f24027ba531ca P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.062s user 0.003s sys 0.009s
I20250901 01:52:23.223381 1910 tablet_loader.cc:96] loaded metadata for tablet a8483e7212d94daba7e5401256c33aef (table default.Table_142ccc8cc7414621bc0c31326d5aa0e9 [id=6a01b6c6977546be9c6128ca5f200a92])
I20250901 01:52:23.223848 1910 tablet_loader.cc:96] loaded metadata for tablet a9b7f2e6548043a7865e25387cdd06c9 (table default.Table_d28d7742171c4465a8b438d2e337b5f7 [id=5bb33f9da2dc4a9cbf17b31cd14fdd23])
I20250901 01:52:23.224036 1990 consensus_queue.cc:1035] T 831e42de95854541a49ae6b3e039d3e9 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:23.225183 992 leader_election.cc:304] T fae2779fac164b9f8052c3666db803b8 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:23.225445 1990 raft_consensus.cc:2802] T fae2779fac164b9f8052c3666db803b8 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:23.225572 1990 raft_consensus.cc:695] T fae2779fac164b9f8052c3666db803b8 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:23.225816 1990 consensus_queue.cc:237] T fae2779fac164b9f8052c3666db803b8 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.226949 1872 raft_consensus.cc:491] T 9127f9c5239e445ba86012951b7c75d9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:23.227073 1872 raft_consensus.cc:513] T 9127f9c5239e445ba86012951b7c75d9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.227584 1872 leader_election.cc:290] T 9127f9c5239e445ba86012951b7c75d9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:23.228807 1910 tablet_loader.cc:96] loaded metadata for tablet aa424ac6ef1441e6b1cc2867b1c48186 (table default.Table_aba07c28a3e248e4b102730106d6a2ba [id=76928d8caa2f4134a014656cf2cc871a])
I20250901 01:52:23.229480 1910 tablet_loader.cc:96] loaded metadata for tablet abf6e3be60f74872b2308dc95bf9cd37 (table default.Table_fea398de1f1948809aebf95c387f4b17 [id=6975ac6754db4664a168e25ad61db164])
I20250901 01:52:23.229811 1910 tablet_loader.cc:96] loaded metadata for tablet ad1a6be9fcc045d3bb4e71a2f6bd5ccd (table default.Table_5e84ae9a9c2a4084a1f9c5ffb4594de2 [id=9bebb2e486ab48888e6b523281ac1e88])
I20250901 01:52:23.230182 1910 tablet_loader.cc:96] loaded metadata for tablet ad45c90ba0da43c9860745e534664a98 (table default.Table_1cd3f3fb6b244b3f8796bfca9f7622c8 [id=6fd739b0e725424ca844dbb652ea8028])
I20250901 01:52:23.230535 1910 tablet_loader.cc:96] loaded metadata for tablet ae21b9ae2e3b472eb67fb5f96388a50f (table default.Table_046289f393ff432aae0b033bdec1fb17 [id=db57b18a9ce24e068b362f5fbe940531])
I20250901 01:52:23.230917 1910 tablet_loader.cc:96] loaded metadata for tablet aec951f2cc84499b9162b9fa24d5440e (table default.Table_8770c960d4bd4c49b043ffe889b4b87b [id=55c31cd42fac43e6b8eddc048f565d0a])
I20250901 01:52:23.231019 992 leader_election.cc:304] T 92f0ad59fbbb435ea872c669a52f854a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:23.231413 1150 raft_consensus.cc:357] T 5e2472ee5d6e4d3bbf1f24027ba531ca P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.231607 1993 raft_consensus.cc:2802] T 92f0ad59fbbb435ea872c669a52f854a P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:23.231632 1150 raft_consensus.cc:383] T 5e2472ee5d6e4d3bbf1f24027ba531ca P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:23.231691 1150 raft_consensus.cc:738] T 5e2472ee5d6e4d3bbf1f24027ba531ca P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:23.231719 1993 raft_consensus.cc:695] T 92f0ad59fbbb435ea872c669a52f854a P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:23.231889 1150 consensus_queue.cc:260] T 5e2472ee5d6e4d3bbf1f24027ba531ca P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.231926 1993 consensus_queue.cc:237] T 92f0ad59fbbb435ea872c669a52f854a P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.234288 1910 tablet_loader.cc:96] loaded metadata for tablet b009cc95c819431ea9827a3e453b8da3 (table default.Table_203547c9458e43b3b48ea051d56892c6 [id=cc26c89db2484a6fad919231116285f2])
I20250901 01:52:23.234700 1910 tablet_loader.cc:96] loaded metadata for tablet b11e8f77243e4764b96548d1132b8e5a (table default.Table_3c7c628f35fb4d02bb79a34995f12938 [id=a39e8b171a874df1890d072732e55df9])
I20250901 01:52:23.235059 1910 tablet_loader.cc:96] loaded metadata for tablet b13afacd37474781877b9f2cd20b313a (table default.Table_36f5f8854ed14d1ea48eb18865d00659 [id=bdf8a102ae7d43228173d7b82507d80d])
I20250901 01:52:23.235464 1910 tablet_loader.cc:96] loaded metadata for tablet b164b4a67a9143ceaff69b0ce49d63fe (table default.Table_da2c70a8f2fe49a0a8184e88c0a9e0ea [id=c1b1e9e3876244f68d31c63ce7bdebf1])
I20250901 01:52:23.237043 1910 tablet_loader.cc:96] loaded metadata for tablet b2b172757e50404aae78e818921a7d9d (table default.Table_21b93b5fe2ce49c6bc637d8cab41fc06 [id=5419e36671ec470a98dff53acdbd4760])
W20250901 01:52:23.236797 992 leader_election.cc:336] T fae2779fac164b9f8052c3666db803b8 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:23.237298 1150 ts_tablet_manager.cc:1428] T 5e2472ee5d6e4d3bbf1f24027ba531ca P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
I20250901 01:52:23.237605 1910 tablet_loader.cc:96] loaded metadata for tablet b2e8b2cd10154de08ddd30c214592a4a (table default.Table_4a412a59b6db421eb0187ffd02ecf541 [id=fd72e46c84634b6c9ea74970480ecc62])
I20250901 01:52:23.238029 1910 tablet_loader.cc:96] loaded metadata for tablet b4278a013ae84082bc7a8497b04ac147 (table default.Table_aa1bfa4e8b59486f96c587bc9cf84d73 [id=6b36921a6764455aa19284d36c3f8d32])
I20250901 01:52:23.238173 1151 tablet_bootstrap.cc:492] T 7b16ca1daf394a6da65a4af8da82ec33 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:23.238299 1151 ts_tablet_manager.cc:1397] T 7b16ca1daf394a6da65a4af8da82ec33 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.034s user 0.001s sys 0.005s
I20250901 01:52:23.239040 921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c49d1143d54b4c319739488906b0e271" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:23.239086 1151 raft_consensus.cc:357] T 7b16ca1daf394a6da65a4af8da82ec33 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.239261 1151 raft_consensus.cc:383] T 7b16ca1daf394a6da65a4af8da82ec33 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:23.239321 1151 raft_consensus.cc:738] T 7b16ca1daf394a6da65a4af8da82ec33 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:23.239356 921 raft_consensus.cc:2466] T c49d1143d54b4c319739488906b0e271 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:23.240096 1150 tablet_bootstrap.cc:492] T f6a8b425968e49deaecfa3adacfd3f1f P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:23.240346 1910 tablet_loader.cc:96] loaded metadata for tablet b8ca4e66c6a74f9d8d90a69be5d6a58f (table default.Table_259a716264db43c49671b6fef6665f0e [id=aebce875b7dd4718a3611eec4ca56917])
I20250901 01:52:23.241017 1910 tablet_loader.cc:96] loaded metadata for tablet bd9b59f0864a497faf281edd6077668d (table default.Table_0c9bfebabc22494489b1c1251a412c81 [id=8b74044a78e0493dbaf1835f33334212])
I20250901 01:52:23.241551 1910 tablet_loader.cc:96] loaded metadata for tablet be8dfbc466564842a2fdb4953eee939b (table default.Table_75ba3ba70ca24a94920e7bce96190ff2 [id=a72f676761b045e29dc0daa74687333d])
I20250901 01:52:23.241977 1910 tablet_loader.cc:96] loaded metadata for tablet becbaae4fd7e44faa679e9479465a7c4 (table default.Table_0e73f80424cd48408f18c2eff1411706 [id=47c13af47c944d0fa79dee115033265d])
I20250901 01:52:23.241952 1151 consensus_queue.cc:260] T 7b16ca1daf394a6da65a4af8da82ec33 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.242537 1150 tablet_bootstrap.cc:654] T f6a8b425968e49deaecfa3adacfd3f1f P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:23.243443 1922 webserver.cc:480] Webserver started at http://127.0.104.62:34679/ using document root <none> and password file <none>
I20250901 01:52:23.243976 1910 tablet_loader.cc:96] loaded metadata for tablet c422881b40f54971a68530abfbd1945b (table default.Table_f611c1794e564a0496a761da46171d0e [id=dc22dac261644596b2d65e804791462d])
I20250901 01:52:23.244793 1922 fs_manager.cc:362] Metadata directory not provided
I20250901 01:52:23.244952 1922 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250901 01:52:23.245818 1910 tablet_loader.cc:96] loaded metadata for tablet c6f7c525f59b49938b697de98d4b5835 (table default.Table_db61e055dbd7450898e82c4c0c4763be [id=aabca67f6f754820977ac9c4c20f3e4f])
I20250901 01:52:23.246737 1910 tablet_loader.cc:96] loaded metadata for tablet c754129977904f978f87fd89d7092e36 (table default.Table_e113aaaa38db48fdb16d32569b81b930 [id=f09f20dc4db844a8abc825e9086093ba])
I20250901 01:52:23.246763 1151 ts_tablet_manager.cc:1428] T 7b16ca1daf394a6da65a4af8da82ec33 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20250901 01:52:23.247377 1910 tablet_loader.cc:96] loaded metadata for tablet c83ce95fd6fa41478de0e2168be201b1 (table default.Table_bb44f1b7c92140309a212038f8bf1286 [id=bfd87b9b708449d3b16e3e0639561a49])
I20250901 01:52:23.248220 1910 tablet_loader.cc:96] loaded metadata for tablet cb7d7e1796c0456f8de98fb812b9e374 (table default.Table_c4148158ed7b48a3946559ba978e72fd [id=327c5a63904540508142ab600c308757])
I20250901 01:52:23.248533 1151 tablet_bootstrap.cc:492] T b7b802f976f3471faaaecc964ec87e48 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:23.249367 1910 tablet_loader.cc:96] loaded metadata for tablet ced4813805004bba9c4667a79ac201fc (table default.Table_c7bfa05d564246a6be270afb7a3f0bde [id=16030cb64ac64ff68762651bd095fed1])
I20250901 01:52:23.249819 1910 tablet_loader.cc:96] loaded metadata for tablet cf806854a8074b0798b3f9f3df587c63 (table default.Table_2d7b072b55354009b9cce1aef9408e35 [id=e42970b06935463186a7d4a23b9e74c2])
I20250901 01:52:23.250128 1151 tablet_bootstrap.cc:654] T b7b802f976f3471faaaecc964ec87e48 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:23.250348 1910 tablet_loader.cc:96] loaded metadata for tablet d000ba05e12843b4925e622ab55471c0 (table default.Table_820e1c5a64cd4128b7ae1b70bd951964 [id=b43a92ed46164d199b445287ef039bc6])
I20250901 01:52:23.250797 1910 tablet_loader.cc:96] loaded metadata for tablet d0463a8d655d4db4800b4e1b221c8315 (table default.Table_0b769136b0664331945b30cfe0482fc6 [id=135c19f9795c4195bb211d0be5fba6a5])
I20250901 01:52:23.251230 1910 tablet_loader.cc:96] loaded metadata for tablet d0b7c84ec9394a20ad77add3808e555a (table default.Table_85b9e2f3f8f7400aa833201daae93dc3 [id=cac94430e07342dfa9b266df6472b6e3])
I20250901 01:52:23.251616 1910 tablet_loader.cc:96] loaded metadata for tablet d0c3c363dda444519a17dca63ad9caa7 (table default.Table_0b4ac0a1caa5472d909448e41318e679 [id=8c1b263824ec4c15a5ec424364c78895])
I20250901 01:52:23.252004 1910 tablet_loader.cc:96] loaded metadata for tablet d10c193452954c198bba686169b8025b (table default.Table_03fb48eaab9b42858ab7cefb722e43a8 [id=6c0576410d7340d6aa493a747e95565b])
I20250901 01:52:23.252864 1910 tablet_loader.cc:96] loaded metadata for tablet d1f2d98712ff4ea6aec665512de068bd (table default.Table_82d91a1ba2eb4265896824028dec8d3a [id=3f7624d2901445e39155c12fa68e34e9])
I20250901 01:52:23.253306 1910 tablet_loader.cc:96] loaded metadata for tablet d27c8d99528a42ed8dd1c5a7f373bbae (table default.Table_d6b752d82c2b4be88b762e67bac0d5a1 [id=ce82be9787f14aa6bcdcb070228abc51])
I20250901 01:52:23.254248 1150 tablet_bootstrap.cc:492] T f6a8b425968e49deaecfa3adacfd3f1f P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:23.254429 1150 ts_tablet_manager.cc:1397] T f6a8b425968e49deaecfa3adacfd3f1f P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20250901 01:52:23.254838 1910 tablet_loader.cc:96] loaded metadata for tablet d4c9aa723aee4dc4b6237cb5c3722b52 (table default.Table_5e65c5f068b84995a2d7a76fa65dda0f [id=b9d81ddd38fe47838877b010be2b40d0])
I20250901 01:52:23.255589 1150 raft_consensus.cc:357] T f6a8b425968e49deaecfa3adacfd3f1f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.257308 1910 tablet_loader.cc:96] loaded metadata for tablet da5f69c5b0ca4257ad2ff7918ee2af2e (table default.Table_8f7db2ae75a24dfab71c2a95c6a0a404 [id=61f1e8a842844ed899e5913c76f362ff])
I20250901 01:52:23.257529 1150 raft_consensus.cc:383] T f6a8b425968e49deaecfa3adacfd3f1f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:23.257632 1150 raft_consensus.cc:738] T f6a8b425968e49deaecfa3adacfd3f1f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:23.257875 1150 consensus_queue.cc:260] T f6a8b425968e49deaecfa3adacfd3f1f P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.258594 1150 ts_tablet_manager.cc:1428] T f6a8b425968e49deaecfa3adacfd3f1f P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250901 01:52:23.258908 1150 tablet_bootstrap.cc:492] T 71941b815d6840478c5acc1cbb187cc9 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:23.259305 1910 tablet_loader.cc:96] loaded metadata for tablet dabebc77f72e4bb19e264be2f496202a (table default.Table_364cf1adeb2c4102aad338f2e228fb0b [id=a4d0af97df5949cca940a81a41b8e454])
I20250901 01:52:23.260728 1150 tablet_bootstrap.cc:654] T 71941b815d6840478c5acc1cbb187cc9 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:23.261224 1910 tablet_loader.cc:96] loaded metadata for tablet dac01ff54b97443289d4f3d10f80cd41 (table default.Table_54422fcb2b624e7fa8d42cba3558a327 [id=ee0eb3570f614c5099121396d0e52ca3])
I20250901 01:52:23.261247 1151 tablet_bootstrap.cc:492] T b7b802f976f3471faaaecc964ec87e48 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:23.261369 1151 ts_tablet_manager.cc:1397] T b7b802f976f3471faaaecc964ec87e48 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.001s
I20250901 01:52:23.262001 1910 tablet_loader.cc:96] loaded metadata for tablet ddbc15685fca4c708329420109ad0c57 (table default.Table_942175dadf4b496195b2fa53f39f6ddc [id=2bd332b452f84f25af4225af8e6c6740])
I20250901 01:52:23.262257 1151 raft_consensus.cc:357] T b7b802f976f3471faaaecc964ec87e48 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.262430 1151 raft_consensus.cc:383] T b7b802f976f3471faaaecc964ec87e48 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:23.262503 1151 raft_consensus.cc:738] T b7b802f976f3471faaaecc964ec87e48 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:23.262763 1151 consensus_queue.cc:260] T b7b802f976f3471faaaecc964ec87e48 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.263388 1151 ts_tablet_manager.cc:1428] T b7b802f976f3471faaaecc964ec87e48 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250901 01:52:23.263443 1910 tablet_loader.cc:96] loaded metadata for tablet ded33e0950b544509af11739353c620d (table default.Table_93afd37289414655980b22934695bd55 [id=01c391f6c76943c9af685a74348eaf41])
I20250901 01:52:23.263680 1151 tablet_bootstrap.cc:492] T 50e3da53433f4c14b57db8a733fa53bc P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:23.263960 1910 tablet_loader.cc:96] loaded metadata for tablet df01d5df9fc2424eaf9de4e68e781803 (table default.Table_d56ff20d129b440cbab7b15c4af6fa46 [id=c6acca4c20ba4dfaa385fc3c124507a3])
I20250901 01:52:23.264775 1910 tablet_loader.cc:96] loaded metadata for tablet e1d18e999b5e42b1873631e0f1445081 (table default.Table_fd17a2ad225546aa821489ce94a788a2 [id=033c76e0598342c495ce2be42a0df5dd])
I20250901 01:52:23.265926 1910 tablet_loader.cc:96] loaded metadata for tablet e4fdbc8d4bff4173a20db2c182acfd9e (table default.Table_5e8920128d214deb96ef83f369a8b3f1 [id=d7615918ae0f48d394fbe509b82127e7])
I20250901 01:52:23.266373 1910 tablet_loader.cc:96] loaded metadata for tablet e5e250e6c84d4248b857bf1478d47703 (table default.Table_da76b13a22bc47eea908959d5c1720df [id=815cc5de9497417ea88a3f72e3e729ed])
I20250901 01:52:23.267113 1910 tablet_loader.cc:96] loaded metadata for tablet e6c10426f60f43e8bde69d9b1b9c8eb6 (table default.Table_eecfab7239cc40c5a2f1fb6ac6d35dc9 [id=d3dbeb4258424133844e20b181ee44e4])
I20250901 01:52:23.268378 1910 tablet_loader.cc:96] loaded metadata for tablet e87d1281f35648b0b115b67c95e244e7 (table default.Table_f840789a58984799bd8057128f6ea439 [id=4453959e273c473fadd737f56f7db3dd])
W20250901 01:52:23.269161 992 leader_election.cc:336] T 92f0ad59fbbb435ea872c669a52f854a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:23.269537 921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9127f9c5239e445ba86012951b7c75d9" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:23.269718 921 raft_consensus.cc:2466] T 9127f9c5239e445ba86012951b7c75d9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:23.270447 992 leader_election.cc:304] T c49d1143d54b4c319739488906b0e271 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:23.270684 1993 raft_consensus.cc:2802] T c49d1143d54b4c319739488906b0e271 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:23.270758 1993 raft_consensus.cc:491] T c49d1143d54b4c319739488906b0e271 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:23.270816 1993 raft_consensus.cc:3058] T c49d1143d54b4c319739488906b0e271 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:23.272029 992 leader_election.cc:304] T 9127f9c5239e445ba86012951b7c75d9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:23.272450 1872 raft_consensus.cc:2802] T 9127f9c5239e445ba86012951b7c75d9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:23.272531 1872 raft_consensus.cc:491] T 9127f9c5239e445ba86012951b7c75d9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:23.272598 1872 raft_consensus.cc:3058] T 9127f9c5239e445ba86012951b7c75d9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:23.272598 1761 consensus_queue.cc:1035] T 606fac5c6efe48d1877104b01c23ba5b P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:23.273095 1761 consensus_queue.cc:1035] T f06a77ecc841462396a8d5026c7327f4 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:23.273156 1910 tablet_loader.cc:96] loaded metadata for tablet e9f1f892658b4736859ff507536920a1 (table default.Table_8825dd28afbe464193eb1b668cc9326a [id=9aa62cb634314a0a849612c54c1220d9])
I20250901 01:52:23.273629 1981 raft_consensus.cc:491] T a289a66b2dfa44dc8c8547ff83c1716e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:23.273752 1981 raft_consensus.cc:513] T a289a66b2dfa44dc8c8547ff83c1716e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.274266 1981 leader_election.cc:290] T a289a66b2dfa44dc8c8547ff83c1716e P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:23.274658 1993 raft_consensus.cc:513] T c49d1143d54b4c319739488906b0e271 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.275305 1993 leader_election.cc:290] T c49d1143d54b4c319739488906b0e271 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:23.275413 1872 raft_consensus.cc:513] T 9127f9c5239e445ba86012951b7c75d9 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.273490 1910 tablet_loader.cc:96] loaded metadata for tablet ea846f88b3534beebd0001b7f17b5ca9 (table default.Table_2b74c001cb4f4d82ab314557f4f09422 [id=85b64046d2fa4f1d871ab43e5c800fb8])
I20250901 01:52:23.275918 1872 leader_election.cc:290] T 9127f9c5239e445ba86012951b7c75d9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:23.276209 1151 tablet_bootstrap.cc:654] T 50e3da53433f4c14b57db8a733fa53bc P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:23.276300 1910 tablet_loader.cc:96] loaded metadata for tablet eac50cbae691416e903cb2cbc907f96d (table default.Table_d9e28b97a2e94ca8bf0ee8a01acd76e8 [id=e4a6d6fe44f947e59089e1d7855b785c])
I20250901 01:52:23.276707 1910 tablet_loader.cc:96] loaded metadata for tablet eb19e3a424324a24824f6caaf5848952 (table default.Table_3b46a0a1f77a4cea842e19b3c89de475 [id=fac15e527f974e228cf2e53026fba40e])
I20250901 01:52:23.276829 1981 raft_consensus.cc:491] T 6741c56499b44f648fb032c8cf50205b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:23.276937 1981 raft_consensus.cc:513] T 6741c56499b44f648fb032c8cf50205b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.277097 1910 tablet_loader.cc:96] loaded metadata for tablet ed1b41340ce742a88c3e4fd85cd25ca6 (table default.Table_139f6d174d794a6e9a618e90587d0c0d [id=0baa4993b72e4a8d94ee3059c9d2db6c])
I20250901 01:52:23.277395 1981 leader_election.cc:290] T 6741c56499b44f648fb032c8cf50205b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:23.277652 1910 tablet_loader.cc:96] loaded metadata for tablet eee34044c39d4013baf78d3705489626 (table default.Table_99afdac28a4e4ac9ae93b4fbbe704a4d [id=4639e91c14d8427d86c89d3c2b0918ad])
W20250901 01:52:23.278451 858 leader_election.cc:336] T a289a66b2dfa44dc8c8547ff83c1716e P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:23.278688 1910 tablet_loader.cc:96] loaded metadata for tablet f13b972e25214f168421439fe3943e8a (table default.Table_d5a6ac03c9ff4969ada556d1318b64f0 [id=c1483e9d0db64d468b12344500f9cac8])
W20250901 01:52:23.279428 858 leader_election.cc:336] T 6741c56499b44f648fb032c8cf50205b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:23.280174 1922 fs_manager.cc:714] Time spent opening directory manager: real 0.026s user 0.004s sys 0.000s
I20250901 01:52:23.282351 1910 tablet_loader.cc:96] loaded metadata for tablet f195b040a6ae4b7b83a8308ab0a12b37 (table default.Table_8b1086c9a4c44239851c586ea7267e3c [id=523de440fbab4bf687c0f6168e48513c])
I20250901 01:52:23.282735 1910 tablet_loader.cc:96] loaded metadata for tablet f23df6f253a943fa8e146bb8c811ef51 (table default.Table_59039012009944fdbe0179002ea44633 [id=218947d08fb24732b5d0375d4af973e9])
I20250901 01:52:23.283340 1910 tablet_loader.cc:96] loaded metadata for tablet f4fd117c395e41c2ab679c684908c5b7 (table default.Table_5e32028e03b04431b57a5d3a653afda4 [id=305490ecfddb4d46927f7bb128c7f878])
I20250901 01:52:23.284233 1151 tablet_bootstrap.cc:492] T 50e3da53433f4c14b57db8a733fa53bc P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:23.284394 1151 ts_tablet_manager.cc:1397] T 50e3da53433f4c14b57db8a733fa53bc P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.000s
I20250901 01:52:23.285348 1151 raft_consensus.cc:357] T 50e3da53433f4c14b57db8a733fa53bc P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.285465 1910 tablet_loader.cc:96] loaded metadata for tablet f62d96c37cca4143a4b098b7c69979e6 (table default.Table_0877ac8202a24e9da650a589bc6dbfcc [id=1f65b3e09c9748f189ecb2b56c5d1132])
I20250901 01:52:23.285533 1151 raft_consensus.cc:383] T 50e3da53433f4c14b57db8a733fa53bc P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:23.285588 1151 raft_consensus.cc:738] T 50e3da53433f4c14b57db8a733fa53bc P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:23.285849 1910 tablet_loader.cc:96] loaded metadata for tablet f65ec84958eb488d8b0f812b59b5b3d5 (table default.Table_66ad08ce0b6840d4848bcddb304cf88b [id=d1cd5288671540c0a95f7a037363014e])
I20250901 01:52:23.286614 1151 consensus_queue.cc:260] T 50e3da53433f4c14b57db8a733fa53bc P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.287174 1151 ts_tablet_manager.cc:1428] T 50e3da53433f4c14b57db8a733fa53bc P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250901 01:52:23.287629 1910 tablet_loader.cc:96] loaded metadata for tablet f6a8b425968e49deaecfa3adacfd3f1f (table default.Table_c2ed44ae561c4b88b22e8975a61b77aa [id=004b578e7e5d47ed9dd12d89076b0f28])
I20250901 01:52:23.288676 1150 tablet_bootstrap.cc:492] T 71941b815d6840478c5acc1cbb187cc9 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:23.288911 1150 ts_tablet_manager.cc:1397] T 71941b815d6840478c5acc1cbb187cc9 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.009s
I20250901 01:52:23.290026 2051 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 01:52:23.290470 1150 raft_consensus.cc:357] T 71941b815d6840478c5acc1cbb187cc9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.290633 1150 raft_consensus.cc:383] T 71941b815d6840478c5acc1cbb187cc9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:23.290685 1150 raft_consensus.cc:738] T 71941b815d6840478c5acc1cbb187cc9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:23.290907 1150 consensus_queue.cc:260] T 71941b815d6840478c5acc1cbb187cc9 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.291282 1150 ts_tablet_manager.cc:1428] T 71941b815d6840478c5acc1cbb187cc9 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250901 01:52:23.291543 1150 tablet_bootstrap.cc:492] T 3d90c6ef06d1436d8e2bb1333e7d9985 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:23.292544 1922 fs_manager.cc:730] Time spent opening block manager: real 0.010s user 0.002s sys 0.003s
I20250901 01:52:23.292888 1150 tablet_bootstrap.cc:654] T 3d90c6ef06d1436d8e2bb1333e7d9985 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:23.293715 1910 tablet_loader.cc:96] loaded metadata for tablet f80f13db501c42c7950499df8fcb6fc8 (table default.Table_ce86974fd8974de9a5a737954fac2ad7 [id=ff772d20b287476f94716379b86d2a71])
I20250901 01:52:23.294016 1151 tablet_bootstrap.cc:492] T af2c4bcbb00741f789edee109252d8cb P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:23.292773 1922 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/master-0/wal
uuid: "e3add7edb97a40ef9f7c6bc011d13227"
format_stamp: "Formatted at 2025-09-01 01:52:04 on dist-test-slave-l1vd"
I20250901 01:52:23.294466 1910 tablet_loader.cc:96] loaded metadata for tablet f8d9bbf0c5be462a8ddc539238cbdf97 (table default.Table_dfb6e260190a40df8db77152b962cb8e [id=0e10c4b90e884aaeb0a75dbcf3dfd1bc])
I20250901 01:52:23.294301 1922 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 01:52:23.294857 1910 tablet_loader.cc:96] loaded metadata for tablet f98d1d8d660443cb8066a547e5d9b77f (table default.Table_583918346a424cb293d2c35bfa4a5b5f [id=1f677c66803b4cb1bf585cb621a4df6e])
I20250901 01:52:23.295243 1910 tablet_loader.cc:96] loaded metadata for tablet f9b293bf6a17453b9ea9892a6970d0b0 (table default.Table_8d0db1143f1547c2a7d41775b4d9ef09 [id=b67bfe518a094e0781ef931524b55990])
I20250901 01:52:23.295334 1151 tablet_bootstrap.cc:654] T af2c4bcbb00741f789edee109252d8cb P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:23.295653 1910 tablet_loader.cc:96] loaded metadata for tablet fa7fb88ded21432a8757f37ec3db5e71 (table default.Table_22d8d38d0b624912a0d8968fb764420e [id=808a8849c99e485b8be7cd17397874ec])
I20250901 01:52:23.296325 1910 tablet_loader.cc:96] loaded metadata for tablet fae2779fac164b9f8052c3666db803b8 (table default.Table_7736ed4b5a0d4b1a816500c525e68ad3 [id=7191ff77d44c4c86aa44473814570d09])
I20250901 01:52:23.296762 1910 tablet_loader.cc:96] loaded metadata for tablet fcb7b5b7d63d4f0b906fea97cb548de2 (table default.Table_1c8baaaa506548bd85b9f6a7483e5c7a [id=4d9ea39016ca4bf4859f4f80ea4e9ed6])
I20250901 01:52:23.297590 1910 tablet_loader.cc:96] loaded metadata for tablet fecf7d538b4c4c619182d5eccb44dd8b (table default.Table_1b5729d9768546fe99eaafc4e2604f42 [id=02d0056abe2146ac9af81ebf4447b99d])
I20250901 01:52:23.298048 1910 tablet_loader.cc:96] loaded metadata for tablet ff013ea889d44cba8797834e4611b489 (table default.Table_0f91070385df441b96af4d5406a94724 [id=4747cdd47a954e0ab19609c9526b688d])
I20250901 01:52:23.298568 1910 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250901 01:52:23.301987 1503 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d2ae2cb6af0f4b3a8f47ab1cdd6567d3" candidate_term: 5 candidate_status { last_received { term: 4 index: 634 } } ignore_live_leader: false dest_uuid: "7ce2288f784044eea010664e3def133f" is_pre_election: true
W20250901 01:52:23.302326 1503 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33470 (request call id 4) took 504 ms (client timeout 518 ms). Trace:
W20250901 01:52:23.302420 1503 rpcz_store.cc:269] 0901 01:52:22.797619 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:22.798013 (+ 394us) service_pool.cc:225] Handling call
0901 01:52:23.302251 (+504238us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250901 01:52:23.302873 513 leader_election.cc:304] T 00000000000000000000000000000000 P d2ae2cb6af0f4b3a8f47ab1cdd6567d3 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2ae2cb6af0f4b3a8f47ab1cdd6567d3; no voters: 7ce2288f784044eea010664e3def133f, e3add7edb97a40ef9f7c6bc011d13227
I20250901 01:52:23.305400 2055 raft_consensus.cc:2747] T 00000000000000000000000000000000 P d2ae2cb6af0f4b3a8f47ab1cdd6567d3 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250901 01:52:23.311874 921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c49d1143d54b4c319739488906b0e271" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:23.312038 921 raft_consensus.cc:3058] T c49d1143d54b4c319739488906b0e271 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:23.314693 921 raft_consensus.cc:2466] T c49d1143d54b4c319739488906b0e271 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:23.315379 921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9127f9c5239e445ba86012951b7c75d9" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:23.315791 921 raft_consensus.cc:3058] T 9127f9c5239e445ba86012951b7c75d9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:23.318529 921 raft_consensus.cc:2466] T 9127f9c5239e445ba86012951b7c75d9 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:23.321795 1151 tablet_bootstrap.cc:492] T af2c4bcbb00741f789edee109252d8cb P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:23.321988 1151 ts_tablet_manager.cc:1397] T af2c4bcbb00741f789edee109252d8cb P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.028s user 0.005s sys 0.000s
I20250901 01:52:23.326635 1910 catalog_manager.cc:1261] Loaded cluster ID: d2561e56f09e40169f2e331183eec812
I20250901 01:52:23.327478 1910 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250901 01:52:23.322777 1151 raft_consensus.cc:357] T af2c4bcbb00741f789edee109252d8cb P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.327914 1151 raft_consensus.cc:383] T af2c4bcbb00741f789edee109252d8cb P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:23.327970 1151 raft_consensus.cc:738] T af2c4bcbb00741f789edee109252d8cb P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:23.328151 1151 consensus_queue.cc:260] T af2c4bcbb00741f789edee109252d8cb P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.328799 1151 ts_tablet_manager.cc:1428] T af2c4bcbb00741f789edee109252d8cb P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20250901 01:52:23.329072 1151 tablet_bootstrap.cc:492] T 2240302d9178445d8700e35f5ea274f7 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:23.330498 1151 tablet_bootstrap.cc:654] T 2240302d9178445d8700e35f5ea274f7 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:23.331851 1761 raft_consensus.cc:491] T 7b16ca1daf394a6da65a4af8da82ec33 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:23.331972 1761 raft_consensus.cc:513] T 7b16ca1daf394a6da65a4af8da82ec33 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.332531 1761 leader_election.cc:290] T 7b16ca1daf394a6da65a4af8da82ec33 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:23.339164 1761 raft_consensus.cc:491] T d58f69fad01f403eab74a47d3695ed83 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:23.339285 1761 raft_consensus.cc:513] T d58f69fad01f403eab74a47d3695ed83 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.339701 1761 leader_election.cc:290] T d58f69fad01f403eab74a47d3695ed83 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:23.339952 1055 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a289a66b2dfa44dc8c8547ff83c1716e" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:23.340466 1055 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6741c56499b44f648fb032c8cf50205b" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:23.341635 992 leader_election.cc:304] T c49d1143d54b4c319739488906b0e271 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:23.344368 1761 raft_consensus.cc:2802] T c49d1143d54b4c319739488906b0e271 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:23.344514 1761 raft_consensus.cc:695] T c49d1143d54b4c319739488906b0e271 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:23.344745 1761 consensus_queue.cc:237] T c49d1143d54b4c319739488906b0e271 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.344884 1910 catalog_manager.cc:1506] Loading token signing keys...
I20250901 01:52:23.351553 992 leader_election.cc:304] T 9127f9c5239e445ba86012951b7c75d9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:23.352061 1922 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250901 01:52:23.352200 1990 raft_consensus.cc:2802] T 9127f9c5239e445ba86012951b7c75d9 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:23.352324 1990 raft_consensus.cc:695] T 9127f9c5239e445ba86012951b7c75d9 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:23.352603 1990 consensus_queue.cc:237] T 9127f9c5239e445ba86012951b7c75d9 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.353459 1922 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 01:52:23.353879 1922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 01:52:23.354208 1981 raft_consensus.cc:491] T c58edaa76ebf47dfb1d9d5fab6de7dfa P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:23.354627 1981 raft_consensus.cc:513] T c58edaa76ebf47dfb1d9d5fab6de7dfa P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.355773 1981 leader_election.cc:290] T c58edaa76ebf47dfb1d9d5fab6de7dfa P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:23.358052 858 leader_election.cc:336] T c58edaa76ebf47dfb1d9d5fab6de7dfa P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:23.358323 1981 raft_consensus.cc:491] T ad45c90ba0da43c9860745e534664a98 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:23.358457 1981 raft_consensus.cc:513] T ad45c90ba0da43c9860745e534664a98 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.359074 1981 leader_election.cc:290] T ad45c90ba0da43c9860745e534664a98 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:23.361078 858 leader_election.cc:336] T ad45c90ba0da43c9860745e534664a98 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:23.363572 1981 raft_consensus.cc:491] T 2599f98abfdc4871b573edfc2ad97d22 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:23.363677 1981 raft_consensus.cc:513] T 2599f98abfdc4871b573edfc2ad97d22 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.364257 1981 leader_election.cc:290] T 2599f98abfdc4871b573edfc2ad97d22 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:23.366304 858 leader_election.cc:336] T 2599f98abfdc4871b573edfc2ad97d22 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:23.379230 1910 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Loaded TSK: 0
I20250901 01:52:23.384989 1910 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250901 01:52:23.386651 1873 raft_consensus.cc:491] T a54af99bc10045e8b0b6868e5aed98b4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:23.386801 1873 raft_consensus.cc:513] T a54af99bc10045e8b0b6868e5aed98b4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:23.387373 1873 leader_election.cc:290] T a54af99bc10045e8b0b6868e5aed98b4 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:23.393219 992 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111) [suppressed 166 similar messages]
W20250901 01:52:23.395179 992 leader_election.cc:336] T 9127f9c5239e445ba86012951b7c75d9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:23.395874 992 leader_election.cc:336] T c49d1143d54b4c319739488906b0e271 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:23.399652 917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7b16ca1daf394a6da65a4af8da82ec33" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:23.400035 917 raft_consensus.cc:2466] T 7b16ca1daf394a6da65a4af8da82ec33 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:23.403998 1151 tablet_bootstrap.cc:492] T 2240302d9178445d8700e35f5ea274f7 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:23.404203 1151 ts_tablet_manager.cc:1397] T 2240302d9178445d8700e35f5ea274f7 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.075s user 0.001s sys 0.002s
I20250901 01:52:23.405005 1151 raft_consensus.cc:357] T 2240302d9178445d8700e35f5ea274f7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.409415 917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d58f69fad01f403eab74a47d3695ed83" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:23.409726 917 raft_consensus.cc:2466] T d58f69fad01f403eab74a47d3695ed83 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
W20250901 01:52:23.411337 859 leader_election.cc:343] T a289a66b2dfa44dc8c8547ff83c1716e P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:23.411465 859 leader_election.cc:304] T a289a66b2dfa44dc8c8547ff83c1716e P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:23.411732 1150 tablet_bootstrap.cc:492] T 3d90c6ef06d1436d8e2bb1333e7d9985 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:23.411849 1150 ts_tablet_manager.cc:1397] T 3d90c6ef06d1436d8e2bb1333e7d9985 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.120s user 0.032s sys 0.052s
W20250901 01:52:23.411898 859 leader_election.cc:343] T 6741c56499b44f648fb032c8cf50205b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:23.412022 859 leader_election.cc:304] T 6741c56499b44f648fb032c8cf50205b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:23.412392 1999 raft_consensus.cc:2747] T a289a66b2dfa44dc8c8547ff83c1716e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:23.412586 1150 raft_consensus.cc:357] T 3d90c6ef06d1436d8e2bb1333e7d9985 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.412402 1981 raft_consensus.cc:2747] T 6741c56499b44f648fb032c8cf50205b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:23.412753 1150 raft_consensus.cc:383] T 3d90c6ef06d1436d8e2bb1333e7d9985 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:23.412804 1150 raft_consensus.cc:738] T 3d90c6ef06d1436d8e2bb1333e7d9985 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:23.412961 1150 consensus_queue.cc:260] T 3d90c6ef06d1436d8e2bb1333e7d9985 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.413373 1151 raft_consensus.cc:383] T 2240302d9178445d8700e35f5ea274f7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:23.413515 1151 raft_consensus.cc:738] T 2240302d9178445d8700e35f5ea274f7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:23.413537 1150 ts_tablet_manager.cc:1428] T 3d90c6ef06d1436d8e2bb1333e7d9985 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250901 01:52:23.413779 1150 tablet_bootstrap.cc:492] T 7954393c8c1c4457ab8bb723c03a9c8b P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:23.414480 1151 consensus_queue.cc:260] T 2240302d9178445d8700e35f5ea274f7 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.415472 1150 tablet_bootstrap.cc:654] T 7954393c8c1c4457ab8bb723c03a9c8b P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:23.415414 921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a54af99bc10045e8b0b6868e5aed98b4" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:23.415627 921 raft_consensus.cc:2466] T a54af99bc10045e8b0b6868e5aed98b4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:23.418459 1151 ts_tablet_manager.cc:1428] T 2240302d9178445d8700e35f5ea274f7 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.014s user 0.001s sys 0.000s
I20250901 01:52:23.419358 1055 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c58edaa76ebf47dfb1d9d5fab6de7dfa" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:23.419636 1151 tablet_bootstrap.cc:492] T 49aaaec611784e6bac131efd58dbca51 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:23.421293 1151 tablet_bootstrap.cc:654] T 49aaaec611784e6bac131efd58dbca51 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:23.422456 992 consensus_peers.cc:489] T 670d90a37b934579b353bb4e4820127b P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250901 01:52:23.422557 992 leader_election.cc:336] T c49d1143d54b4c319739488906b0e271 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:23.419067 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad45c90ba0da43c9860745e534664a98" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:23.419246 1058 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2599f98abfdc4871b573edfc2ad97d22" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:23.431632 992 leader_election.cc:336] T 9127f9c5239e445ba86012951b7c75d9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:23.432971 992 leader_election.cc:304] T 7b16ca1daf394a6da65a4af8da82ec33 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:23.433247 1761 raft_consensus.cc:2802] T 7b16ca1daf394a6da65a4af8da82ec33 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:23.433544 1761 raft_consensus.cc:491] T 7b16ca1daf394a6da65a4af8da82ec33 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:23.433636 1761 raft_consensus.cc:3058] T 7b16ca1daf394a6da65a4af8da82ec33 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:23.440462 1761 raft_consensus.cc:513] T 7b16ca1daf394a6da65a4af8da82ec33 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.441058 1761 leader_election.cc:290] T 7b16ca1daf394a6da65a4af8da82ec33 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:23.441361 992 leader_election.cc:304] T d58f69fad01f403eab74a47d3695ed83 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:23.441720 1990 raft_consensus.cc:2802] T d58f69fad01f403eab74a47d3695ed83 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:23.441797 1990 raft_consensus.cc:491] T d58f69fad01f403eab74a47d3695ed83 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:23.441861 1990 raft_consensus.cc:3058] T d58f69fad01f403eab74a47d3695ed83 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:23.445082 1990 raft_consensus.cc:513] T d58f69fad01f403eab74a47d3695ed83 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.445649 1990 leader_election.cc:290] T d58f69fad01f403eab74a47d3695ed83 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:23.446159 992 leader_election.cc:304] T a54af99bc10045e8b0b6868e5aed98b4 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:23.446508 1872 raft_consensus.cc:2802] T a54af99bc10045e8b0b6868e5aed98b4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:23.446579 1872 raft_consensus.cc:491] T a54af99bc10045e8b0b6868e5aed98b4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:23.446635 1872 raft_consensus.cc:3058] T a54af99bc10045e8b0b6868e5aed98b4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:23.449527 1872 raft_consensus.cc:513] T a54af99bc10045e8b0b6868e5aed98b4 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:23.450129 1872 leader_election.cc:290] T a54af99bc10045e8b0b6868e5aed98b4 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:23.450719 859 leader_election.cc:343] T c58edaa76ebf47dfb1d9d5fab6de7dfa P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:23.450847 859 leader_election.cc:304] T c58edaa76ebf47dfb1d9d5fab6de7dfa P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:23.451217 1981 raft_consensus.cc:2747] T c58edaa76ebf47dfb1d9d5fab6de7dfa P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:23.452102 859 leader_election.cc:343] T ad45c90ba0da43c9860745e534664a98 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:23.452198 859 leader_election.cc:304] T ad45c90ba0da43c9860745e534664a98 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:23.452536 1981 raft_consensus.cc:2747] T ad45c90ba0da43c9860745e534664a98 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:23.453008 859 leader_election.cc:343] T 2599f98abfdc4871b573edfc2ad97d22 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:23.453099 859 leader_election.cc:304] T 2599f98abfdc4871b573edfc2ad97d22 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:23.453552 1981 raft_consensus.cc:2747] T 2599f98abfdc4871b573edfc2ad97d22 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:23.454141 992 leader_election.cc:336] T a54af99bc10045e8b0b6868e5aed98b4 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:23.454411 992 consensus_peers.cc:489] T f06a77ecc841462396a8d5026c7327f4 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250901 01:52:23.454566 992 consensus_peers.cc:489] T 831e42de95854541a49ae6b3e039d3e9 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250901 01:52:23.454680 992 leader_election.cc:336] T 7b16ca1daf394a6da65a4af8da82ec33 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:23.455237 992 leader_election.cc:336] T d58f69fad01f403eab74a47d3695ed83 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:23.456566 917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7b16ca1daf394a6da65a4af8da82ec33" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:23.456797 917 raft_consensus.cc:3058] T 7b16ca1daf394a6da65a4af8da82ec33 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:23.458571 1981 raft_consensus.cc:491] T e87d1281f35648b0b115b67c95e244e7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:23.458688 1981 raft_consensus.cc:513] T e87d1281f35648b0b115b67c95e244e7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.459229 1981 leader_election.cc:290] T e87d1281f35648b0b115b67c95e244e7 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:23.461107 858 leader_election.cc:336] T e87d1281f35648b0b115b67c95e244e7 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:23.461977 917 raft_consensus.cc:2466] T 7b16ca1daf394a6da65a4af8da82ec33 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:23.460774 1483 catalog_manager.cc:5582] T 606fac5c6efe48d1877104b01c23ba5b P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: HEALTHY } } }
I20250901 01:52:23.462569 1483 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 606fac5c6efe48d1877104b01c23ba5b (table default.Table_7c12d66048a6470e8c227dc1d9f45fa4 [id=91472f220fe44297b79d3eec8b9b80e8]). Expected version 1 got 0
I20250901 01:52:23.462982 917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d58f69fad01f403eab74a47d3695ed83" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:23.463234 917 raft_consensus.cc:3058] T d58f69fad01f403eab74a47d3695ed83 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:23.466040 917 raft_consensus.cc:2466] T d58f69fad01f403eab74a47d3695ed83 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:23.468461 921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a54af99bc10045e8b0b6868e5aed98b4" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:23.469004 921 raft_consensus.cc:3058] T a54af99bc10045e8b0b6868e5aed98b4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:23.471647 921 raft_consensus.cc:2466] T a54af99bc10045e8b0b6868e5aed98b4 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:23.473273 1151 tablet_bootstrap.cc:492] T 49aaaec611784e6bac131efd58dbca51 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:23.473500 1151 ts_tablet_manager.cc:1397] T 49aaaec611784e6bac131efd58dbca51 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.054s user 0.000s sys 0.003s
I20250901 01:52:23.474282 1151 raft_consensus.cc:357] T 49aaaec611784e6bac131efd58dbca51 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.474560 1151 raft_consensus.cc:383] T 49aaaec611784e6bac131efd58dbca51 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:23.474673 1151 raft_consensus.cc:738] T 49aaaec611784e6bac131efd58dbca51 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:23.474888 1151 consensus_queue.cc:260] T 49aaaec611784e6bac131efd58dbca51 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.475474 1151 ts_tablet_manager.cc:1428] T 49aaaec611784e6bac131efd58dbca51 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250901 01:52:23.477043 1150 tablet_bootstrap.cc:492] T 7954393c8c1c4457ab8bb723c03a9c8b P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:23.477210 1150 ts_tablet_manager.cc:1397] T 7954393c8c1c4457ab8bb723c03a9c8b P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.063s user 0.015s sys 0.015s
I20250901 01:52:23.477794 1151 tablet_bootstrap.cc:492] T 579b9cdf819f4609a532e96bec093fd5 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:23.478154 1150 raft_consensus.cc:357] T 7954393c8c1c4457ab8bb723c03a9c8b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.478358 1150 raft_consensus.cc:383] T 7954393c8c1c4457ab8bb723c03a9c8b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:23.478416 1150 raft_consensus.cc:738] T 7954393c8c1c4457ab8bb723c03a9c8b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:23.478725 1150 consensus_queue.cc:260] T 7954393c8c1c4457ab8bb723c03a9c8b P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:23.479267 992 consensus_peers.cc:489] T 756cfe84a517459ca015aecf5cd81571 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250901 01:52:23.479828 1150 ts_tablet_manager.cc:1428] T 7954393c8c1c4457ab8bb723c03a9c8b P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20250901 01:52:23.480777 1761 raft_consensus.cc:491] T 822382a1781b4304bce29355ce83730e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:23.480958 1761 raft_consensus.cc:513] T 822382a1781b4304bce29355ce83730e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.481537 1761 leader_election.cc:290] T 822382a1781b4304bce29355ce83730e P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:23.481734 992 leader_election.cc:304] T 7b16ca1daf394a6da65a4af8da82ec33 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:23.481927 1761 raft_consensus.cc:2802] T 7b16ca1daf394a6da65a4af8da82ec33 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:23.482014 1761 raft_consensus.cc:695] T 7b16ca1daf394a6da65a4af8da82ec33 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:23.483131 1150 tablet_bootstrap.cc:492] T afc002ae2486476b9208e1cf8215157c P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:23.483764 1761 consensus_queue.cc:237] T 7b16ca1daf394a6da65a4af8da82ec33 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.484874 992 leader_election.cc:304] T a54af99bc10045e8b0b6868e5aed98b4 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:23.485365 1150 tablet_bootstrap.cc:654] T afc002ae2486476b9208e1cf8215157c P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:23.485580 1151 tablet_bootstrap.cc:654] T 579b9cdf819f4609a532e96bec093fd5 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:23.488067 1990 raft_consensus.cc:2802] T a54af99bc10045e8b0b6868e5aed98b4 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:23.488212 1990 raft_consensus.cc:695] T a54af99bc10045e8b0b6868e5aed98b4 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:23.488526 1990 consensus_queue.cc:237] T a54af99bc10045e8b0b6868e5aed98b4 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:23.489969 992 leader_election.cc:304] T d58f69fad01f403eab74a47d3695ed83 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:23.490401 1990 raft_consensus.cc:2802] T d58f69fad01f403eab74a47d3695ed83 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:23.490649 1990 raft_consensus.cc:695] T d58f69fad01f403eab74a47d3695ed83 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:23.490942 1990 consensus_queue.cc:237] T d58f69fad01f403eab74a47d3695ed83 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.492580 1058 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e87d1281f35648b0b115b67c95e244e7" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:23.497634 917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "822382a1781b4304bce29355ce83730e" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:23.497967 917 raft_consensus.cc:2466] T 822382a1781b4304bce29355ce83730e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
W20250901 01:52:23.498642 859 leader_election.cc:343] T e87d1281f35648b0b115b67c95e244e7 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:23.498816 859 leader_election.cc:304] T e87d1281f35648b0b115b67c95e244e7 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:23.499363 1981 raft_consensus.cc:2747] T e87d1281f35648b0b115b67c95e244e7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:23.510959 992 leader_election.cc:336] T d58f69fad01f403eab74a47d3695ed83 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:23.511302 992 leader_election.cc:336] T a54af99bc10045e8b0b6868e5aed98b4 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:23.511519 992 leader_election.cc:336] T 7b16ca1daf394a6da65a4af8da82ec33 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:23.512135 992 leader_election.cc:304] T 822382a1781b4304bce29355ce83730e P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:23.512923 1990 raft_consensus.cc:2802] T 822382a1781b4304bce29355ce83730e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:23.513063 1990 raft_consensus.cc:491] T 822382a1781b4304bce29355ce83730e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:23.513401 1990 raft_consensus.cc:3058] T 822382a1781b4304bce29355ce83730e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:23.516088 1492 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 606fac5c6efe48d1877104b01c23ba5b (table default.Table_7c12d66048a6470e8c227dc1d9f45fa4 [id=91472f220fe44297b79d3eec8b9b80e8]). Expected version 1 got 0
I20250901 01:52:23.516286 1990 raft_consensus.cc:513] T 822382a1781b4304bce29355ce83730e P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.516961 1990 leader_election.cc:290] T 822382a1781b4304bce29355ce83730e P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:23.517248 1150 tablet_bootstrap.cc:492] T afc002ae2486476b9208e1cf8215157c P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:23.517395 1150 ts_tablet_manager.cc:1397] T afc002ae2486476b9208e1cf8215157c P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.034s user 0.002s sys 0.013s
I20250901 01:52:23.518370 1150 raft_consensus.cc:357] T afc002ae2486476b9208e1cf8215157c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.518591 1150 raft_consensus.cc:383] T afc002ae2486476b9208e1cf8215157c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:23.518652 1150 raft_consensus.cc:738] T afc002ae2486476b9208e1cf8215157c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:23.518729 917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "822382a1781b4304bce29355ce83730e" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:23.518934 917 raft_consensus.cc:3058] T 822382a1781b4304bce29355ce83730e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:23.518895 1150 consensus_queue.cc:260] T afc002ae2486476b9208e1cf8215157c P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.521334 1103 heartbeater.cc:499] Master 127.0.104.60:44231 was elected leader, sending a full tablet report...
I20250901 01:52:23.521689 917 raft_consensus.cc:2466] T 822382a1781b4304bce29355ce83730e P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:23.522589 1150 ts_tablet_manager.cc:1428] T afc002ae2486476b9208e1cf8215157c P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
I20250901 01:52:23.522936 1150 tablet_bootstrap.cc:492] T 33f637f6a8134215bb1995ec38440073 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:23.525214 1150 tablet_bootstrap.cc:654] T 33f637f6a8134215bb1995ec38440073 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:23.525944 969 heartbeater.cc:499] Master 127.0.104.60:44231 was elected leader, sending a full tablet report...
I20250901 01:52:23.530843 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9ceeaada147147bd9aea2b14e93413ff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:23.532052 1490 master_service.cc:948] ReplaceTablet: received request to replace tablet 17e8340dbd1a406cb782dad3fe88cc7e from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:23.533834 1489 master_service.cc:948] ReplaceTablet: received request to replace tablet ded33e0950b544509af11739353c620d from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:23.534076 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_44eb9cf1ae1548eda81ed4b077032b54"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:23.535137 1482 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_8cd9c1e6a8e74232b32d36451269562a" } new_table_name: "default.Table_1bdd9041d74045cab6f23de11bf4dd09" modify_external_catalogs: true
I20250901 01:52:23.535406 1488 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_5a32379f33204b9e9b2dfe82995134dc" } modify_external_catalogs: true
I20250901 01:52:23.535594 1488 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_5a32379f33204b9e9b2dfe82995134dc" } modify_external_catalogs: true
I20250901 01:52:23.537171 1981 raft_consensus.cc:491] T 3d90c6ef06d1436d8e2bb1333e7d9985 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:23.537329 1981 raft_consensus.cc:513] T 3d90c6ef06d1436d8e2bb1333e7d9985 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.537568 1486 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_1c8baaaa506548bd85b9f6a7483e5c7a" } new_table_name: "default.Table_2ceaff93c8f44f00aab53e48d8466dfd" modify_external_catalogs: true
I20250901 01:52:23.538035 1981 leader_election.cc:290] T 3d90c6ef06d1436d8e2bb1333e7d9985 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:23.539036 1485 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_277b91103c4d41d2a9f9424f99082b29" } new_table_name: "default.Table_63d0f2ead9d946d39ce40162ebd769d6" modify_external_catalogs: true
I20250901 01:52:23.540346 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_0c3199ac9a044579b83ab238ce44ec94"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:23.541294 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5cbefcad1c684442adca3294331bf53c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:23.541965 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9dfaa5f57fb34061a360a0e17bdfe2d0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:23.542619 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e4a0ca41c94e49cd90bc95feee482b56"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:23.543320 1484 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_e37cc89ad7ee4d96852b121d67b06248" } new_table_name: "default.Table_23fd89096be149b791975d89d65bc280" modify_external_catalogs: true
I20250901 01:52:23.547248 1151 tablet_bootstrap.cc:492] T 579b9cdf819f4609a532e96bec093fd5 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:23.547402 1151 ts_tablet_manager.cc:1397] T 579b9cdf819f4609a532e96bec093fd5 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.070s user 0.001s sys 0.002s
I20250901 01:52:23.548179 1151 raft_consensus.cc:357] T 579b9cdf819f4609a532e96bec093fd5 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.548952 1151 raft_consensus.cc:383] T 579b9cdf819f4609a532e96bec093fd5 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:23.549019 1151 raft_consensus.cc:738] T 579b9cdf819f4609a532e96bec093fd5 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:23.549211 1151 consensus_queue.cc:260] T 579b9cdf819f4609a532e96bec093fd5 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.549417 1761 raft_consensus.cc:491] T a59ac57aebe24860aa02f53ce0485910 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:23.549516 1761 raft_consensus.cc:513] T a59ac57aebe24860aa02f53ce0485910 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:23.550004 1761 leader_election.cc:290] T a59ac57aebe24860aa02f53ce0485910 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:23.557090 1491 master_service.cc:948] ReplaceTablet: received request to replace tablet 088ef4861dc14f81ae470c3aaee5b4e1 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:23.557272 1151 ts_tablet_manager.cc:1428] T 579b9cdf819f4609a532e96bec093fd5 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20250901 01:52:23.560652 1492 master_service.cc:948] ReplaceTablet: received request to replace tablet 56475ba491a24301a534304811f30140 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:23.563819 1490 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 17e8340dbd1a406cb782dad3fe88cc7e
W20250901 01:52:23.564169 1490 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 17e8340dbd1a406cb782dad3fe88cc7e: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:23.564615 1490 catalog_manager.cc:6652] ReplaceTablet: tablet 17e8340dbd1a406cb782dad3fe88cc7e deleted and replaced by tablet e2ed32b3292141278af28026528e96c2
I20250901 01:52:23.563827 1489 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet ded33e0950b544509af11739353c620d
W20250901 01:52:23.565845 1489 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet ded33e0950b544509af11739353c620d: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:23.566213 1489 catalog_manager.cc:6652] ReplaceTablet: tablet ded33e0950b544509af11739353c620d deleted and replaced by tablet cc3b8736b9094d7e892c80da4d5b20f6
I20250901 01:52:23.566323 1151 tablet_bootstrap.cc:492] T 25ce0fe8c9544f828694184d19d13df9 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
W20250901 01:52:23.567509 992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:40731, user_credentials={real_user=slave}} blocked reactor thread for 44932us
W20250901 01:52:23.568006 992 leader_election.cc:336] T 822382a1781b4304bce29355ce83730e P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:23.568086 1483 master_service.cc:948] ReplaceTablet: received request to replace tablet 74b709cb5ec64006b2f12bead8217d80 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:23.568378 992 leader_election.cc:304] T 822382a1781b4304bce29355ce83730e P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:23.571686 1761 raft_consensus.cc:2802] T 822382a1781b4304bce29355ce83730e P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:23.571822 1761 raft_consensus.cc:695] T 822382a1781b4304bce29355ce83730e P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:23.572070 1761 consensus_queue.cc:237] T 822382a1781b4304bce29355ce83730e P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.581998 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d90c6ef06d1436d8e2bb1333e7d9985" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:23.582191 1151 tablet_bootstrap.cc:654] T 25ce0fe8c9544f828694184d19d13df9 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:23.584369 992 leader_election.cc:336] T 822382a1781b4304bce29355ce83730e P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:23.585225 1486 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fcb7b5b7d63d4f0b906fea97cb548de2 consensus state has no leader
W20250901 01:52:23.586469 992 leader_election.cc:343] T a59ac57aebe24860aa02f53ce0485910 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: a59ac57aebe24860aa02f53ce0485910
I20250901 01:52:23.587137 1488 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 26c8ed083cea4dc0acca80d00b11b29b
W20250901 01:52:23.587356 1488 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 26c8ed083cea4dc0acca80d00b11b29b: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
W20250901 01:52:23.590730 1485 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 61ddcf86c9f34726939cc116434f0822 consensus state has no leader
W20250901 01:52:23.592346 1482 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8ed6616ce54548ba83ba299c4ecf3c38 consensus state has no leader
I20250901 01:52:23.592747 1482 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_4401d83f75f8464da49d5b811d7dd89d" } modify_external_catalogs: true
I20250901 01:52:23.592967 1482 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_4401d83f75f8464da49d5b811d7dd89d" } modify_external_catalogs: true
I20250901 01:52:23.592993 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d4d52bbc912d428fa018d1044927c76f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:23.593909 1490 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_0e73f80424cd48408f18c2eff1411706" } modify_external_catalogs: true
I20250901 01:52:23.594120 1490 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_0e73f80424cd48408f18c2eff1411706" } modify_external_catalogs: true
I20250901 01:52:23.595548 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_241e1d06219f4aa2963f2c326cb82a0a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:23.596388 1483 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_fbc57d539a2b41e59dfc5369ac8c92e1" } modify_external_catalogs: true
I20250901 01:52:23.596611 1483 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_fbc57d539a2b41e59dfc5369ac8c92e1" } modify_external_catalogs: true
I20250901 01:52:23.596338 1489 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_d9e28b97a2e94ca8bf0ee8a01acd76e8" } modify_external_catalogs: true
I20250901 01:52:23.596949 1981 raft_consensus.cc:491] T f13b972e25214f168421439fe3943e8a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:23.597080 1489 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_d9e28b97a2e94ca8bf0ee8a01acd76e8" } modify_external_catalogs: true
I20250901 01:52:23.597117 1981 raft_consensus.cc:513] T f13b972e25214f168421439fe3943e8a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.597926 1981 leader_election.cc:290] T f13b972e25214f168421439fe3943e8a P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:23.598551 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_429d55410fd14d2e9e5af8fae4d60c34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:23.599056 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_49d02f1d1ad74c7c86afe53720d117df"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:23.599500 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c7ccb6cfd132489694ff25ba21c254c7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:23.599658 1981 raft_consensus.cc:491] T 081e2630e284498aa819f85bf8326315 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:23.599821 1981 raft_consensus.cc:513] T 081e2630e284498aa819f85bf8326315 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.600241 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7762f8fcb21245e1bf1bde83dac75087"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:23.600518 1981 leader_election.cc:290] T 081e2630e284498aa819f85bf8326315 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:23.600936 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_6d0deba836f34a2eb37e21ab5a1452de"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:23.601379 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7d966dbb133a41558f6724ad2f4a6e1b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:23.601679 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c05d7d175e454d989b4e669e8c6e3325"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:23.602268 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ac30f7b444da4acfb3e7b8246fa07c90"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:23.602404 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_aab1f77d071d4a07b11cd121ab90c3e9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:23.604377 1483 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_1186b4165684445da50e713ed6a227f9" } modify_external_catalogs: true
I20250901 01:52:23.604466 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b304df88f34540a98dc27e701bbbcae5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:23.604553 1483 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_1186b4165684445da50e713ed6a227f9" } modify_external_catalogs: true
I20250901 01:52:23.609864 1993 raft_consensus.cc:491] T 121803e79ab64ec0b948b448646b9d0a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:23.610009 1993 raft_consensus.cc:513] T 121803e79ab64ec0b948b448646b9d0a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:23.610589 1993 leader_election.cc:290] T 121803e79ab64ec0b948b448646b9d0a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:23.614329 1491 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 088ef4861dc14f81ae470c3aaee5b4e1
W20250901 01:52:23.614565 1491 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 088ef4861dc14f81ae470c3aaee5b4e1: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:23.614900 1491 catalog_manager.cc:6652] ReplaceTablet: tablet 088ef4861dc14f81ae470c3aaee5b4e1 deleted and replaced by tablet 6ce9521b925b4f97bb4a810ce77e147e
W20250901 01:52:23.616714 1474 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 61ddcf86c9f34726939cc116434f0822 (table default.Table_63d0f2ead9d946d39ce40162ebd769d6 [id=f9dcf4937b1143248bfa9d88053fce17], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet 61ddcf86c9f34726939cc116434f0822 consensus state has no leader
W20250901 01:52:23.617482 859 leader_election.cc:343] T 3d90c6ef06d1436d8e2bb1333e7d9985 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:23.617859 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_da3d72a5262b40cabba94309d04e4752"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:23.618568 1151 tablet_bootstrap.cc:492] T 25ce0fe8c9544f828694184d19d13df9 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:23.618723 1151 ts_tablet_manager.cc:1397] T 25ce0fe8c9544f828694184d19d13df9 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.052s user 0.002s sys 0.002s
I20250901 01:52:23.619539 1151 raft_consensus.cc:357] T 25ce0fe8c9544f828694184d19d13df9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:23.619732 1151 raft_consensus.cc:383] T 25ce0fe8c9544f828694184d19d13df9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:23.619797 1151 raft_consensus.cc:738] T 25ce0fe8c9544f828694184d19d13df9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:23.619742 1488 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_d51ef24ec0f846c7ad212360eb478cd7" } new_table_name: "default.Table_8445fcd774874cadaf4acc3581eee417" modify_external_catalogs: true
I20250901 01:52:23.619994 1151 consensus_queue.cc:260] T 25ce0fe8c9544f828694184d19d13df9 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:23.624797 917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "121803e79ab64ec0b948b448646b9d0a" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:23.625042 917 raft_consensus.cc:2466] T 121803e79ab64ec0b948b448646b9d0a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:23.625944 1058 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "081e2630e284498aa819f85bf8326315" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:23.626267 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f13b972e25214f168421439fe3943e8a" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:23.626658 992 leader_election.cc:336] T a59ac57aebe24860aa02f53ce0485910 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:23.628254 1492 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 56475ba491a24301a534304811f30140
I20250901 01:52:23.628420 992 leader_election.cc:304] T a59ac57aebe24860aa02f53ce0485910 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
W20250901 01:52:23.628507 1492 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 56475ba491a24301a534304811f30140: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:23.629005 1492 catalog_manager.cc:6652] ReplaceTablet: tablet 56475ba491a24301a534304811f30140 deleted and replaced by tablet 7317eb7848ee44698038a4b404d4bf21
I20250901 01:52:23.631531 1151 ts_tablet_manager.cc:1428] T 25ce0fe8c9544f828694184d19d13df9 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20250901 01:52:23.631734 1872 raft_consensus.cc:2747] T a59ac57aebe24860aa02f53ce0485910 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:23.632256 1482 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 764bfeb2a7d14c40815b5ec1b07bdd62
W20250901 01:52:23.632458 1482 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 764bfeb2a7d14c40815b5ec1b07bdd62: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:23.632762 1490 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet becbaae4fd7e44faa679e9479465a7c4
W20250901 01:52:23.632938 1490 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet becbaae4fd7e44faa679e9479465a7c4: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:23.641515 1151 tablet_bootstrap.cc:492] T 7f64ebd080944190be2e64d02d70a8a7 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:23.642727 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5c2776bec11d450899e073a190c110de"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:23.643370 1151 tablet_bootstrap.cc:654] T 7f64ebd080944190be2e64d02d70a8a7 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:23.645323 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_bbe7fb184e114abeabcde2163a85aec3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:23.645323 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_28d1bbf3141946648598d6433aa3baa0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:23.646494 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ab5eb245b99e40c1b8de3ef6bebfb6df"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:23.647079 1872 consensus_queue.cc:1035] T 3a8231590ea643b0a1f83eec4d9a8191 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:23.647634 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_827864b3dc3f4b4782687a343ddecfdc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:23.647931 992 leader_election.cc:304] T 121803e79ab64ec0b948b448646b9d0a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:23.648176 1872 raft_consensus.cc:2802] T 121803e79ab64ec0b948b448646b9d0a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:23.648258 1872 raft_consensus.cc:491] T 121803e79ab64ec0b948b448646b9d0a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:23.648324 1872 raft_consensus.cc:3058] T 121803e79ab64ec0b948b448646b9d0a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:23.649008 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_09df5f19cb4d42b39bd866490d1c893c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:23.650956 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_418a942b23704944b252277b6155766d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:23.651363 1872 raft_consensus.cc:513] T 121803e79ab64ec0b948b448646b9d0a P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:23.651824 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d3c0ac29555a4656b4a1e37a34d23109"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:23.651953 1872 leader_election.cc:290] T 121803e79ab64ec0b948b448646b9d0a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:23.652349 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ac1d3177ad9942ea9d213f4655833275"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:23.659858 859 leader_election.cc:343] T 081e2630e284498aa819f85bf8326315 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
W20250901 01:52:23.660089 859 leader_election.cc:343] T f13b972e25214f168421439fe3943e8a P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:23.661129 1489 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet eac50cbae691416e903cb2cbc907f96d
W20250901 01:52:23.661339 1489 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet eac50cbae691416e903cb2cbc907f96d: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:23.666262 921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "121803e79ab64ec0b948b448646b9d0a" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:23.666486 921 raft_consensus.cc:3058] T 121803e79ab64ec0b948b448646b9d0a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:23.670464 921 raft_consensus.cc:2466] T 121803e79ab64ec0b948b448646b9d0a P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
W20250901 01:52:23.673903 992 leader_election.cc:336] T 121803e79ab64ec0b948b448646b9d0a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:23.674681 992 leader_election.cc:304] T 121803e79ab64ec0b948b448646b9d0a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:23.681542 1872 raft_consensus.cc:2802] T 121803e79ab64ec0b948b448646b9d0a P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:23.681705 1872 raft_consensus.cc:695] T 121803e79ab64ec0b948b448646b9d0a P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:23.681998 1872 consensus_queue.cc:237] T 121803e79ab64ec0b948b448646b9d0a P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:23.694178 1151 tablet_bootstrap.cc:492] T 7f64ebd080944190be2e64d02d70a8a7 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:23.695363 1981 raft_consensus.cc:491] T 71941b815d6840478c5acc1cbb187cc9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:23.695554 1981 raft_consensus.cc:513] T 71941b815d6840478c5acc1cbb187cc9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.696362 1981 leader_election.cc:290] T 71941b815d6840478c5acc1cbb187cc9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:23.700539 1981 raft_consensus.cc:491] T ea846f88b3534beebd0001b7f17b5ca9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:23.700659 1981 raft_consensus.cc:513] T ea846f88b3534beebd0001b7f17b5ca9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.701191 1981 leader_election.cc:290] T ea846f88b3534beebd0001b7f17b5ca9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:23.704195 1151 ts_tablet_manager.cc:1397] T 7f64ebd080944190be2e64d02d70a8a7 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.063s user 0.002s sys 0.002s
I20250901 01:52:23.704733 1873 raft_consensus.cc:491] T 2240302d9178445d8700e35f5ea274f7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:23.704871 1873 raft_consensus.cc:513] T 2240302d9178445d8700e35f5ea274f7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:23.705153 858 leader_election.cc:336] T ea846f88b3534beebd0001b7f17b5ca9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:23.705327 858 leader_election.cc:336] T 3d90c6ef06d1436d8e2bb1333e7d9985 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:23.705423 858 leader_election.cc:304] T 3d90c6ef06d1436d8e2bb1333e7d9985 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:23.705478 1873 leader_election.cc:290] T 2240302d9178445d8700e35f5ea274f7 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:23.705585 1873 raft_consensus.cc:491] T af2c4bcbb00741f789edee109252d8cb P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:23.705667 1981 raft_consensus.cc:2747] T 3d90c6ef06d1436d8e2bb1333e7d9985 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:23.705667 1873 raft_consensus.cc:513] T af2c4bcbb00741f789edee109252d8cb P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:23.706015 858 leader_election.cc:336] T f13b972e25214f168421439fe3943e8a P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:23.706104 1873 leader_election.cc:290] T af2c4bcbb00741f789edee109252d8cb P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:23.706117 858 leader_election.cc:304] T f13b972e25214f168421439fe3943e8a P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:23.706331 1981 raft_consensus.cc:2747] T f13b972e25214f168421439fe3943e8a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:23.706593 858 leader_election.cc:336] T 081e2630e284498aa819f85bf8326315 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:23.706729 858 leader_election.cc:304] T 081e2630e284498aa819f85bf8326315 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:23.706935 1981 raft_consensus.cc:2747] T 081e2630e284498aa819f85bf8326315 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:23.707201 858 leader_election.cc:336] T 71941b815d6840478c5acc1cbb187cc9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:23.708686 992 leader_election.cc:336] T 121803e79ab64ec0b948b448646b9d0a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:23.711035 1151 raft_consensus.cc:357] T 7f64ebd080944190be2e64d02d70a8a7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.711189 1151 raft_consensus.cc:383] T 7f64ebd080944190be2e64d02d70a8a7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:23.711236 1151 raft_consensus.cc:738] T 7f64ebd080944190be2e64d02d70a8a7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:23.711422 1151 consensus_queue.cc:260] T 7f64ebd080944190be2e64d02d70a8a7 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.712723 921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "af2c4bcbb00741f789edee109252d8cb" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:23.712924 921 raft_consensus.cc:2466] T af2c4bcbb00741f789edee109252d8cb P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:23.713311 917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2240302d9178445d8700e35f5ea274f7" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:23.713527 917 raft_consensus.cc:2466] T 2240302d9178445d8700e35f5ea274f7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:23.714725 992 leader_election.cc:304] T af2c4bcbb00741f789edee109252d8cb P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:23.715075 1873 raft_consensus.cc:2802] T af2c4bcbb00741f789edee109252d8cb P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:23.715144 1873 raft_consensus.cc:491] T af2c4bcbb00741f789edee109252d8cb P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:23.715198 1873 raft_consensus.cc:3058] T af2c4bcbb00741f789edee109252d8cb P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:23.716660 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_baab4083a70e4a44b6bc7f5b64a9dae1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:23.717885 1873 raft_consensus.cc:513] T af2c4bcbb00741f789edee109252d8cb P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.718093 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_07bef6ee9d004d8281741416eac68004"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:23.718425 1873 leader_election.cc:290] T af2c4bcbb00741f789edee109252d8cb P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:23.719008 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e1541b95b7be4b88adb13e80559ecf42"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:23.719144 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a09d49f6c43248349389b48a4a878154"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:23.719743 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_38b0e04cd3354d36aa0475e99d91df1b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:23.719782 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_715cafa3affb4b81b40dac89ea1cfd46"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:23.720844 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_39ecbf9c1d3d41b5ae375dcb073767fc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:23.723098 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e77af48b67c743c0bc87af72f8a3b279"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:23.723973 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f41841e273624410be0316b46e4b2e4b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:23.724368 992 leader_election.cc:304] T 2240302d9178445d8700e35f5ea274f7 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:23.724910 1151 ts_tablet_manager.cc:1428] T 7f64ebd080944190be2e64d02d70a8a7 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20250901 01:52:23.725210 1151 tablet_bootstrap.cc:492] T eac50cbae691416e903cb2cbc907f96d P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:23.725867 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_59d1b2dd09d8497aaf00156299b05a9b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:23.727006 1151 tablet_bootstrap.cc:654] T eac50cbae691416e903cb2cbc907f96d P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:23.735600 1981 raft_consensus.cc:491] T 5e2472ee5d6e4d3bbf1f24027ba531ca P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:23.735801 1981 raft_consensus.cc:513] T 5e2472ee5d6e4d3bbf1f24027ba531ca P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.736388 1981 leader_election.cc:290] T 5e2472ee5d6e4d3bbf1f24027ba531ca P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:23.738225 858 leader_election.cc:336] T 5e2472ee5d6e4d3bbf1f24027ba531ca P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:23.741667 1993 raft_consensus.cc:2802] T 2240302d9178445d8700e35f5ea274f7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:23.741837 1993 raft_consensus.cc:491] T 2240302d9178445d8700e35f5ea274f7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:23.741901 1993 raft_consensus.cc:3058] T 2240302d9178445d8700e35f5ea274f7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:23.742261 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_929002c933404ff3acd18c7daf489d94"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:23.744700 1993 raft_consensus.cc:513] T 2240302d9178445d8700e35f5ea274f7 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.745311 1993 leader_election.cc:290] T 2240302d9178445d8700e35f5ea274f7 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:23.745937 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "71941b815d6840478c5acc1cbb187cc9" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:23.746495 1058 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ea846f88b3534beebd0001b7f17b5ca9" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:23.747218 1058 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e2472ee5d6e4d3bbf1f24027ba531ca" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:23.748086 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1eaedc4ea6784ec2b9f823dd872dc61a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:23.750733 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_fd37d5dfbd4e4a7396a79ea630573f5d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:23.752280 917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "af2c4bcbb00741f789edee109252d8cb" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:23.752506 917 raft_consensus.cc:3058] T af2c4bcbb00741f789edee109252d8cb P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:23.752988 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_19b3c1b404554e5cbc29b2252ec8a1b2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:23.754765 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_da7e811b08154a3aac118e54dcfff3d6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:23.755415 917 raft_consensus.cc:2466] T af2c4bcbb00741f789edee109252d8cb P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:23.756332 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f76aad33e5ca45b9afa2c41bdef753a4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:23.757934 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_6c177e6829bb455e8bc804acef446c2c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:23.760253 917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2240302d9178445d8700e35f5ea274f7" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:23.760505 917 raft_consensus.cc:3058] T 2240302d9178445d8700e35f5ea274f7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:23.760440 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_07d5cffc48a743c5a1508746d7f0abec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:23.762111 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_347a3e78da8a4d06bd0af0515bcbc430"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:23.762334 1981 raft_consensus.cc:491] T f6a8b425968e49deaecfa3adacfd3f1f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:23.762459 1981 raft_consensus.cc:513] T f6a8b425968e49deaecfa3adacfd3f1f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.763101 1981 leader_election.cc:290] T f6a8b425968e49deaecfa3adacfd3f1f P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:23.764333 858 leader_election.cc:336] T f6a8b425968e49deaecfa3adacfd3f1f P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:23.764356 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_85c53eacdafe4eb5be72fe5da51f1ad8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:23.764382 917 raft_consensus.cc:2466] T 2240302d9178445d8700e35f5ea274f7 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
W20250901 01:52:23.765218 859 leader_election.cc:343] T 71941b815d6840478c5acc1cbb187cc9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:23.765358 859 leader_election.cc:304] T 71941b815d6840478c5acc1cbb187cc9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:23.765735 1981 raft_consensus.cc:2747] T 71941b815d6840478c5acc1cbb187cc9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:23.766181 859 leader_election.cc:343] T ea846f88b3534beebd0001b7f17b5ca9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:23.766287 859 leader_election.cc:304] T ea846f88b3534beebd0001b7f17b5ca9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:23.766659 1981 raft_consensus.cc:2747] T ea846f88b3534beebd0001b7f17b5ca9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:23.776007 1036 tablet_service.cc:1515] Processing DeleteTablet for tablet 2240302d9178445d8700e35f5ea274f7 with delete_type TABLET_DATA_DELETED (replaced by tablet b13afacd37474781877b9f2cd20b313a) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:23.776526 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 17e8340dbd1a406cb782dad3fe88cc7e with delete_type TABLET_DATA_DELETED (replaced by tablet e2ed32b3292141278af28026528e96c2) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:23.776811 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet f06a77ecc841462396a8d5026c7327f4 with delete_type TABLET_DATA_DELETED (replaced by tablet f3962b471bdb49aea6e9fa95f1f9dfe2) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:23.776872 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet ded33e0950b544509af11739353c620d with delete_type TABLET_DATA_DELETED (replaced by tablet cc3b8736b9094d7e892c80da4d5b20f6) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:23.777033 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet becbaae4fd7e44faa679e9479465a7c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:23 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:23.777070 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 26c8ed083cea4dc0acca80d00b11b29b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:23 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:23.777227 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 088ef4861dc14f81ae470c3aaee5b4e1 with delete_type TABLET_DATA_DELETED (replaced by tablet 6ce9521b925b4f97bb4a810ce77e147e) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:23.777959 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet 56475ba491a24301a534304811f30140 with delete_type TABLET_DATA_DELETED (replaced by tablet 7317eb7848ee44698038a4b404d4bf21) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:23.778164 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 764bfeb2a7d14c40815b5ec1b07bdd62 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:23 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:23.778908 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet eac50cbae691416e903cb2cbc907f96d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:23 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:23.781266 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_746c5c51a1694d2091bc5c87361d3a47"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:23.782402 1981 raft_consensus.cc:491] T f3962b471bdb49aea6e9fa95f1f9dfe2 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:23.782549 1981 raft_consensus.cc:513] T f3962b471bdb49aea6e9fa95f1f9dfe2 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:23.782703 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_6637fb8bbb0d44ea9514dd10c933c156"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:23.783166 1981 leader_election.cc:290] T f3962b471bdb49aea6e9fa95f1f9dfe2 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:23.784520 858 leader_election.cc:336] T f3962b471bdb49aea6e9fa95f1f9dfe2 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:23.787128 2113 tablet_replica.cc:331] T 2240302d9178445d8700e35f5ea274f7 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:23.787415 2113 raft_consensus.cc:2241] T 2240302d9178445d8700e35f5ea274f7 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:23.787663 2113 raft_consensus.cc:2270] T 2240302d9178445d8700e35f5ea274f7 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:23.787459 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_fdc346a3b0844dfd8909dd8bf8f0768d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:23.790431 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_750c7cfde0564e03997a2608f7f55bf3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:23.794113 859 leader_election.cc:343] T 5e2472ee5d6e4d3bbf1f24027ba531ca P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
W20250901 01:52:23.797144 992 consensus_peers.cc:489] T 3a8231590ea643b0a1f83eec4d9a8191 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:23.799309 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_36734f303b7046e787c59debf5e6c950"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:23.800840 992 leader_election.cc:336] T af2c4bcbb00741f789edee109252d8cb P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:23.804762 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a0f24b0e0b4a4a5cbb2ffa14d67c9ef6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:23.794240 859 leader_election.cc:304] T 5e2472ee5d6e4d3bbf1f24027ba531ca P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:23.806828 1981 raft_consensus.cc:2747] T 5e2472ee5d6e4d3bbf1f24027ba531ca P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:23.807113 1150 tablet_bootstrap.cc:492] T 33f637f6a8134215bb1995ec38440073 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:23.807247 1150 ts_tablet_manager.cc:1397] T 33f637f6a8134215bb1995ec38440073 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.284s user 0.042s sys 0.066s
I20250901 01:52:23.808595 1150 raft_consensus.cc:357] T 33f637f6a8134215bb1995ec38440073 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.808804 1150 raft_consensus.cc:383] T 33f637f6a8134215bb1995ec38440073 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:23.808859 1150 raft_consensus.cc:738] T 33f637f6a8134215bb1995ec38440073 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:23.809120 1150 consensus_queue.cc:260] T 33f637f6a8134215bb1995ec38440073 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.809607 1150 ts_tablet_manager.cc:1428] T 33f637f6a8134215bb1995ec38440073 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250901 01:52:23.809904 1150 tablet_bootstrap.cc:492] T 6789505328b1409ea288e8b4eaf94935 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:23.812299 1150 tablet_bootstrap.cc:654] T 6789505328b1409ea288e8b4eaf94935 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:23.814595 1150 tablet_bootstrap.cc:492] T 6789505328b1409ea288e8b4eaf94935 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:23.814764 1150 ts_tablet_manager.cc:1397] T 6789505328b1409ea288e8b4eaf94935 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250901 01:52:23.816269 1150 raft_consensus.cc:357] T 6789505328b1409ea288e8b4eaf94935 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.816511 1150 raft_consensus.cc:383] T 6789505328b1409ea288e8b4eaf94935 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:23.816591 1150 raft_consensus.cc:738] T 6789505328b1409ea288e8b4eaf94935 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:23.816845 1150 consensus_queue.cc:260] T 6789505328b1409ea288e8b4eaf94935 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.816982 885 tablet_service.cc:1515] Processing DeleteTablet for tablet f06a77ecc841462396a8d5026c7327f4 with delete_type TABLET_DATA_DELETED (replaced by tablet f3962b471bdb49aea6e9fa95f1f9dfe2) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:23.817606 1150 ts_tablet_manager.cc:1428] T 6789505328b1409ea288e8b4eaf94935 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250901 01:52:23.817863 897 tablet_service.cc:1515] Processing DeleteTablet for tablet 17e8340dbd1a406cb782dad3fe88cc7e with delete_type TABLET_DATA_DELETED (replaced by tablet e2ed32b3292141278af28026528e96c2) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:23.817999 1150 tablet_bootstrap.cc:492] T 482a9a2cb81f448bb2823892e35920d4 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:23.818127 887 tablet_service.cc:1515] Processing DeleteTablet for tablet ded33e0950b544509af11739353c620d with delete_type TABLET_DATA_DELETED (replaced by tablet cc3b8736b9094d7e892c80da4d5b20f6) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:23.818315 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 26c8ed083cea4dc0acca80d00b11b29b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:23 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:23.818491 898 tablet_service.cc:1515] Processing DeleteTablet for tablet 088ef4861dc14f81ae470c3aaee5b4e1 with delete_type TABLET_DATA_DELETED (replaced by tablet 6ce9521b925b4f97bb4a810ce77e147e) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:23.818655 901 tablet_service.cc:1515] Processing DeleteTablet for tablet 56475ba491a24301a534304811f30140 with delete_type TABLET_DATA_DELETED (replaced by tablet 7317eb7848ee44698038a4b404d4bf21) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:23.818819 904 tablet_service.cc:1515] Processing DeleteTablet for tablet 764bfeb2a7d14c40815b5ec1b07bdd62 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:23 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:23.819015 902 tablet_service.cc:1515] Processing DeleteTablet for tablet becbaae4fd7e44faa679e9479465a7c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:23 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:23.819207 894 tablet_service.cc:1515] Processing DeleteTablet for tablet eac50cbae691416e903cb2cbc907f96d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:23 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:23.824663 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a0dde0ae371e45edba46a0fe0d58c440"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:23.827649 1150 tablet_bootstrap.cc:654] T 482a9a2cb81f448bb2823892e35920d4 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:23.827087 1990 raft_consensus.cc:491] T 7f64ebd080944190be2e64d02d70a8a7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:23.835193 1990 raft_consensus.cc:513] T 7f64ebd080944190be2e64d02d70a8a7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.836066 1990 leader_election.cc:290] T 7f64ebd080944190be2e64d02d70a8a7 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:23.798436 1151 tablet_bootstrap.cc:492] T eac50cbae691416e903cb2cbc907f96d P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:23.836580 1151 ts_tablet_manager.cc:1397] T eac50cbae691416e903cb2cbc907f96d P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.111s user 0.003s sys 0.001s
W20250901 01:52:23.836845 992 leader_election.cc:336] T 2240302d9178445d8700e35f5ea274f7 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
E20250901 01:52:23.839650 1696 catalog_manager.cc:6205] Error selecting replicas for tablet be8dfbc466564842a2fdb4953eee939b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_75ba3ba70ca24a94920e7bce96190ff2'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:23.839866 1696 catalog_manager.cc:6205] Error selecting replicas for tablet e2ed32b3292141278af28026528e96c2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_50ae3908a7944f70949c54be8fbf1564'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:23.840025 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 15a4f99ae43745b9a3f60052d916a7bd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_23fb8cbcf1bd43fa8346f3c26c8b4c65'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:23.840198 1696 catalog_manager.cc:6205] Error selecting replicas for tablet a2ca69fabadd4ad69e1670cae7a0c503, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_97a5ec9ec34c4a89b0f917704b43e52b'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:23.840325 1696 catalog_manager.cc:6205] Error selecting replicas for tablet cc3b8736b9094d7e892c80da4d5b20f6, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_93afd37289414655980b22934695bd55'. Need at least 3 replicas, but only 2 tablet servers are available
W20250901 01:52:23.840565 992 consensus_peers.cc:489] T bb0cf5a6a27045a08df6845d7473184d P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250901 01:52:23.840659 992 consensus_peers.cc:489] T 40d74d7b5a3b4d1dae09eebd9e07a0c4 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:23.841187 992 leader_election.cc:304] T af2c4bcbb00741f789edee109252d8cb P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:23.843686 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f6a8b425968e49deaecfa3adacfd3f1f" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:23.844023 1058 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f3962b471bdb49aea6e9fa95f1f9dfe2" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:23.839355 1151 raft_consensus.cc:357] T eac50cbae691416e903cb2cbc907f96d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.845216 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_92c729a67dad4582bbe1aee344c02e3a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:23.846783 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3e76ae6fadb64bdf8912b4d926cad3cc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:23.847386 2118 tablet_replica.cc:331] T f06a77ecc841462396a8d5026c7327f4 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
E20250901 01:52:23.847679 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 18498c9cb09141d08c1c5c0854b63ebd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77f4810bc83c4d948531da2a45a3792e'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:23.847731 2118 raft_consensus.cc:2241] T f06a77ecc841462396a8d5026c7327f4 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:23.848590 2118 raft_consensus.cc:2270] T f06a77ecc841462396a8d5026c7327f4 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:23.850550 1151 raft_consensus.cc:383] T eac50cbae691416e903cb2cbc907f96d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:23.850693 1151 raft_consensus.cc:738] T eac50cbae691416e903cb2cbc907f96d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:23.851009 1151 consensus_queue.cc:260] T eac50cbae691416e903cb2cbc907f96d P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.851677 1491 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_2227d0e5eae04a08853aa24bfe533ee5" } modify_external_catalogs: true
I20250901 01:52:23.851886 1491 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_2227d0e5eae04a08853aa24bfe533ee5" } modify_external_catalogs: true
I20250901 01:52:23.852777 1484 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_15c9f4f02eb24474936a43d705f90f7b" } new_table_name: "default.Table_9978497447394e5694ae928412634ecb" modify_external_catalogs: true
I20250901 01:52:23.856798 1486 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_8f7db2ae75a24dfab71c2a95c6a0a404" } new_table_name: "default.Table_ecb9e8f638a043cc87c38afbebce250f" modify_external_catalogs: true
I20250901 01:52:23.865799 1489 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_da76b13a22bc47eea908959d5c1720df" } new_table_name: "default.Table_99855394cd424528b7f54ee20375788e" modify_external_catalogs: true
I20250901 01:52:23.871732 1482 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_b3ec3b285fac447f8c5c4a9ac4196628" } new_table_name: "default.Table_ce31a83b98ea4b698acbbdc1379419f5" modify_external_catalogs: true
W20250901 01:52:23.874936 1489 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e5e250e6c84d4248b857bf1478d47703 consensus state has no leader
I20250901 01:52:23.876547 1990 raft_consensus.cc:2802] T af2c4bcbb00741f789edee109252d8cb P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:23.876830 1990 raft_consensus.cc:695] T af2c4bcbb00741f789edee109252d8cb P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:23.877220 1990 consensus_queue.cc:237] T af2c4bcbb00741f789edee109252d8cb P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.878891 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_dd882d00b8874af7bdc3df7acba5c1da"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:23.880863 1873 raft_consensus.cc:491] T 8dd587304e6342edbe96c9897b40e5c9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:23.880976 1873 raft_consensus.cc:513] T 8dd587304e6342edbe96c9897b40e5c9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.881536 1873 leader_election.cc:290] T 8dd587304e6342edbe96c9897b40e5c9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:23.881860 992 leader_election.cc:304] T 2240302d9178445d8700e35f5ea274f7 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
W20250901 01:52:23.882018 992 raft_consensus.cc:2684] T 2240302d9178445d8700e35f5ea274f7 P fd27ab6665304aaea53adbd102dd4f58: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250901 01:52:23.885311 1872 consensus_queue.cc:1035] T b2e8b2cd10154de08ddd30c214592a4a P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:23.887306 1151 ts_tablet_manager.cc:1428] T eac50cbae691416e903cb2cbc907f96d P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.049s user 0.001s sys 0.001s
I20250901 01:52:23.888104 1489 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_b362441111a241e19cc1a09bdeefea49" } modify_external_catalogs: true
I20250901 01:52:23.888302 1489 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_b362441111a241e19cc1a09bdeefea49" } modify_external_catalogs: true
I20250901 01:52:23.888873 1151 tablet_bootstrap.cc:492] T e1d18e999b5e42b1873631e0f1445081 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:23.891227 1151 tablet_bootstrap.cc:654] T e1d18e999b5e42b1873631e0f1445081 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:23.900745 1872 raft_consensus.cc:491] T 579b9cdf819f4609a532e96bec093fd5 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:23.900926 1872 raft_consensus.cc:513] T 579b9cdf819f4609a532e96bec093fd5 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.901453 1872 leader_election.cc:290] T 579b9cdf819f4609a532e96bec093fd5 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:23.904268 1485 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_583918346a424cb293d2c35bfa4a5b5f" } modify_external_catalogs: true
I20250901 01:52:23.904593 1485 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_583918346a424cb293d2c35bfa4a5b5f" } modify_external_catalogs: true
I20250901 01:52:23.888554 2113 ts_tablet_manager.cc:1905] T 2240302d9178445d8700e35f5ea274f7 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:23.916425 2118 ts_tablet_manager.cc:1905] T f06a77ecc841462396a8d5026c7327f4 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:23.923056 918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f64ebd080944190be2e64d02d70a8a7" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
W20250901 01:52:23.923147 859 leader_election.cc:343] T f6a8b425968e49deaecfa3adacfd3f1f P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:23.923255 918 raft_consensus.cc:2466] T 7f64ebd080944190be2e64d02d70a8a7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:23.923311 859 leader_election.cc:304] T f6a8b425968e49deaecfa3adacfd3f1f P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:23.923748 859 leader_election.cc:343] T f3962b471bdb49aea6e9fa95f1f9dfe2 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:23.923880 859 leader_election.cc:304] T f3962b471bdb49aea6e9fa95f1f9dfe2 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:23.924261 1981 raft_consensus.cc:2747] T f6a8b425968e49deaecfa3adacfd3f1f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:23.924345 2123 raft_consensus.cc:2747] T f3962b471bdb49aea6e9fa95f1f9dfe2 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:23.925516 992 leader_election.cc:336] T 2240302d9178445d8700e35f5ea274f7 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:23.925806 992 leader_election.cc:336] T af2c4bcbb00741f789edee109252d8cb P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:23.927708 917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8dd587304e6342edbe96c9897b40e5c9" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:23.927661 1873 consensus_queue.cc:1035] T 5d794471dbed4477b9d52fd411fa4d10 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:23.927896 917 raft_consensus.cc:2466] T 8dd587304e6342edbe96c9897b40e5c9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:23.928390 1990 consensus_queue.cc:1035] T bb0cf5a6a27045a08df6845d7473184d P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:23.928532 992 leader_election.cc:304] T 7f64ebd080944190be2e64d02d70a8a7 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:23.930903 917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "579b9cdf819f4609a532e96bec093fd5" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:23.931089 917 raft_consensus.cc:2466] T 579b9cdf819f4609a532e96bec093fd5 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:23.949863 1151 tablet_bootstrap.cc:492] T e1d18e999b5e42b1873631e0f1445081 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:23.949903 1150 tablet_bootstrap.cc:492] T 482a9a2cb81f448bb2823892e35920d4 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:23.950047 1150 ts_tablet_manager.cc:1397] T 482a9a2cb81f448bb2823892e35920d4 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.132s user 0.025s sys 0.067s
I20250901 01:52:23.950183 1151 ts_tablet_manager.cc:1397] T e1d18e999b5e42b1873631e0f1445081 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.061s user 0.002s sys 0.002s
I20250901 01:52:23.950852 1150 raft_consensus.cc:357] T 482a9a2cb81f448bb2823892e35920d4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.951058 1150 raft_consensus.cc:383] T 482a9a2cb81f448bb2823892e35920d4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:23.951164 1150 raft_consensus.cc:738] T 482a9a2cb81f448bb2823892e35920d4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:23.951406 1150 consensus_queue.cc:260] T 482a9a2cb81f448bb2823892e35920d4 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.951934 1150 ts_tablet_manager.cc:1428] T 482a9a2cb81f448bb2823892e35920d4 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250901 01:52:23.952240 1150 tablet_bootstrap.cc:492] T ed1b41340ce742a88c3e4fd85cd25ca6 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:23.953119 1873 raft_consensus.cc:2802] T 7f64ebd080944190be2e64d02d70a8a7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:23.953208 1873 raft_consensus.cc:491] T 7f64ebd080944190be2e64d02d70a8a7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:23.953280 1873 raft_consensus.cc:3058] T 7f64ebd080944190be2e64d02d70a8a7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:23.955900 1873 raft_consensus.cc:513] T 7f64ebd080944190be2e64d02d70a8a7 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.958781 1491 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet a4e3a8180fc14dacb1973e04f99c4492
W20250901 01:52:23.958981 1491 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet a4e3a8180fc14dacb1973e04f99c4492: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:23.959508 1491 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_23fd89096be149b791975d89d65bc280" } modify_external_catalogs: true
I20250901 01:52:23.959699 1491 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_23fd89096be149b791975d89d65bc280" } modify_external_catalogs: true
I20250901 01:52:23.959672 1151 raft_consensus.cc:357] T e1d18e999b5e42b1873631e0f1445081 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.959815 1151 raft_consensus.cc:383] T e1d18e999b5e42b1873631e0f1445081 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:23.959875 1151 raft_consensus.cc:738] T e1d18e999b5e42b1873631e0f1445081 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:23.960071 1151 consensus_queue.cc:260] T e1d18e999b5e42b1873631e0f1445081 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.960642 1151 ts_tablet_manager.cc:1428] T e1d18e999b5e42b1873631e0f1445081 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.010s user 0.000s sys 0.001s
I20250901 01:52:23.960865 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet a4e3a8180fc14dacb1973e04f99c4492 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:23 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:23.961627 1873 leader_election.cc:290] T 7f64ebd080944190be2e64d02d70a8a7 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:23.962544 992 leader_election.cc:336] T 8dd587304e6342edbe96c9897b40e5c9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:23.965865 992 consensus_peers.cc:489] T e25923b2daf9429683b2d44a75a56189 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250901 01:52:23.966056 992 leader_election.cc:336] T 7f64ebd080944190be2e64d02d70a8a7 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:23.966353 992 consensus_peers.cc:489] T 3c1fd0820d8a4995abe027f685da7f7b P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250901 01:52:23.967350 992 consensus_peers.cc:489] T 7cc1efcb6bff4aba8015d2d3b7f4c923 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250901 01:52:23.967542 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 1495de70f0844f82a0eea99c059f076b (table default.Table_02e911ce389e4ad581c0a2f976f99c4e [id=70a5ce71d16b47d4bc6dae97be4548e7]). Expected version 1 got 0
I20250901 01:52:23.967847 1492 catalog_manager.cc:5582] T 9127f9c5239e445ba86012951b7c75d9 P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: UNKNOWN } } }
I20250901 01:52:23.968827 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet ed1b41340ce742a88c3e4fd85cd25ca6 (table default.Table_139f6d174d794a6e9a618e90587d0c0d [id=0baa4993b72e4a8d94ee3059c9d2db6c]). Expected version 1 got 0
I20250901 01:52:23.969048 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet d10c193452954c198bba686169b8025b (table default.Table_03fb48eaab9b42858ab7cefb722e43a8 [id=6c0576410d7340d6aa493a747e95565b]). Expected version 1 got 0
I20250901 01:52:23.969239 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 435b1cca5a7d4d49959a310cb250b2a0 (table default.Table_dc9c5d18c7ec478592a76dbd76424b88 [id=126b298fef6c4653952d672adff1db7e]). Expected version 1 got 0
I20250901 01:52:23.969509 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 410d44e5f1974ed69e5dee65e98856a4 (table default.Table_bec114a616504798bede9a9066e57192 [id=ae79f2c40fd7483ea400a36cf3c48888]). Expected version 4 got 0
I20250901 01:52:23.969889 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet e87d1281f35648b0b115b67c95e244e7 (table default.Table_f840789a58984799bd8057128f6ea439 [id=4453959e273c473fadd737f56f7db3dd]). Expected version 1 got 0
I20250901 01:52:23.970233 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 7954393c8c1c4457ab8bb723c03a9c8b (table default.Table_5ea919f20c6b4a4e91e599357a17a53a [id=543f2074598d4c29972e564be1123fca]). Expected version 1 got 0
I20250901 01:52:23.970425 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 6c0abbfc8d904dce989ce51961e4237d (table default.Table_3f9422b08b0349b9a5466218f0fe1b06 [id=f0dfb3811e744bd4ba6d59195cebde6f]). Expected version 2 got 0
I20250901 01:52:23.971112 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet c6f7c525f59b49938b697de98d4b5835 (table default.Table_db61e055dbd7450898e82c4c0c4763be [id=aabca67f6f754820977ac9c4c20f3e4f]). Expected version 1 got 0
I20250901 01:52:23.971509 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 8ed6616ce54548ba83ba299c4ecf3c38 (table default.Table_1bdd9041d74045cab6f23de11bf4dd09 [id=946ec3e53dcd4de4b8501b3e87e99072]). Expected version 2 got 0
I20250901 01:52:23.971719 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 081e2630e284498aa819f85bf8326315 (table default.Table_9e7ca61fcc324a8984bd1e470162c61f [id=8199118617cb4707bc3ae9b1363750f7]). Expected version 2 got 0
I20250901 01:52:23.970809 1151 tablet_bootstrap.cc:492] T 3c12d6a70ac24dce82e851782e6fe5aa P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:23.974103 2113 ts_tablet_manager.cc:1918] T 2240302d9178445d8700e35f5ea274f7 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:23.974238 2113 log.cc:1199] T 2240302d9178445d8700e35f5ea274f7 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/2240302d9178445d8700e35f5ea274f7
I20250901 01:52:23.974435 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 2599f98abfdc4871b573edfc2ad97d22 (table default.Table_0e63d21acfcb4c028c9ea6406b6a04e4 [id=81323e6fbe424655b8e185f52657fadc]). Expected version 1 got 0
I20250901 01:52:23.974578 2113 ts_tablet_manager.cc:1939] T 2240302d9178445d8700e35f5ea274f7 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:23.975010 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 81ce448b148447ddbc41875d863e0518 (table default.Table_6e461d9cb0ae4dd095975bbc21533794 [id=49673fa73f8c436e9b8026efc4f3cbbc]). Expected version 1 got 0
I20250901 01:52:23.975368 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 42c1156e8dd84c6fa1396b46de13eb6f (table default.Table_b5d681f51fc84395bbabe1e5b4cfb358 [id=40d05d90b0354334840ad17764f5b6bb]). Expected version 1 got 0
I20250901 01:52:23.975625 992 leader_election.cc:304] T 8dd587304e6342edbe96c9897b40e5c9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:23.976198 1873 raft_consensus.cc:2802] T 8dd587304e6342edbe96c9897b40e5c9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:23.976330 1873 raft_consensus.cc:491] T 8dd587304e6342edbe96c9897b40e5c9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:23.976444 1873 raft_consensus.cc:3058] T 8dd587304e6342edbe96c9897b40e5c9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:23.978392 917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f64ebd080944190be2e64d02d70a8a7" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:23.978608 917 raft_consensus.cc:3058] T 7f64ebd080944190be2e64d02d70a8a7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:23.979169 1873 raft_consensus.cc:513] T 8dd587304e6342edbe96c9897b40e5c9 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.979736 1873 leader_election.cc:290] T 8dd587304e6342edbe96c9897b40e5c9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:23.980674 1151 tablet_bootstrap.cc:654] T 3c12d6a70ac24dce82e851782e6fe5aa P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:23.980746 1873 raft_consensus.cc:491] T 50e3da53433f4c14b57db8a733fa53bc P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:23.980908 1873 raft_consensus.cc:513] T 50e3da53433f4c14b57db8a733fa53bc P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.981451 1873 leader_election.cc:290] T 50e3da53433f4c14b57db8a733fa53bc P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:23.981685 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 616ff34e8d314e0aa0e11ce60be11789 (table default.Table_0451d0afa26f4d5f8d4214a946eea04f [id=5ee13d46e06a4fa59b1070547573e523]). Expected version 1 got 0
I20250901 01:52:23.981971 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 3c12d6a70ac24dce82e851782e6fe5aa (table default.Table_d2049d0c20bb4949ae6c249b690e7f86 [id=26bba0e60b6b43eb87d084be403c556b]). Expected version 1 got 0
I20250901 01:52:23.982142 1990 consensus_queue.cc:1035] T 40d74d7b5a3b4d1dae09eebd9e07a0c4 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:23.982266 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet e1d18e999b5e42b1873631e0f1445081 (table default.Table_fd17a2ad225546aa821489ce94a788a2 [id=033c76e0598342c495ce2be42a0df5dd]). Expected version 1 got 0
I20250901 01:52:23.982735 992 leader_election.cc:304] T 579b9cdf819f4609a532e96bec093fd5 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:23.982590 1492 catalog_manager.cc:5582] T b2e8b2cd10154de08ddd30c214592a4a P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: UNKNOWN } } }
I20250901 01:52:23.983049 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet b2e8b2cd10154de08ddd30c214592a4a (table default.Table_4a412a59b6db421eb0187ffd02ecf541 [id=fd72e46c84634b6c9ea74970480ecc62]). Expected version 4 got 0
I20250901 01:52:23.983142 1990 raft_consensus.cc:2802] T 579b9cdf819f4609a532e96bec093fd5 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:23.983263 1990 raft_consensus.cc:491] T 579b9cdf819f4609a532e96bec093fd5 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:23.983364 1990 raft_consensus.cc:3058] T 579b9cdf819f4609a532e96bec093fd5 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:23.983844 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 49aaaec611784e6bac131efd58dbca51 (table default.Table_95e1338a49a04f22b679e363e253262e [id=e24b91b4ac904ef0a8c4725f70bced2f]). Expected version 1 got 0
I20250901 01:52:23.984140 1492 catalog_manager.cc:5582] T 7b16ca1daf394a6da65a4af8da82ec33 P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: UNKNOWN } } }
I20250901 01:52:23.984879 1492 catalog_manager.cc:5582] T ddbc15685fca4c708329420109ad0c57 P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: HEALTHY } } }
I20250901 01:52:23.985253 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet ddbc15685fca4c708329420109ad0c57 (table default.Table_942175dadf4b496195b2fa53f39f6ddc [id=2bd332b452f84f25af4225af8e6c6740]). Expected version 1 got 0
I20250901 01:52:23.985603 1492 catalog_manager.cc:5582] T 72ba689f9230411995dc96e2b45601df P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: UNKNOWN } } }
I20250901 01:52:23.987071 1990 raft_consensus.cc:513] T 579b9cdf819f4609a532e96bec093fd5 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:23.987629 1990 leader_election.cc:290] T 579b9cdf819f4609a532e96bec093fd5 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:23.988533 885 tablet_service.cc:1515] Processing DeleteTablet for tablet a4e3a8180fc14dacb1973e04f99c4492 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:23 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:23.998103 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 2240302d9178445d8700e35f5ea274f7 (table default.Table_36f5f8854ed14d1ea48eb18865d00659 [id=bdf8a102ae7d43228173d7b82507d80d]) successfully deleted
I20250901 01:52:24.000804 2133 tablet.cc:1722] T 34042ea5c1b645969c1de41fe587c343 P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250901 01:52:24.005817 917 raft_consensus.cc:2466] T 7f64ebd080944190be2e64d02d70a8a7 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:24.009922 992 leader_election.cc:304] T 7f64ebd080944190be2e64d02d70a8a7 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:24.010201 1761 raft_consensus.cc:2802] T 7f64ebd080944190be2e64d02d70a8a7 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:24.010363 1761 raft_consensus.cc:695] T 7f64ebd080944190be2e64d02d70a8a7 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:24.010685 1761 consensus_queue.cc:237] T 7f64ebd080944190be2e64d02d70a8a7 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:24.017740 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 17e8340dbd1a406cb782dad3fe88cc7e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 17e8340dbd1a406cb782dad3fe88cc7e already in progress: creating tablet
I20250901 01:52:24.018280 2113 tablet_replica.cc:331] T f06a77ecc841462396a8d5026c7327f4 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:24.018615 2113 raft_consensus.cc:2241] T f06a77ecc841462396a8d5026c7327f4 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:24.019876 2118 ts_tablet_manager.cc:1918] T f06a77ecc841462396a8d5026c7327f4 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:24.020452 2118 log.cc:1199] T f06a77ecc841462396a8d5026c7327f4 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/f06a77ecc841462396a8d5026c7327f4
I20250901 01:52:24.020956 1150 tablet_bootstrap.cc:654] T ed1b41340ce742a88c3e4fd85cd25ca6 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:24.023015 2118 ts_tablet_manager.cc:1939] T f06a77ecc841462396a8d5026c7327f4 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:24.023396 1761 raft_consensus.cc:491] T 0cb3262973934c8689b851adad60afe0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:24.023602 1761 raft_consensus.cc:513] T 0cb3262973934c8689b851adad60afe0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.024231 1761 leader_election.cc:290] T 0cb3262973934c8689b851adad60afe0 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:24.024405 2113 raft_consensus.cc:2270] T f06a77ecc841462396a8d5026c7327f4 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250901 01:52:24.027144 992 consensus_peers.cc:489] T b2e8b2cd10154de08ddd30c214592a4a P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:24.027325 2113 ts_tablet_manager.cc:1905] T f06a77ecc841462396a8d5026c7327f4 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:24.028867 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 72ba689f9230411995dc96e2b45601df (table default.Table_604f2c5c58d74154bacf2a065a467f32 [id=351a05134f4948039c920cafbe050914]). Expected version 1 got 0
I20250901 01:52:24.029163 2133 tablet.cc:1722] T 606fac5c6efe48d1877104b01c23ba5b P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:24.029314 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet d0c3c363dda444519a17dca63ad9caa7 (table default.Table_0b4ac0a1caa5472d909448e41318e679 [id=8c1b263824ec4c15a5ec424364c78895]). Expected version 2 got 0
I20250901 01:52:24.029505 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 606fac5c6efe48d1877104b01c23ba5b (table default.Table_7c12d66048a6470e8c227dc1d9f45fa4 [id=91472f220fe44297b79d3eec8b9b80e8]). Expected version 1 got 0
I20250901 01:52:24.029762 1492 catalog_manager.cc:5582] T 629db80a8ab444e582628578155638fd P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00c282f779c1477d920f24dc0dc16628 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "00c282f779c1477d920f24dc0dc16628" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } }
I20250901 01:52:24.030121 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 629db80a8ab444e582628578155638fd (table default.Table_f0e56dba1e8a4ed49a760e204c1ff96e [id=c3865e058daa4cea9a89f5046fca712a]). Expected version 1 got 0
W20250901 01:52:24.037497 992 leader_election.cc:336] T 579b9cdf819f4609a532e96bec093fd5 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:24.037868 992 consensus_peers.cc:489] T d77aadd7291647a0bae7c421f244c252 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:24.040134 1151 tablet_bootstrap.cc:492] T 3c12d6a70ac24dce82e851782e6fe5aa P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:24.040326 1151 ts_tablet_manager.cc:1397] T 3c12d6a70ac24dce82e851782e6fe5aa P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.070s user 0.004s sys 0.003s
I20250901 01:52:24.041379 1151 raft_consensus.cc:357] T 3c12d6a70ac24dce82e851782e6fe5aa P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.041201 924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8dd587304e6342edbe96c9897b40e5c9" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:24.044572 1990 raft_consensus.cc:491] T 49aaaec611784e6bac131efd58dbca51 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:24.044725 1990 raft_consensus.cc:513] T 49aaaec611784e6bac131efd58dbca51 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.045157 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 75a703c8e20a4a2fa0ab6ec58d41ef92 (table default.Table_2f09538b6fbe401f909105decb56e1cb [id=a90d1f10531042c3bb7a196b8be61c5f]). Expected version 1 got 0
I20250901 01:52:24.045310 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet f06a77ecc841462396a8d5026c7327f4 (table default.Table_e2ea7aa90231448888c6cfead7e3d9c1 [id=c264ed0bf1d84bc9bd5c2bb0d1e8a45b]) successfully deleted
I20250901 01:52:24.048031 924 raft_consensus.cc:3058] T 8dd587304e6342edbe96c9897b40e5c9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:24.048185 922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "579b9cdf819f4609a532e96bec093fd5" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:24.048458 922 raft_consensus.cc:3058] T 579b9cdf819f4609a532e96bec093fd5 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:24.051997 922 raft_consensus.cc:2466] T 579b9cdf819f4609a532e96bec093fd5 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:24.053123 1990 leader_election.cc:290] T 49aaaec611784e6bac131efd58dbca51 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:24.053249 919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50e3da53433f4c14b57db8a733fa53bc" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:24.054657 919 raft_consensus.cc:2466] T 50e3da53433f4c14b57db8a733fa53bc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:24.056432 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet e9f1f892658b4736859ff507536920a1 (table default.Table_8825dd28afbe464193eb1b668cc9326a [id=9aa62cb634314a0a849612c54c1220d9]). Expected version 1 got 0
I20250901 01:52:24.057366 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet fa7fb88ded21432a8757f37ec3db5e71 (table default.Table_22d8d38d0b624912a0d8968fb764420e [id=808a8849c99e485b8be7cd17397874ec]). Expected version 1 got 0
I20250901 01:52:24.058440 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 5f5396045979450193d461ba1334a5c9 (table default.Table_a89cc8e48563460baabc34c821ceae35 [id=9c3930e9fcfe4b49ba4bea67a2b0ad07]). Expected version 1 got 0
I20250901 01:52:24.059060 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 246b33338de647689d7404c70676e76f (table default.Table_219f0092f43743039ae1a61fedb3e4d1 [id=a7b1d2a8ace14ef8bcce0a3bed41e85d]). Expected version 1 got 0
I20250901 01:52:24.058835 2118 tablet_replica.cc:331] T 17e8340dbd1a406cb782dad3fe88cc7e P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:24.059518 1151 raft_consensus.cc:383] T 3c12d6a70ac24dce82e851782e6fe5aa P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:24.059234 924 raft_consensus.cc:2466] T 8dd587304e6342edbe96c9897b40e5c9 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:24.060065 2118 raft_consensus.cc:2241] T 17e8340dbd1a406cb782dad3fe88cc7e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:24.060026 921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cb3262973934c8689b851adad60afe0" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:24.060559 2138 tablet.cc:1722] T 34042ea5c1b645969c1de41fe587c343 P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250901 01:52:24.060565 2118 raft_consensus.cc:2270] T 17e8340dbd1a406cb782dad3fe88cc7e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:24.060153 1150 tablet_bootstrap.cc:492] T ed1b41340ce742a88c3e4fd85cd25ca6 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:24.060891 921 raft_consensus.cc:2466] T 0cb3262973934c8689b851adad60afe0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:24.061055 1150 ts_tablet_manager.cc:1397] T ed1b41340ce742a88c3e4fd85cd25ca6 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.109s user 0.007s sys 0.007s
I20250901 01:52:24.059626 1151 raft_consensus.cc:738] T 3c12d6a70ac24dce82e851782e6fe5aa P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:24.061503 2118 ts_tablet_manager.cc:1905] T 17e8340dbd1a406cb782dad3fe88cc7e P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:24.061807 1151 consensus_queue.cc:260] T 3c12d6a70ac24dce82e851782e6fe5aa P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:24.059863 992 consensus_peers.cc:489] T f8a5886ddb644a86b76df44aab309057 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250901 01:52:24.062251 992 consensus_peers.cc:489] T b6daeff0070a4db08e90020752d21c29 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:24.061898 1150 raft_consensus.cc:357] T ed1b41340ce742a88c3e4fd85cd25ca6 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:24.062395 992 leader_election.cc:336] T 7f64ebd080944190be2e64d02d70a8a7 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:24.062500 1150 raft_consensus.cc:383] T ed1b41340ce742a88c3e4fd85cd25ca6 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:24.062600 1150 raft_consensus.cc:738] T ed1b41340ce742a88c3e4fd85cd25ca6 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:24.062839 1150 consensus_queue.cc:260] T ed1b41340ce742a88c3e4fd85cd25ca6 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.063561 1150 ts_tablet_manager.cc:1428] T ed1b41340ce742a88c3e4fd85cd25ca6 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250901 01:52:24.064152 2113 ts_tablet_manager.cc:1918] T f06a77ecc841462396a8d5026c7327f4 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:24.066069 921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "49aaaec611784e6bac131efd58dbca51" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:24.066310 921 raft_consensus.cc:2466] T 49aaaec611784e6bac131efd58dbca51 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:24.068214 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 617cc33d0d23439cb5f1e0c31bf11281 (table default.Table_ce1fd26ac65a4ed085d023e10a849e5c [id=fb088d31bba64f04871fbfd355d6263b]). Expected version 1 got 0
I20250901 01:52:24.067742 992 leader_election.cc:304] T 579b9cdf819f4609a532e96bec093fd5 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:24.068579 1761 raft_consensus.cc:2802] T 579b9cdf819f4609a532e96bec093fd5 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:24.068713 1761 raft_consensus.cc:695] T 579b9cdf819f4609a532e96bec093fd5 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:24.068964 1761 consensus_queue.cc:237] T 579b9cdf819f4609a532e96bec093fd5 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.068346 2113 log.cc:1199] T f06a77ecc841462396a8d5026c7327f4 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/f06a77ecc841462396a8d5026c7327f4
I20250901 01:52:24.069881 2113 ts_tablet_manager.cc:1939] T f06a77ecc841462396a8d5026c7327f4 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:24.077368 2123 raft_consensus.cc:491] T 7a607fac57994001b2304e5d81c9decc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:24.077692 2123 raft_consensus.cc:513] T 7a607fac57994001b2304e5d81c9decc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.078491 2123 leader_election.cc:290] T 7a607fac57994001b2304e5d81c9decc P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:24.079633 1150 tablet_bootstrap.cc:492] T 14b466b2a5a249aa841f1b8b06301edc P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:24.079704 2113 tablet_replica.cc:331] T ded33e0950b544509af11739353c620d P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:24.079927 2113 raft_consensus.cc:2241] T ded33e0950b544509af11739353c620d P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
W20250901 01:52:24.080111 858 leader_election.cc:336] T 7a607fac57994001b2304e5d81c9decc P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:24.080386 2113 raft_consensus.cc:2270] T ded33e0950b544509af11739353c620d P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:24.081099 2140 tablet.cc:1722] T 606fac5c6efe48d1877104b01c23ba5b P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:24.081197 2113 ts_tablet_manager.cc:1905] T ded33e0950b544509af11739353c620d P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:24.084192 2123 raft_consensus.cc:491] T afc002ae2486476b9208e1cf8215157c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:24.084317 2123 raft_consensus.cc:513] T afc002ae2486476b9208e1cf8215157c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.084550 1872 raft_consensus.cc:491] T b7b802f976f3471faaaecc964ec87e48 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:24.084661 1872 raft_consensus.cc:513] T b7b802f976f3471faaaecc964ec87e48 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.085022 2123 leader_election.cc:290] T afc002ae2486476b9208e1cf8215157c P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:24.085189 1872 leader_election.cc:290] T b7b802f976f3471faaaecc964ec87e48 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:24.085958 2118 ts_tablet_manager.cc:1918] T 17e8340dbd1a406cb782dad3fe88cc7e P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250901 01:52:24.086934 858 leader_election.cc:336] T afc002ae2486476b9208e1cf8215157c P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:24.087813 2118 log.cc:1199] T 17e8340dbd1a406cb782dad3fe88cc7e P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/17e8340dbd1a406cb782dad3fe88cc7e
I20250901 01:52:24.077064 992 leader_election.cc:304] T 50e3da53433f4c14b57db8a733fa53bc P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:24.089049 1990 raft_consensus.cc:2802] T 50e3da53433f4c14b57db8a733fa53bc P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:24.089159 1990 raft_consensus.cc:491] T 50e3da53433f4c14b57db8a733fa53bc P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:24.089255 1990 raft_consensus.cc:3058] T 50e3da53433f4c14b57db8a733fa53bc P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:24.093138 2118 ts_tablet_manager.cc:1939] T 17e8340dbd1a406cb782dad3fe88cc7e P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:24.091638 1990 raft_consensus.cc:513] T 50e3da53433f4c14b57db8a733fa53bc P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.096670 1990 leader_election.cc:290] T 50e3da53433f4c14b57db8a733fa53bc P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:24.096984 992 leader_election.cc:304] T 8dd587304e6342edbe96c9897b40e5c9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:24.097329 1993 raft_consensus.cc:2802] T 8dd587304e6342edbe96c9897b40e5c9 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:24.097467 1993 raft_consensus.cc:695] T 8dd587304e6342edbe96c9897b40e5c9 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:24.097714 1993 consensus_queue.cc:237] T 8dd587304e6342edbe96c9897b40e5c9 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.103482 992 leader_election.cc:304] T 0cb3262973934c8689b851adad60afe0 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:24.104629 1761 raft_consensus.cc:2802] T 0cb3262973934c8689b851adad60afe0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:24.104734 1761 raft_consensus.cc:491] T 0cb3262973934c8689b851adad60afe0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:24.104869 1761 raft_consensus.cc:3058] T 0cb3262973934c8689b851adad60afe0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:24.105070 1873 consensus_queue.cc:1035] T e25923b2daf9429683b2d44a75a56189 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:24.107488 1761 raft_consensus.cc:513] T 0cb3262973934c8689b851adad60afe0 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.108135 1761 leader_election.cc:290] T 0cb3262973934c8689b851adad60afe0 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:24.108196 2118 tablet_replica.cc:331] T ded33e0950b544509af11739353c620d P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:24.108453 992 leader_election.cc:304] T 49aaaec611784e6bac131efd58dbca51 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:24.108582 2118 raft_consensus.cc:2241] T ded33e0950b544509af11739353c620d P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:24.108809 2118 raft_consensus.cc:2270] T ded33e0950b544509af11739353c620d P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:24.109833 2118 ts_tablet_manager.cc:1905] T ded33e0950b544509af11739353c620d P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:24.111075 2113 ts_tablet_manager.cc:1918] T ded33e0950b544509af11739353c620d P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:24.111198 2113 log.cc:1199] T ded33e0950b544509af11739353c620d P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/ded33e0950b544509af11739353c620d
I20250901 01:52:24.111564 2113 ts_tablet_manager.cc:1939] T ded33e0950b544509af11739353c620d P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:24.112121 1873 raft_consensus.cc:2802] T 49aaaec611784e6bac131efd58dbca51 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:24.112234 1873 raft_consensus.cc:491] T 49aaaec611784e6bac131efd58dbca51 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:24.112244 1993 consensus_queue.cc:1035] T f8a5886ddb644a86b76df44aab309057 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:24.112356 1873 raft_consensus.cc:3058] T 49aaaec611784e6bac131efd58dbca51 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
W20250901 01:52:24.114153 992 leader_election.cc:336] T 0cb3262973934c8689b851adad60afe0 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:24.114853 1873 raft_consensus.cc:513] T 49aaaec611784e6bac131efd58dbca51 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.079382 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet f06a77ecc841462396a8d5026c7327f4 (table default.Table_e2ea7aa90231448888c6cfead7e3d9c1 [id=c264ed0bf1d84bc9bd5c2bb0d1e8a45b]) successfully deleted
I20250901 01:52:24.094049 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 17e8340dbd1a406cb782dad3fe88cc7e (table default.Table_50ae3908a7944f70949c54be8fbf1564 [id=58ff38d356d74951b663d8ef7530f1c8]) successfully deleted
I20250901 01:52:24.115485 1873 leader_election.cc:290] T 49aaaec611784e6bac131efd58dbca51 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:24.116380 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet b164b4a67a9143ceaff69b0ce49d63fe (table default.Table_da2c70a8f2fe49a0a8184e88c0a9e0ea [id=c1b1e9e3876244f68d31c63ce7bdebf1]). Expected version 1 got 0
I20250901 01:52:24.116840 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet ae21b9ae2e3b472eb67fb5f96388a50f (table default.Table_046289f393ff432aae0b033bdec1fb17 [id=db57b18a9ce24e068b362f5fbe940531]). Expected version 1 got 0
I20250901 01:52:24.116906 1872 raft_consensus.cc:491] T e1d18e999b5e42b1873631e0f1445081 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:24.117058 1872 raft_consensus.cc:513] T e1d18e999b5e42b1873631e0f1445081 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.117573 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet d4c9aa723aee4dc4b6237cb5c3722b52 (table default.Table_5e65c5f068b84995a2d7a76fa65dda0f [id=b9d81ddd38fe47838877b010be2b40d0]). Expected version 1 got 0
I20250901 01:52:24.117662 1872 leader_election.cc:290] T e1d18e999b5e42b1873631e0f1445081 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:24.118916 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 93bcac1fd95d44dc927abfca81692921 (table default.Table_b8f8ee74648c406eaf589d3ab80934db [id=b8f69313439543be9c11eea88dfa69c5]). Expected version 1 got 0
I20250901 01:52:24.119282 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 7f46e9ddadaf4ff986d8c991178daa0c (table default.Table_15c9f4f02eb24474936a43d705f90f7b [id=ea928501fe954cd0b81d38add4ef7128]). Expected version 1 got 0
I20250901 01:52:24.120890 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 3dae00aa38204ee3ac77325eb334be42 (table default.Table_faf7ef12324f454396e5ae64d64ebe51 [id=5812902df9ea406ba59ff0ff23f130b6]). Expected version 1 got 0
I20250901 01:52:24.062397 1151 ts_tablet_manager.cc:1428] T 3c12d6a70ac24dce82e851782e6fe5aa P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.022s user 0.001s sys 0.000s
I20250901 01:52:24.121127 1492 catalog_manager.cc:5582] T fcb7b5b7d63d4f0b906fea97cb548de2 P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: HEALTHY } } }
I20250901 01:52:24.121522 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet fcb7b5b7d63d4f0b906fea97cb548de2 (table default.Table_2ceaff93c8f44f00aab53e48d8466dfd [id=4d9ea39016ca4bf4859f4f80ea4e9ed6]). Expected version 4 got 0
I20250901 01:52:24.121642 1151 tablet_bootstrap.cc:492] T 616ff34e8d314e0aa0e11ce60be11789 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:24.123068 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet cb7d7e1796c0456f8de98fb812b9e374 (table default.Table_c4148158ed7b48a3946559ba978e72fd [id=327c5a63904540508142ab600c308757]). Expected version 1 got 0
I20250901 01:52:24.123926 1151 tablet_bootstrap.cc:654] T 616ff34e8d314e0aa0e11ce60be11789 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:24.124608 992 leader_election.cc:336] T 8dd587304e6342edbe96c9897b40e5c9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:24.123728 1492 catalog_manager.cc:5582] T b009cc95c819431ea9827a3e453b8da3 P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00c282f779c1477d920f24dc0dc16628 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "00c282f779c1477d920f24dc0dc16628" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } }
W20250901 01:52:24.124881 992 consensus_peers.cc:489] T 75e7a90815e449c39f386901a504143c P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250901 01:52:24.125038 992 leader_election.cc:336] T 50e3da53433f4c14b57db8a733fa53bc P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:24.125277 992 leader_election.cc:336] T 579b9cdf819f4609a532e96bec093fd5 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:24.125533 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet b009cc95c819431ea9827a3e453b8da3 (table default.Table_203547c9458e43b3b48ea051d56892c6 [id=cc26c89db2484a6fad919231116285f2]). Expected version 1 got 0
I20250901 01:52:24.126952 923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b7b802f976f3471faaaecc964ec87e48" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:24.127136 923 raft_consensus.cc:2466] T b7b802f976f3471faaaecc964ec87e48 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:24.127261 924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50e3da53433f4c14b57db8a733fa53bc" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:24.127471 924 raft_consensus.cc:3058] T 50e3da53433f4c14b57db8a733fa53bc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:24.128841 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cb3262973934c8689b851adad60afe0" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:24.129015 916 raft_consensus.cc:3058] T 0cb3262973934c8689b851adad60afe0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:24.130529 924 raft_consensus.cc:2466] T 50e3da53433f4c14b57db8a733fa53bc P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:24.130968 1150 tablet_bootstrap.cc:654] T 14b466b2a5a249aa841f1b8b06301edc P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:24.131502 916 raft_consensus.cc:2466] T 0cb3262973934c8689b851adad60afe0 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:24.133896 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet ded33e0950b544509af11739353c620d (table default.Table_93afd37289414655980b22934695bd55 [id=01c391f6c76943c9af685a74348eaf41]) successfully deleted
I20250901 01:52:24.135610 1058 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a607fac57994001b2304e5d81c9decc" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:24.136193 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet becbaae4fd7e44faa679e9479465a7c4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet becbaae4fd7e44faa679e9479465a7c4 already in progress: creating tablet
I20250901 01:52:24.136535 1058 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afc002ae2486476b9208e1cf8215157c" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:24.137143 2123 raft_consensus.cc:491] T ed1b41340ce742a88c3e4fd85cd25ca6 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:24.137292 2123 raft_consensus.cc:513] T ed1b41340ce742a88c3e4fd85cd25ca6 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:24.137480 992 leader_election.cc:336] T 49aaaec611784e6bac131efd58dbca51 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:24.137750 992 consensus_peers.cc:489] T 92f0ad59fbbb435ea872c669a52f854a P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:24.137892 2123 leader_election.cc:290] T ed1b41340ce742a88c3e4fd85cd25ca6 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:24.139497 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 26c8ed083cea4dc0acca80d00b11b29b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 26c8ed083cea4dc0acca80d00b11b29b already in progress: creating tablet
W20250901 01:52:24.139801 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 088ef4861dc14f81ae470c3aaee5b4e1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 088ef4861dc14f81ae470c3aaee5b4e1 already in progress: creating tablet
W20250901 01:52:24.140031 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 56475ba491a24301a534304811f30140 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 56475ba491a24301a534304811f30140 already in progress: creating tablet
I20250901 01:52:24.140787 1150 tablet_bootstrap.cc:492] T 14b466b2a5a249aa841f1b8b06301edc P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:24.140925 1150 ts_tablet_manager.cc:1397] T 14b466b2a5a249aa841f1b8b06301edc P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.077s user 0.017s sys 0.002s
W20250901 01:52:24.141112 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 764bfeb2a7d14c40815b5ec1b07bdd62 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 764bfeb2a7d14c40815b5ec1b07bdd62 already in progress: creating tablet
I20250901 01:52:24.141606 1150 raft_consensus.cc:357] T 14b466b2a5a249aa841f1b8b06301edc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.141808 1150 raft_consensus.cc:383] T 14b466b2a5a249aa841f1b8b06301edc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:24.141786 917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e1d18e999b5e42b1873631e0f1445081" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:24.141860 1150 raft_consensus.cc:738] T 14b466b2a5a249aa841f1b8b06301edc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:24.141912 2118 ts_tablet_manager.cc:1918] T ded33e0950b544509af11739353c620d P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:24.141933 917 raft_consensus.cc:2466] T e1d18e999b5e42b1873631e0f1445081 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:24.141995 2118 log.cc:1199] T ded33e0950b544509af11739353c620d P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/ded33e0950b544509af11739353c620d
I20250901 01:52:24.142042 1150 consensus_queue.cc:260] T 14b466b2a5a249aa841f1b8b06301edc P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.142319 2118 ts_tablet_manager.cc:1939] T ded33e0950b544509af11739353c620d P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:24.142519 921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "49aaaec611784e6bac131efd58dbca51" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:24.142649 921 raft_consensus.cc:3058] T 49aaaec611784e6bac131efd58dbca51 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:24.143613 2118 tablet_replica.cc:331] T eac50cbae691416e903cb2cbc907f96d P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:24.143779 2118 raft_consensus.cc:2241] T eac50cbae691416e903cb2cbc907f96d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:24.143890 2118 raft_consensus.cc:2270] T eac50cbae691416e903cb2cbc907f96d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:24.144328 2118 ts_tablet_manager.cc:1905] T eac50cbae691416e903cb2cbc907f96d P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:24.144555 2113 tablet_replica.cc:331] T eac50cbae691416e903cb2cbc907f96d P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:24.144579 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet ded33e0950b544509af11739353c620d (table default.Table_93afd37289414655980b22934695bd55 [id=01c391f6c76943c9af685a74348eaf41]) successfully deleted
I20250901 01:52:24.144757 2113 raft_consensus.cc:2241] T eac50cbae691416e903cb2cbc907f96d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250901 01:52:24.144802 1472 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 26c8ed083cea4dc0acca80d00b11b29b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 26c8ed083cea4dc0acca80d00b11b29b already in progress: creating tablet
I20250901 01:52:24.144833 1150 ts_tablet_manager.cc:1428] T 14b466b2a5a249aa841f1b8b06301edc P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250901 01:52:24.144893 2113 raft_consensus.cc:2270] T eac50cbae691416e903cb2cbc907f96d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250901 01:52:24.145010 1472 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 088ef4861dc14f81ae470c3aaee5b4e1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 088ef4861dc14f81ae470c3aaee5b4e1 already in progress: creating tablet
I20250901 01:52:24.145092 1150 tablet_bootstrap.cc:492] T ad1a6be9fcc045d3bb4e71a2f6bd5ccd P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
W20250901 01:52:24.145192 1472 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 56475ba491a24301a534304811f30140 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 56475ba491a24301a534304811f30140 already in progress: creating tablet
W20250901 01:52:24.145941 1472 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 764bfeb2a7d14c40815b5ec1b07bdd62 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 764bfeb2a7d14c40815b5ec1b07bdd62 already in progress: creating tablet
W20250901 01:52:24.146183 1472 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet becbaae4fd7e44faa679e9479465a7c4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet becbaae4fd7e44faa679e9479465a7c4 already in progress: creating tablet
W20250901 01:52:24.147032 858 leader_election.cc:336] T ed1b41340ce742a88c3e4fd85cd25ca6 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:24.148437 921 raft_consensus.cc:2466] T 49aaaec611784e6bac131efd58dbca51 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:24.148674 1150 tablet_bootstrap.cc:654] T ad1a6be9fcc045d3bb4e71a2f6bd5ccd P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:24.152909 1151 tablet_bootstrap.cc:492] T 616ff34e8d314e0aa0e11ce60be11789 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:24.153082 1151 ts_tablet_manager.cc:1397] T 616ff34e8d314e0aa0e11ce60be11789 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.032s user 0.005s sys 0.000s
I20250901 01:52:24.153532 1492 catalog_manager.cc:5582] T 6c05eff658924ca4a9a1e7a37f917d67 P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } }
I20250901 01:52:24.153963 1151 raft_consensus.cc:357] T 616ff34e8d314e0aa0e11ce60be11789 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.154192 1151 raft_consensus.cc:383] T 616ff34e8d314e0aa0e11ce60be11789 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:24.154305 1151 raft_consensus.cc:738] T 616ff34e8d314e0aa0e11ce60be11789 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:24.154780 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 33f637f6a8134215bb1995ec38440073 (table default.Table_2ece91972d494680a8ad00273c10dee4 [id=557afe29976d40f6bfb6ae4ca348f887]). Expected version 1 got 0
I20250901 01:52:24.155423 1151 consensus_queue.cc:260] T 616ff34e8d314e0aa0e11ce60be11789 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.156066 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet f80f13db501c42c7950499df8fcb6fc8 (table default.Table_ce86974fd8974de9a5a737954fac2ad7 [id=ff772d20b287476f94716379b86d2a71]). Expected version 1 got 0
I20250901 01:52:24.156572 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 3618a3eb4f174c43805ce7dc0e5315cb (table default.Table_83f2c0acedbc4c6982caf5313646e90b [id=31a2e50c32644321825da40176127015]). Expected version 1 got 0
I20250901 01:52:24.155565 2113 ts_tablet_manager.cc:1905] T eac50cbae691416e903cb2cbc907f96d P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:24.161104 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet d1f2d98712ff4ea6aec665512de068bd (table default.Table_82d91a1ba2eb4265896824028dec8d3a [id=3f7624d2901445e39155c12fa68e34e9]). Expected version 1 got 0
I20250901 01:52:24.161582 1492 catalog_manager.cc:5582] T e6c10426f60f43e8bde69d9b1b9c8eb6 P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: HEALTHY } } }
I20250901 01:52:24.162528 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet e6c10426f60f43e8bde69d9b1b9c8eb6 (table default.Table_eecfab7239cc40c5a2f1fb6ac6d35dc9 [id=d3dbeb4258424133844e20b181ee44e4]). Expected version 1 got 0
I20250901 01:52:24.163012 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 615b2306fca84003b264da9d5f95ddab (table default.Table_0f959237963c4e33932e193285c0eacb [id=cecdd8d6ca6d445c9fd82decd2a95366]). Expected version 1 got 0
I20250901 01:52:24.163384 2123 raft_consensus.cc:491] T 482a9a2cb81f448bb2823892e35920d4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:24.163503 2123 raft_consensus.cc:513] T 482a9a2cb81f448bb2823892e35920d4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.164325 2123 leader_election.cc:290] T 482a9a2cb81f448bb2823892e35920d4 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:24.164682 1151 ts_tablet_manager.cc:1428] T 616ff34e8d314e0aa0e11ce60be11789 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
I20250901 01:52:24.164973 1151 tablet_bootstrap.cc:492] T e9b179dba2cc49428ba8090c962d6136 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
W20250901 01:52:24.166226 858 leader_election.cc:336] T 482a9a2cb81f448bb2823892e35920d4 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:24.166716 1151 tablet_bootstrap.cc:654] T e9b179dba2cc49428ba8090c962d6136 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:24.167768 1492 catalog_manager.cc:5582] T 3c1fd0820d8a4995abe027f685da7f7b P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: UNKNOWN } } }
I20250901 01:52:24.168253 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 3c1fd0820d8a4995abe027f685da7f7b (table default.Table_bcd502e1360b48b1953fa97eadce7006 [id=b3914f33e856466e8479d0a2263ec56f]). Expected version 1 got 0
I20250901 01:52:24.168609 1492 catalog_manager.cc:5582] T f62d96c37cca4143a4b098b7c69979e6 P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: HEALTHY } } }
I20250901 01:52:24.169025 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet f62d96c37cca4143a4b098b7c69979e6 (table default.Table_0877ac8202a24e9da650a589bc6dbfcc [id=1f65b3e09c9748f189ecb2b56c5d1132]). Expected version 1 got 0
I20250901 01:52:24.169293 1492 catalog_manager.cc:5582] T 5adf6b492640417ca071c94702b48cb9 P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } }
I20250901 01:52:24.169739 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 5adf6b492640417ca071c94702b48cb9 (table default.Table_589c302e02474fc3bc5d3a290a8bd54a [id=5d14706774d345c48ca3575feabb17b0]). Expected version 2 got 0
I20250901 01:52:24.169920 2118 ts_tablet_manager.cc:1918] T eac50cbae691416e903cb2cbc907f96d P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:24.170049 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 34042ea5c1b645969c1de41fe587c343 (table default.Table_23fd89096be149b791975d89d65bc280 [id=edbc04078c904f29a16a1bbb354dad92]). Expected version 2 got 0
I20250901 01:52:24.170079 2118 log.cc:1199] T eac50cbae691416e903cb2cbc907f96d P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/eac50cbae691416e903cb2cbc907f96d
I20250901 01:52:24.170518 1492 catalog_manager.cc:5582] T 5d794471dbed4477b9d52fd411fa4d10 P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: UNKNOWN } } }
I20250901 01:52:24.171061 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet ad45c90ba0da43c9860745e534664a98 (table default.Table_1cd3f3fb6b244b3f8796bfca9f7622c8 [id=6fd739b0e725424ca844dbb652ea8028]). Expected version 2 got 0
I20250901 01:52:24.171279 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet fecf7d538b4c4c619182d5eccb44dd8b (table default.Table_1b5729d9768546fe99eaafc4e2604f42 [id=02d0056abe2146ac9af81ebf4447b99d]). Expected version 1 got 0
I20250901 01:52:24.171660 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 8ca5cf4e99014c0d84a3acad4ef713af (table default.Table_ed6fd284be6f4ae3a008e21e7ce17930 [id=303e76c918bc4c39aa47a4d24849a358]). Expected version 1 got 0
I20250901 01:52:24.171944 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 61ddcf86c9f34726939cc116434f0822 (table default.Table_63d0f2ead9d946d39ce40162ebd769d6 [id=f9dcf4937b1143248bfa9d88053fce17]). Expected version 3 got 0
I20250901 01:52:24.172164 1492 catalog_manager.cc:5582] T 3a8231590ea643b0a1f83eec4d9a8191 P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: UNKNOWN } } }
I20250901 01:52:24.172492 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 3a8231590ea643b0a1f83eec4d9a8191 (table default.Table_a4ebd8ab87c842f082da5cd04d1bddac [id=fd6d8e92d6e64a89a7d38aa761376955]). Expected version 1 got 0
I20250901 01:52:24.178347 2118 ts_tablet_manager.cc:1939] T eac50cbae691416e903cb2cbc907f96d P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:24.179404 2113 ts_tablet_manager.cc:1918] T eac50cbae691416e903cb2cbc907f96d P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:24.179565 2113 log.cc:1199] T eac50cbae691416e903cb2cbc907f96d P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/eac50cbae691416e903cb2cbc907f96d
I20250901 01:52:24.179904 2113 ts_tablet_manager.cc:1939] T eac50cbae691416e903cb2cbc907f96d P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:24.180287 1993 raft_consensus.cc:491] T 154574e17f214cc0ad56e6e6be3af899 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:24.180444 1993 raft_consensus.cc:513] T 154574e17f214cc0ad56e6e6be3af899 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:24.181095 1993 leader_election.cc:290] T 154574e17f214cc0ad56e6e6be3af899 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:24.181866 1151 tablet_bootstrap.cc:492] T e9b179dba2cc49428ba8090c962d6136 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
W20250901 01:52:24.182008 859 leader_election.cc:343] T 7a607fac57994001b2304e5d81c9decc P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:24.182081 1151 ts_tablet_manager.cc:1397] T e9b179dba2cc49428ba8090c962d6136 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20250901 01:52:24.182126 859 leader_election.cc:304] T 7a607fac57994001b2304e5d81c9decc P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:24.182473 2123 raft_consensus.cc:2747] T 7a607fac57994001b2304e5d81c9decc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:24.182814 859 leader_election.cc:343] T afc002ae2486476b9208e1cf8215157c P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:24.182917 859 leader_election.cc:304] T afc002ae2486476b9208e1cf8215157c P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:24.183216 2123 raft_consensus.cc:2747] T afc002ae2486476b9208e1cf8215157c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:24.183230 1151 raft_consensus.cc:357] T e9b179dba2cc49428ba8090c962d6136 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.183620 1993 consensus_queue.cc:1035] T 75e7a90815e449c39f386901a504143c P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:24.183672 992 leader_election.cc:304] T b7b802f976f3471faaaecc964ec87e48 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:24.183902 1872 raft_consensus.cc:2802] T b7b802f976f3471faaaecc964ec87e48 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:24.183984 1872 raft_consensus.cc:491] T b7b802f976f3471faaaecc964ec87e48 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:24.184098 1872 raft_consensus.cc:3058] T b7b802f976f3471faaaecc964ec87e48 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:24.183518 1151 raft_consensus.cc:383] T e9b179dba2cc49428ba8090c962d6136 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:24.184283 1151 raft_consensus.cc:738] T e9b179dba2cc49428ba8090c962d6136 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:24.184687 1151 consensus_queue.cc:260] T e9b179dba2cc49428ba8090c962d6136 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.186738 1872 raft_consensus.cc:513] T b7b802f976f3471faaaecc964ec87e48 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.187863 1872 leader_election.cc:290] T b7b802f976f3471faaaecc964ec87e48 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:24.188287 1872 consensus_queue.cc:1035] T 3c1fd0820d8a4995abe027f685da7f7b P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:24.189095 1151 ts_tablet_manager.cc:1428] T e9b179dba2cc49428ba8090c962d6136 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250901 01:52:24.189940 1151 tablet_bootstrap.cc:492] T 20f7bfab18134e079e91c52c89468cc3 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:24.191015 992 leader_election.cc:304] T 50e3da53433f4c14b57db8a733fa53bc P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:24.191358 1872 raft_consensus.cc:2802] T 50e3da53433f4c14b57db8a733fa53bc P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:24.191615 1872 raft_consensus.cc:695] T 50e3da53433f4c14b57db8a733fa53bc P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:24.191882 1872 consensus_queue.cc:237] T 50e3da53433f4c14b57db8a733fa53bc P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.192817 992 leader_election.cc:304] T 0cb3262973934c8689b851adad60afe0 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:24.193657 1993 raft_consensus.cc:2802] T 0cb3262973934c8689b851adad60afe0 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:24.193768 1993 raft_consensus.cc:695] T 0cb3262973934c8689b851adad60afe0 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:24.193940 1993 consensus_queue.cc:237] T 0cb3262973934c8689b851adad60afe0 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.195259 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet eac50cbae691416e903cb2cbc907f96d (table default.Table_d9e28b97a2e94ca8bf0ee8a01acd76e8 [id=e4a6d6fe44f947e59089e1d7855b785c]) successfully deleted
W20250901 01:52:24.195535 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet a4e3a8180fc14dacb1973e04f99c4492 because the tablet was not found. No further retry: Not found: Tablet not found: a4e3a8180fc14dacb1973e04f99c4492
I20250901 01:52:24.172724 1492 catalog_manager.cc:5582] T a6445b29591548be9f9577db1ba0bfa5 P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } }
I20250901 01:52:24.201835 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 89d40f0331114490b51de3ba6d773571 (table default.Table_4dacb3b1d2e94e42ac8937eb6eb58375 [id=8ee0aa1ba24042dd8ded004644913b9c]). Expected version 1 got 0
I20250901 01:52:24.203887 1492 catalog_manager.cc:5582] T 92f0ad59fbbb435ea872c669a52f854a P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: UNKNOWN } } }
I20250901 01:52:24.204766 1492 catalog_manager.cc:5582] T fae2779fac164b9f8052c3666db803b8 P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: UNKNOWN } } }
I20250901 01:52:24.205755 992 leader_election.cc:304] T e1d18e999b5e42b1873631e0f1445081 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:24.206265 1761 raft_consensus.cc:2802] T e1d18e999b5e42b1873631e0f1445081 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:24.206354 1761 raft_consensus.cc:491] T e1d18e999b5e42b1873631e0f1445081 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:24.206427 1761 raft_consensus.cc:3058] T e1d18e999b5e42b1873631e0f1445081 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:24.207422 1150 tablet_bootstrap.cc:492] T ad1a6be9fcc045d3bb4e71a2f6bd5ccd P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:24.207638 1150 ts_tablet_manager.cc:1397] T ad1a6be9fcc045d3bb4e71a2f6bd5ccd P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.063s user 0.016s sys 0.006s
I20250901 01:52:24.208652 1150 raft_consensus.cc:357] T ad1a6be9fcc045d3bb4e71a2f6bd5ccd P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.208885 1150 raft_consensus.cc:383] T ad1a6be9fcc045d3bb4e71a2f6bd5ccd P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:24.208900 1761 raft_consensus.cc:513] T e1d18e999b5e42b1873631e0f1445081 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.208958 1150 raft_consensus.cc:738] T ad1a6be9fcc045d3bb4e71a2f6bd5ccd P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:24.209194 1150 consensus_queue.cc:260] T ad1a6be9fcc045d3bb4e71a2f6bd5ccd P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.209992 1150 ts_tablet_manager.cc:1428] T ad1a6be9fcc045d3bb4e71a2f6bd5ccd P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250901 01:52:24.210466 1150 tablet_bootstrap.cc:492] T 615b2306fca84003b264da9d5f95ddab P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:24.212472 1150 tablet_bootstrap.cc:654] T 615b2306fca84003b264da9d5f95ddab P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:24.213536 2123 raft_consensus.cc:491] T 6789505328b1409ea288e8b4eaf94935 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:24.213655 2123 raft_consensus.cc:513] T 6789505328b1409ea288e8b4eaf94935 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.214180 2123 leader_election.cc:290] T 6789505328b1409ea288e8b4eaf94935 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:24.215852 1761 leader_election.cc:290] T e1d18e999b5e42b1873631e0f1445081 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:24.239918 992 leader_election.cc:304] T 49aaaec611784e6bac131efd58dbca51 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:24.240648 1872 raft_consensus.cc:2802] T 49aaaec611784e6bac131efd58dbca51 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:24.240823 1872 raft_consensus.cc:695] T 49aaaec611784e6bac131efd58dbca51 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:24.241134 1872 consensus_queue.cc:237] T 49aaaec611784e6bac131efd58dbca51 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.244374 2170 raft_consensus.cc:491] T a7ff996482164776b468cf305989866b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:24.244588 2170 raft_consensus.cc:513] T a7ff996482164776b468cf305989866b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:24.245355 2170 leader_election.cc:290] T a7ff996482164776b468cf305989866b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:24.245512 992 leader_election.cc:336] T 50e3da53433f4c14b57db8a733fa53bc P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:24.245800 992 consensus_peers.cc:489] T 5d794471dbed4477b9d52fd411fa4d10 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250901 01:52:24.245903 992 consensus_peers.cc:489] T f65ec84958eb488d8b0f812b59b5b3d5 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250901 01:52:24.245962 992 leader_election.cc:336] T b7b802f976f3471faaaecc964ec87e48 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:24.248204 992 leader_election.cc:336] T e1d18e999b5e42b1873631e0f1445081 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:24.248458 992 leader_election.cc:336] T 49aaaec611784e6bac131efd58dbca51 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:24.248660 992 leader_election.cc:336] T 0cb3262973934c8689b851adad60afe0 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:24.252357 858 leader_election.cc:336] T 6789505328b1409ea288e8b4eaf94935 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:24.255053 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet eac50cbae691416e903cb2cbc907f96d (table default.Table_d9e28b97a2e94ca8bf0ee8a01acd76e8 [id=e4a6d6fe44f947e59089e1d7855b785c]) successfully deleted
W20250901 01:52:24.258567 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet a4e3a8180fc14dacb1973e04f99c4492 because the tablet was not found. No further retry: Not found: Tablet not found: a4e3a8180fc14dacb1973e04f99c4492
I20250901 01:52:24.265054 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b7b802f976f3471faaaecc964ec87e48" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:24.265376 915 raft_consensus.cc:3058] T b7b802f976f3471faaaecc964ec87e48 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:24.268535 915 raft_consensus.cc:2466] T b7b802f976f3471faaaecc964ec87e48 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:24.273052 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e1d18e999b5e42b1873631e0f1445081" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:24.273258 915 raft_consensus.cc:3058] T e1d18e999b5e42b1873631e0f1445081 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:24.275681 1058 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ed1b41340ce742a88c3e4fd85cd25ca6" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:24.275812 915 raft_consensus.cc:2466] T e1d18e999b5e42b1873631e0f1445081 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:24.275885 1054 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6789505328b1409ea288e8b4eaf94935" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:24.276034 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "482a9a2cb81f448bb2823892e35920d4" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:24.278538 992 consensus_peers.cc:489] T fae2779fac164b9f8052c3666db803b8 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250901 01:52:24.282368 992 leader_election.cc:343] T 154574e17f214cc0ad56e6e6be3af899 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 154574e17f214cc0ad56e6e6be3af899
I20250901 01:52:24.282647 992 leader_election.cc:304] T b7b802f976f3471faaaecc964ec87e48 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:24.283505 2170 raft_consensus.cc:2802] T b7b802f976f3471faaaecc964ec87e48 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:24.283636 2170 raft_consensus.cc:695] T b7b802f976f3471faaaecc964ec87e48 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:24.283936 2170 consensus_queue.cc:237] T b7b802f976f3471faaaecc964ec87e48 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.284406 1151 tablet_bootstrap.cc:654] T 20f7bfab18134e079e91c52c89468cc3 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:24.283527 1150 tablet_bootstrap.cc:492] T 615b2306fca84003b264da9d5f95ddab P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:24.284801 1150 ts_tablet_manager.cc:1397] T 615b2306fca84003b264da9d5f95ddab P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.074s user 0.017s sys 0.021s
I20250901 01:52:24.287995 1150 raft_consensus.cc:357] T 615b2306fca84003b264da9d5f95ddab P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.288214 1150 raft_consensus.cc:383] T 615b2306fca84003b264da9d5f95ddab P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:24.288302 1150 raft_consensus.cc:738] T 615b2306fca84003b264da9d5f95ddab P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:24.288606 1150 consensus_queue.cc:260] T 615b2306fca84003b264da9d5f95ddab P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.289148 1150 ts_tablet_manager.cc:1428] T 615b2306fca84003b264da9d5f95ddab P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20250901 01:52:24.289574 1151 tablet_bootstrap.cc:492] T 20f7bfab18134e079e91c52c89468cc3 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:24.289750 1151 ts_tablet_manager.cc:1397] T 20f7bfab18134e079e91c52c89468cc3 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.100s user 0.020s sys 0.001s
I20250901 01:52:24.290853 1150 tablet_bootstrap.cc:492] T 05630a72ddae4de0b890dece2feacc31 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:24.290942 1151 raft_consensus.cc:357] T 20f7bfab18134e079e91c52c89468cc3 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.291160 1151 raft_consensus.cc:383] T 20f7bfab18134e079e91c52c89468cc3 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:24.291229 1151 raft_consensus.cc:738] T 20f7bfab18134e079e91c52c89468cc3 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:24.291462 1151 consensus_queue.cc:260] T 20f7bfab18134e079e91c52c89468cc3 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.292011 1151 ts_tablet_manager.cc:1428] T 20f7bfab18134e079e91c52c89468cc3 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250901 01:52:24.292299 1151 tablet_bootstrap.cc:492] T 4855448edb4b40c184838aa7f81c112b P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:24.292680 1150 tablet_bootstrap.cc:654] T 05630a72ddae4de0b890dece2feacc31 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:24.293521 1151 tablet_bootstrap.cc:654] T 4855448edb4b40c184838aa7f81c112b P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:24.300674 1490 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 81ce448b148447ddbc41875d863e0518 (table default.Table_6e461d9cb0ae4dd095975bbc21533794 [id=49673fa73f8c436e9b8026efc4f3cbbc]). Expected version 1 got 0
I20250901 01:52:24.301180 1490 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet cb7d7e1796c0456f8de98fb812b9e374 (table default.Table_c4148158ed7b48a3946559ba978e72fd [id=327c5a63904540508142ab600c308757]). Expected version 1 got 0
I20250901 01:52:24.301441 1490 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet fcb7b5b7d63d4f0b906fea97cb548de2 (table default.Table_2ceaff93c8f44f00aab53e48d8466dfd [id=4d9ea39016ca4bf4859f4f80ea4e9ed6]). Expected version 4 got 0
I20250901 01:52:24.301702 1490 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet b164b4a67a9143ceaff69b0ce49d63fe (table default.Table_da2c70a8f2fe49a0a8184e88c0a9e0ea [id=c1b1e9e3876244f68d31c63ce7bdebf1]). Expected version 1 got 0
I20250901 01:52:24.302274 1490 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet d1f2d98712ff4ea6aec665512de068bd (table default.Table_82d91a1ba2eb4265896824028dec8d3a [id=3f7624d2901445e39155c12fa68e34e9]). Expected version 1 got 0
I20250901 01:52:24.302672 1490 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet fa7fb88ded21432a8757f37ec3db5e71 (table default.Table_22d8d38d0b624912a0d8968fb764420e [id=808a8849c99e485b8be7cd17397874ec]). Expected version 1 got 0
I20250901 01:52:24.302992 1490 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 410d44e5f1974ed69e5dee65e98856a4 (table default.Table_bec114a616504798bede9a9066e57192 [id=ae79f2c40fd7483ea400a36cf3c48888]). Expected version 4 got 0
I20250901 01:52:24.303332 1490 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 1495de70f0844f82a0eea99c059f076b (table default.Table_02e911ce389e4ad581c0a2f976f99c4e [id=70a5ce71d16b47d4bc6dae97be4548e7]). Expected version 1 got 0
I20250901 01:52:24.304278 1490 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet d10c193452954c198bba686169b8025b (table default.Table_03fb48eaab9b42858ab7cefb722e43a8 [id=6c0576410d7340d6aa493a747e95565b]). Expected version 1 got 0
I20250901 01:52:24.304610 1490 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 435b1cca5a7d4d49959a310cb250b2a0 (table default.Table_dc9c5d18c7ec478592a76dbd76424b88 [id=126b298fef6c4653952d672adff1db7e]). Expected version 1 got 0
I20250901 01:52:24.304941 1490 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet ed1b41340ce742a88c3e4fd85cd25ca6 (table default.Table_139f6d174d794a6e9a618e90587d0c0d [id=0baa4993b72e4a8d94ee3059c9d2db6c]). Expected version 1 got 0
I20250901 01:52:24.305333 1490 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet c6f7c525f59b49938b697de98d4b5835 (table default.Table_db61e055dbd7450898e82c4c0c4763be [id=aabca67f6f754820977ac9c4c20f3e4f]). Expected version 1 got 0
I20250901 01:52:24.305629 1490 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 7954393c8c1c4457ab8bb723c03a9c8b (table default.Table_5ea919f20c6b4a4e91e599357a17a53a [id=543f2074598d4c29972e564be1123fca]). Expected version 1 got 0
I20250901 01:52:24.305928 1490 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 6c0abbfc8d904dce989ce51961e4237d (table default.Table_3f9422b08b0349b9a5466218f0fe1b06 [id=f0dfb3811e744bd4ba6d59195cebde6f]). Expected version 2 got 0
I20250901 01:52:24.306471 1490 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet e87d1281f35648b0b115b67c95e244e7 (table default.Table_f840789a58984799bd8057128f6ea439 [id=4453959e273c473fadd737f56f7db3dd]). Expected version 1 got 0
I20250901 01:52:24.310078 1490 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 606fac5c6efe48d1877104b01c23ba5b (table default.Table_7c12d66048a6470e8c227dc1d9f45fa4 [id=91472f220fe44297b79d3eec8b9b80e8]). Expected version 1 got 0
I20250901 01:52:24.311075 1151 tablet_bootstrap.cc:492] T 4855448edb4b40c184838aa7f81c112b P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:24.311316 1490 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet d0c3c363dda444519a17dca63ad9caa7 (table default.Table_0b4ac0a1caa5472d909448e41318e679 [id=8c1b263824ec4c15a5ec424364c78895]). Expected version 2 got 0
I20250901 01:52:24.311749 1490 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet e6c10426f60f43e8bde69d9b1b9c8eb6 (table default.Table_eecfab7239cc40c5a2f1fb6ac6d35dc9 [id=d3dbeb4258424133844e20b181ee44e4]). Expected version 1 got 0
I20250901 01:52:24.312028 1490 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 615b2306fca84003b264da9d5f95ddab (table default.Table_0f959237963c4e33932e193285c0eacb [id=cecdd8d6ca6d445c9fd82decd2a95366]). Expected version 1 got 0
I20250901 01:52:24.311203 2171 consensus_queue.cc:1035] T b6daeff0070a4db08e90020752d21c29 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:24.312276 1151 ts_tablet_manager.cc:1397] T 4855448edb4b40c184838aa7f81c112b P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.001s
I20250901 01:52:24.312043 992 leader_election.cc:304] T e1d18e999b5e42b1873631e0f1445081 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:24.312621 2171 raft_consensus.cc:2802] T e1d18e999b5e42b1873631e0f1445081 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:24.312703 1490 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet fecf7d538b4c4c619182d5eccb44dd8b (table default.Table_1b5729d9768546fe99eaafc4e2604f42 [id=02d0056abe2146ac9af81ebf4447b99d]). Expected version 1 got 0
I20250901 01:52:24.312811 2171 raft_consensus.cc:695] T e1d18e999b5e42b1873631e0f1445081 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:24.312932 1490 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 629db80a8ab444e582628578155638fd (table default.Table_f0e56dba1e8a4ed49a760e204c1ff96e [id=c3865e058daa4cea9a89f5046fca712a]). Expected version 1 got 0
W20250901 01:52:24.312624 992 leader_election.cc:343] T a7ff996482164776b468cf305989866b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: a7ff996482164776b468cf305989866b
I20250901 01:52:24.313108 1490 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 75a703c8e20a4a2fa0ab6ec58d41ef92 (table default.Table_2f09538b6fbe401f909105decb56e1cb [id=a90d1f10531042c3bb7a196b8be61c5f]). Expected version 1 got 0
I20250901 01:52:24.313041 2171 consensus_queue.cc:237] T e1d18e999b5e42b1873631e0f1445081 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.313421 1490 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 89d40f0331114490b51de3ba6d773571 (table default.Table_4dacb3b1d2e94e42ac8937eb6eb58375 [id=8ee0aa1ba24042dd8ded004644913b9c]). Expected version 1 got 0
W20250901 01:52:24.313450 859 leader_election.cc:343] T ed1b41340ce742a88c3e4fd85cd25ca6 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:24.313565 859 leader_election.cc:304] T ed1b41340ce742a88c3e4fd85cd25ca6 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:24.313623 1490 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet ae21b9ae2e3b472eb67fb5f96388a50f (table default.Table_046289f393ff432aae0b033bdec1fb17 [id=db57b18a9ce24e068b362f5fbe940531]). Expected version 1 got 0
I20250901 01:52:24.313833 1490 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 3618a3eb4f174c43805ce7dc0e5315cb (table default.Table_83f2c0acedbc4c6982caf5313646e90b [id=31a2e50c32644321825da40176127015]). Expected version 1 got 0
I20250901 01:52:24.313956 2123 raft_consensus.cc:2747] T ed1b41340ce742a88c3e4fd85cd25ca6 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:24.314097 1981 raft_consensus.cc:491] T 7954393c8c1c4457ab8bb723c03a9c8b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:24.314181 2170 consensus_queue.cc:1035] T a54af99bc10045e8b0b6868e5aed98b4 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:24.314249 1490 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet f80f13db501c42c7950499df8fcb6fc8 (table default.Table_ce86974fd8974de9a5a737954fac2ad7 [id=ff772d20b287476f94716379b86d2a71]). Expected version 1 got 0
I20250901 01:52:24.314203 1981 raft_consensus.cc:513] T 7954393c8c1c4457ab8bb723c03a9c8b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.314765 1981 leader_election.cc:290] T 7954393c8c1c4457ab8bb723c03a9c8b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:24.315270 1490 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet d4c9aa723aee4dc4b6237cb5c3722b52 (table default.Table_5e65c5f068b84995a2d7a76fa65dda0f [id=b9d81ddd38fe47838877b010be2b40d0]). Expected version 1 got 0
I20250901 01:52:24.315878 1490 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet b009cc95c819431ea9827a3e453b8da3 (table default.Table_203547c9458e43b3b48ea051d56892c6 [id=cc26c89db2484a6fad919231116285f2]). Expected version 1 got 0
W20250901 01:52:24.316551 858 leader_election.cc:336] T 7954393c8c1c4457ab8bb723c03a9c8b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:24.317160 1151 raft_consensus.cc:357] T 4855448edb4b40c184838aa7f81c112b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:24.317559 1150 tablet_bootstrap.cc:492] T 05630a72ddae4de0b890dece2feacc31 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:24.317698 1150 ts_tablet_manager.cc:1397] T 05630a72ddae4de0b890dece2feacc31 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.028s user 0.010s sys 0.012s
I20250901 01:52:24.318394 1151 raft_consensus.cc:383] T 4855448edb4b40c184838aa7f81c112b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:24.318526 1151 raft_consensus.cc:738] T 4855448edb4b40c184838aa7f81c112b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:24.318631 1150 raft_consensus.cc:357] T 05630a72ddae4de0b890dece2feacc31 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:24.318861 1150 raft_consensus.cc:383] T 05630a72ddae4de0b890dece2feacc31 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:24.318928 1150 raft_consensus.cc:738] T 05630a72ddae4de0b890dece2feacc31 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:24.319131 1150 consensus_queue.cc:260] T 05630a72ddae4de0b890dece2feacc31 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:24.319350 1151 consensus_queue.cc:260] T 4855448edb4b40c184838aa7f81c112b P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:24.320770 1150 ts_tablet_manager.cc:1428] T 05630a72ddae4de0b890dece2feacc31 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250901 01:52:24.321450 1151 ts_tablet_manager.cc:1428] T 4855448edb4b40c184838aa7f81c112b P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20250901 01:52:24.321695 1150 tablet_bootstrap.cc:492] T d10c193452954c198bba686169b8025b P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:24.321866 1151 tablet_bootstrap.cc:492] T f4fd117c395e41c2ab679c684908c5b7 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:24.322508 1993 raft_consensus.cc:491] T 3c12d6a70ac24dce82e851782e6fe5aa P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:24.322616 1993 raft_consensus.cc:513] T 3c12d6a70ac24dce82e851782e6fe5aa P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.323127 1993 leader_election.cc:290] T 3c12d6a70ac24dce82e851782e6fe5aa P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:24.323236 1150 tablet_bootstrap.cc:654] T d10c193452954c198bba686169b8025b P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:24.325701 1150 tablet_bootstrap.cc:492] T d10c193452954c198bba686169b8025b P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:24.325819 1150 ts_tablet_manager.cc:1397] T d10c193452954c198bba686169b8025b P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20250901 01:52:24.326440 1150 raft_consensus.cc:357] T d10c193452954c198bba686169b8025b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.326625 1150 raft_consensus.cc:383] T d10c193452954c198bba686169b8025b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:24.326694 1150 raft_consensus.cc:738] T d10c193452954c198bba686169b8025b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:24.326908 1150 consensus_queue.cc:260] T d10c193452954c198bba686169b8025b P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.327277 1151 tablet_bootstrap.cc:654] T f4fd117c395e41c2ab679c684908c5b7 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:24.327584 1490 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 93bcac1fd95d44dc927abfca81692921 (table default.Table_b8f8ee74648c406eaf589d3ab80934db [id=b8f69313439543be9c11eea88dfa69c5]). Expected version 1 got 0
I20250901 01:52:24.328107 1150 ts_tablet_manager.cc:1428] T d10c193452954c198bba686169b8025b P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250901 01:52:24.328444 1490 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet e9f1f892658b4736859ff507536920a1 (table default.Table_8825dd28afbe464193eb1b668cc9326a [id=9aa62cb634314a0a849612c54c1220d9]). Expected version 1 got 0
I20250901 01:52:24.328696 1490 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 617cc33d0d23439cb5f1e0c31bf11281 (table default.Table_ce1fd26ac65a4ed085d023e10a849e5c [id=fb088d31bba64f04871fbfd355d6263b]). Expected version 1 got 0
I20250901 01:52:24.329108 1150 tablet_bootstrap.cc:492] T 56475ba491a24301a534304811f30140 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:24.329118 1490 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 7f46e9ddadaf4ff986d8c991178daa0c (table default.Table_15c9f4f02eb24474936a43d705f90f7b [id=ea928501fe954cd0b81d38add4ef7128]). Expected version 1 got 0
I20250901 01:52:24.329653 1490 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 5f5396045979450193d461ba1334a5c9 (table default.Table_a89cc8e48563460baabc34c821ceae35 [id=9c3930e9fcfe4b49ba4bea67a2b0ad07]). Expected version 1 got 0
I20250901 01:52:24.330235 1490 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 3dae00aa38204ee3ac77325eb334be42 (table default.Table_faf7ef12324f454396e5ae64d64ebe51 [id=5812902df9ea406ba59ff0ff23f130b6]). Expected version 1 got 0
I20250901 01:52:24.330438 1150 tablet_bootstrap.cc:654] T 56475ba491a24301a534304811f30140 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:24.331148 1490 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 33f637f6a8134215bb1995ec38440073 (table default.Table_2ece91972d494680a8ad00273c10dee4 [id=557afe29976d40f6bfb6ae4ca348f887]). Expected version 1 got 0
I20250901 01:52:24.331821 2171 raft_consensus.cc:491] T e9b179dba2cc49428ba8090c962d6136 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:24.331940 2171 raft_consensus.cc:513] T e9b179dba2cc49428ba8090c962d6136 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.332489 2171 leader_election.cc:290] T e9b179dba2cc49428ba8090c962d6136 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:24.332803 1490 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet f62d96c37cca4143a4b098b7c69979e6 (table default.Table_0877ac8202a24e9da650a589bc6dbfcc [id=1f65b3e09c9748f189ecb2b56c5d1132]). Expected version 1 got 0
W20250901 01:52:24.333462 859 leader_election.cc:343] T 6789505328b1409ea288e8b4eaf94935 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:24.333598 859 leader_election.cc:304] T 6789505328b1409ea288e8b4eaf94935 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:24.333972 859 leader_election.cc:343] T 482a9a2cb81f448bb2823892e35920d4 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:24.333948 1490 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 5adf6b492640417ca071c94702b48cb9 (table default.Table_589c302e02474fc3bc5d3a290a8bd54a [id=5d14706774d345c48ca3575feabb17b0]). Expected version 2 got 0
I20250901 01:52:24.334076 859 leader_election.cc:304] T 482a9a2cb81f448bb2823892e35920d4 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:24.334295 2123 raft_consensus.cc:2747] T 6789505328b1409ea288e8b4eaf94935 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:24.334422 1981 raft_consensus.cc:2747] T 482a9a2cb81f448bb2823892e35920d4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:24.334456 1490 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 246b33338de647689d7404c70676e76f (table default.Table_219f0092f43743039ae1a61fedb3e4d1 [id=a7b1d2a8ace14ef8bcce0a3bed41e85d]). Expected version 1 got 0
I20250901 01:52:24.335680 1490 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 61ddcf86c9f34726939cc116434f0822 (table default.Table_63d0f2ead9d946d39ce40162ebd769d6 [id=f9dcf4937b1143248bfa9d88053fce17]). Expected version 3 got 0
I20250901 01:52:24.335961 1150 tablet_bootstrap.cc:492] T 56475ba491a24301a534304811f30140 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:24.336081 1150 ts_tablet_manager.cc:1397] T 56475ba491a24301a534304811f30140 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.002s
I20250901 01:52:24.336354 1490 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 3a8231590ea643b0a1f83eec4d9a8191 (table default.Table_a4ebd8ab87c842f082da5cd04d1bddac [id=fd6d8e92d6e64a89a7d38aa761376955]). Expected version 1 got 0
I20250901 01:52:24.337293 1150 raft_consensus.cc:357] T 56475ba491a24301a534304811f30140 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:24.337468 1150 raft_consensus.cc:383] T 56475ba491a24301a534304811f30140 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:24.337528 1150 raft_consensus.cc:738] T 56475ba491a24301a534304811f30140 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:24.337749 1150 consensus_queue.cc:260] T 56475ba491a24301a534304811f30140 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:24.338270 1150 ts_tablet_manager.cc:1428] T 56475ba491a24301a534304811f30140 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250901 01:52:24.338230 1490 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 8ca5cf4e99014c0d84a3acad4ef713af (table default.Table_ed6fd284be6f4ae3a008e21e7ce17930 [id=303e76c918bc4c39aa47a4d24849a358]). Expected version 1 got 0
I20250901 01:52:24.339243 1150 tablet_bootstrap.cc:492] T 1a60c10e42104502ac2d3b80b8126ceb P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:24.341133 1490 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet ad45c90ba0da43c9860745e534664a98 (table default.Table_1cd3f3fb6b244b3f8796bfca9f7622c8 [id=6fd739b0e725424ca844dbb652ea8028]). Expected version 2 got 0
I20250901 01:52:24.341338 1150 tablet_bootstrap.cc:654] T 1a60c10e42104502ac2d3b80b8126ceb P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:24.341675 1490 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 72ba689f9230411995dc96e2b45601df (table default.Table_604f2c5c58d74154bacf2a065a467f32 [id=351a05134f4948039c920cafbe050914]). Expected version 1 got 0
I20250901 01:52:24.344020 918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3c12d6a70ac24dce82e851782e6fe5aa" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:24.344244 918 raft_consensus.cc:2466] T 3c12d6a70ac24dce82e851782e6fe5aa P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:24.345248 918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9b179dba2cc49428ba8090c962d6136" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:24.345422 918 raft_consensus.cc:2466] T e9b179dba2cc49428ba8090c962d6136 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
W20250901 01:52:24.346222 992 leader_election.cc:336] T e1d18e999b5e42b1873631e0f1445081 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:24.351024 1490 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 3c1fd0820d8a4995abe027f685da7f7b (table default.Table_bcd502e1360b48b1953fa97eadce7006 [id=b3914f33e856466e8479d0a2263ec56f]). Expected version 1 got 0
I20250901 01:52:24.351692 1150 tablet_bootstrap.cc:492] T 1a60c10e42104502ac2d3b80b8126ceb P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:24.351845 1150 ts_tablet_manager.cc:1397] T 1a60c10e42104502ac2d3b80b8126ceb P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.001s
I20250901 01:52:24.352306 1490 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 3c12d6a70ac24dce82e851782e6fe5aa (table default.Table_d2049d0c20bb4949ae6c249b690e7f86 [id=26bba0e60b6b43eb87d084be403c556b]). Expected version 1 got 0
I20250901 01:52:24.352741 1150 raft_consensus.cc:357] T 1a60c10e42104502ac2d3b80b8126ceb P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.352937 1150 raft_consensus.cc:383] T 1a60c10e42104502ac2d3b80b8126ceb P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:24.352996 1150 raft_consensus.cc:738] T 1a60c10e42104502ac2d3b80b8126ceb P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:24.353211 1150 consensus_queue.cc:260] T 1a60c10e42104502ac2d3b80b8126ceb P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.353852 1150 ts_tablet_manager.cc:1428] T 1a60c10e42104502ac2d3b80b8126ceb P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250901 01:52:24.353343 1490 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet ddbc15685fca4c708329420109ad0c57 (table default.Table_942175dadf4b496195b2fa53f39f6ddc [id=2bd332b452f84f25af4225af8e6c6740]). Expected version 1 got 0
I20250901 01:52:24.354156 1150 tablet_bootstrap.cc:492] T 0dbdcb76eeba4fefa56e7835c23c0316 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:24.354367 1490 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 49aaaec611784e6bac131efd58dbca51 (table default.Table_95e1338a49a04f22b679e363e253262e [id=e24b91b4ac904ef0a8c4725f70bced2f]). Expected version 1 got 0
I20250901 01:52:24.349960 2171 raft_consensus.cc:491] T 25ce0fe8c9544f828694184d19d13df9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:24.355101 2171 raft_consensus.cc:513] T 25ce0fe8c9544f828694184d19d13df9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:24.355633 2171 leader_election.cc:290] T 25ce0fe8c9544f828694184d19d13df9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:24.356127 1150 tablet_bootstrap.cc:654] T 0dbdcb76eeba4fefa56e7835c23c0316 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:24.356266 992 leader_election.cc:336] T a7ff996482164776b468cf305989866b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:24.356372 992 leader_election.cc:304] T a7ff996482164776b468cf305989866b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:24.356611 2171 raft_consensus.cc:2747] T a7ff996482164776b468cf305989866b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:24.356846 992 consensus_peers.cc:489] T 9127f9c5239e445ba86012951b7c75d9 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250901 01:52:24.356950 992 leader_election.cc:336] T 154574e17f214cc0ad56e6e6be3af899 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:24.357024 992 leader_election.cc:304] T 154574e17f214cc0ad56e6e6be3af899 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
W20250901 01:52:24.357256 992 leader_election.cc:336] T b7b802f976f3471faaaecc964ec87e48 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:24.357483 2171 raft_consensus.cc:2747] T 154574e17f214cc0ad56e6e6be3af899 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:24.357596 1490 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet b2e8b2cd10154de08ddd30c214592a4a (table default.Table_4a412a59b6db421eb0187ffd02ecf541 [id=fd72e46c84634b6c9ea74970480ecc62]). Expected version 4 got 0
I20250901 01:52:24.358048 1490 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet e1d18e999b5e42b1873631e0f1445081 (table default.Table_fd17a2ad225546aa821489ce94a788a2 [id=033c76e0598342c495ce2be42a0df5dd]). Expected version 1 got 0
I20250901 01:52:24.358450 1873 consensus_queue.cc:1035] T d77aadd7291647a0bae7c421f244c252 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:24.358630 2171 consensus_queue.cc:1035] T 92f0ad59fbbb435ea872c669a52f854a P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:24.359167 1490 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 616ff34e8d314e0aa0e11ce60be11789 (table default.Table_0451d0afa26f4d5f8d4214a946eea04f [id=5ee13d46e06a4fa59b1070547573e523]). Expected version 1 got 0
I20250901 01:52:24.359791 1490 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 42c1156e8dd84c6fa1396b46de13eb6f (table default.Table_b5d681f51fc84395bbabe1e5b4cfb358 [id=40d05d90b0354334840ad17764f5b6bb]). Expected version 1 got 0
I20250901 01:52:24.360751 1151 tablet_bootstrap.cc:492] T f4fd117c395e41c2ab679c684908c5b7 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:24.360929 1151 ts_tablet_manager.cc:1397] T f4fd117c395e41c2ab679c684908c5b7 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.039s user 0.004s sys 0.000s
I20250901 01:52:24.361518 1490 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 2599f98abfdc4871b573edfc2ad97d22 (table default.Table_0e63d21acfcb4c028c9ea6406b6a04e4 [id=81323e6fbe424655b8e185f52657fadc]). Expected version 1 got 0
I20250901 01:52:24.361793 1151 raft_consensus.cc:357] T f4fd117c395e41c2ab679c684908c5b7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.362017 1151 raft_consensus.cc:383] T f4fd117c395e41c2ab679c684908c5b7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:24.362093 1151 raft_consensus.cc:738] T f4fd117c395e41c2ab679c684908c5b7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:24.362372 1151 consensus_queue.cc:260] T f4fd117c395e41c2ab679c684908c5b7 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.362902 1490 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 081e2630e284498aa819f85bf8326315 (table default.Table_9e7ca61fcc324a8984bd1e470162c61f [id=8199118617cb4707bc3ae9b1363750f7]). Expected version 2 got 0
I20250901 01:52:24.363154 1151 ts_tablet_manager.cc:1428] T f4fd117c395e41c2ab679c684908c5b7 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250901 01:52:24.363406 1490 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 8ed6616ce54548ba83ba299c4ecf3c38 (table default.Table_1bdd9041d74045cab6f23de11bf4dd09 [id=946ec3e53dcd4de4b8501b3e87e99072]). Expected version 2 got 0
I20250901 01:52:24.363615 1151 tablet_bootstrap.cc:492] T 64fb46b942ff4e4c8749986ecc3dfe65 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:24.365319 1151 tablet_bootstrap.cc:654] T 64fb46b942ff4e4c8749986ecc3dfe65 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:24.366278 1485 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet f98d1d8d660443cb8066a547e5d9b77f
W20250901 01:52:24.366516 1485 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet f98d1d8d660443cb8066a547e5d9b77f: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:24.367311 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4e8bdf984374478497bc83c80f8cbc97"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.367942 1150 tablet_bootstrap.cc:492] T 0dbdcb76eeba4fefa56e7835c23c0316 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:24.368070 1150 ts_tablet_manager.cc:1397] T 0dbdcb76eeba4fefa56e7835c23c0316 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.000s
I20250901 01:52:24.368801 1150 raft_consensus.cc:357] T 0dbdcb76eeba4fefa56e7835c23c0316 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.368966 1150 raft_consensus.cc:383] T 0dbdcb76eeba4fefa56e7835c23c0316 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:24.369011 1150 raft_consensus.cc:738] T 0dbdcb76eeba4fefa56e7835c23c0316 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:24.370361 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet f98d1d8d660443cb8066a547e5d9b77f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:23 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:24.371248 992 leader_election.cc:304] T 3c12d6a70ac24dce82e851782e6fe5aa P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:24.371464 1761 raft_consensus.cc:2802] T 3c12d6a70ac24dce82e851782e6fe5aa P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:24.371531 1761 raft_consensus.cc:491] T 3c12d6a70ac24dce82e851782e6fe5aa P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:24.371589 1761 raft_consensus.cc:3058] T 3c12d6a70ac24dce82e851782e6fe5aa P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
W20250901 01:52:24.371592 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet f98d1d8d660443cb8066a547e5d9b77f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f98d1d8d660443cb8066a547e5d9b77f already in progress: creating tablet
I20250901 01:52:24.371743 885 tablet_service.cc:1515] Processing DeleteTablet for tablet f98d1d8d660443cb8066a547e5d9b77f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:23 UTC) from {username='slave'} at 127.0.0.1:34472
W20250901 01:52:24.372581 1472 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet f98d1d8d660443cb8066a547e5d9b77f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f98d1d8d660443cb8066a547e5d9b77f already in progress: creating tablet
I20250901 01:52:24.372879 1151 tablet_bootstrap.cc:492] T 64fb46b942ff4e4c8749986ecc3dfe65 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:24.372995 1151 ts_tablet_manager.cc:1397] T 64fb46b942ff4e4c8749986ecc3dfe65 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.004s
I20250901 01:52:24.373188 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_58d362199b3d419fbc3a94c02d1db9af"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.373662 1151 raft_consensus.cc:357] T 64fb46b942ff4e4c8749986ecc3dfe65 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.373802 1151 raft_consensus.cc:383] T 64fb46b942ff4e4c8749986ecc3dfe65 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:24.373845 1151 raft_consensus.cc:738] T 64fb46b942ff4e4c8749986ecc3dfe65 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:24.374030 1151 consensus_queue.cc:260] T 64fb46b942ff4e4c8749986ecc3dfe65 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.374349 1761 raft_consensus.cc:513] T 3c12d6a70ac24dce82e851782e6fe5aa P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.374387 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_915030d2d1454f83b830189d2561877a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.374661 1151 ts_tablet_manager.cc:1428] T 64fb46b942ff4e4c8749986ecc3dfe65 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250901 01:52:24.374907 1761 leader_election.cc:290] T 3c12d6a70ac24dce82e851782e6fe5aa P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:24.374905 1151 tablet_bootstrap.cc:492] T aa424ac6ef1441e6b1cc2867b1c48186 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:24.375061 1150 consensus_queue.cc:260] T 0dbdcb76eeba4fefa56e7835c23c0316 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.375335 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7dcca5a6b6eb4800831bdd70f0c5a064"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.376186 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9759e34deef34f48b8691a564ded6e4d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.376379 1150 ts_tablet_manager.cc:1428] T 0dbdcb76eeba4fefa56e7835c23c0316 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250901 01:52:24.376834 1151 tablet_bootstrap.cc:654] T aa424ac6ef1441e6b1cc2867b1c48186 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:24.377547 1150 tablet_bootstrap.cc:492] T 1495de70f0844f82a0eea99c059f076b P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:24.378868 992 leader_election.cc:304] T e9b179dba2cc49428ba8090c962d6136 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:24.379024 1150 tablet_bootstrap.cc:654] T 1495de70f0844f82a0eea99c059f076b P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:24.379230 2171 raft_consensus.cc:2802] T e9b179dba2cc49428ba8090c962d6136 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:24.379335 2171 raft_consensus.cc:491] T e9b179dba2cc49428ba8090c962d6136 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:24.379420 2171 raft_consensus.cc:3058] T e9b179dba2cc49428ba8090c962d6136 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:24.381392 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8690bbc654764940964257614cc06d09"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.382261 2171 raft_consensus.cc:513] T e9b179dba2cc49428ba8090c962d6136 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.382983 2171 leader_election.cc:290] T e9b179dba2cc49428ba8090c962d6136 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:24.383385 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7954393c8c1c4457ab8bb723c03a9c8b" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:24.384982 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_77d9aca203634ae4a55dce6d0f4e5879"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.387471 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7729c5b9ce9d4f05b1bd644dc5379aef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:24.389964 1486 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet da5f69c5b0ca4257ad2ff7918ee2af2e consensus state has no leader
I20250901 01:52:24.390429 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1018e4a4bbb14b118c9e809f89b5851c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.391376 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_bb4dbbb6a0674830bc103bd658713feb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.392161 1486 master_service.cc:948] ReplaceTablet: received request to replace tablet 75345ea2a3dc40fc8c9f8090d8839d2d from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:24.392493 1150 tablet_bootstrap.cc:492] T 1495de70f0844f82a0eea99c059f076b P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:24.392656 1150 ts_tablet_manager.cc:1397] T 1495de70f0844f82a0eea99c059f076b P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.001s
I20250901 01:52:24.392803 921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25ce0fe8c9544f828694184d19d13df9" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:24.392974 921 raft_consensus.cc:2466] T 25ce0fe8c9544f828694184d19d13df9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:24.393605 1981 raft_consensus.cc:491] T 33f637f6a8134215bb1995ec38440073 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:24.393522 1150 raft_consensus.cc:357] T 1495de70f0844f82a0eea99c059f076b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.393697 1150 raft_consensus.cc:383] T 1495de70f0844f82a0eea99c059f076b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:24.393762 1150 raft_consensus.cc:738] T 1495de70f0844f82a0eea99c059f076b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:24.393715 1981 raft_consensus.cc:513] T 33f637f6a8134215bb1995ec38440073 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.393971 1150 consensus_queue.cc:260] T 1495de70f0844f82a0eea99c059f076b P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.394326 1981 leader_election.cc:290] T 33f637f6a8134215bb1995ec38440073 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:24.396071 1483 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 2ff914352c7c43de800f3d54ecf63072
W20250901 01:52:24.396302 1483 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 2ff914352c7c43de800f3d54ecf63072: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
W20250901 01:52:24.396862 858 leader_election.cc:336] T 33f637f6a8134215bb1995ec38440073 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:24.396958 1483 master_service.cc:948] ReplaceTablet: received request to replace tablet 246b33338de647689d7404c70676e76f from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:24.397545 1150 ts_tablet_manager.cc:1428] T 1495de70f0844f82a0eea99c059f076b P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250901 01:52:24.397679 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 2ff914352c7c43de800f3d54ecf63072 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:23 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:24.398279 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 2ff914352c7c43de800f3d54ecf63072 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:23 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:24.398506 2118 tablet_replica.cc:331] T 2ff914352c7c43de800f3d54ecf63072 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:24.398708 2118 raft_consensus.cc:2241] T 2ff914352c7c43de800f3d54ecf63072 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:24.399020 1485 master_service.cc:948] ReplaceTablet: received request to replace tablet 579b9cdf819f4609a532e96bec093fd5 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:24.399084 2118 raft_consensus.cc:2270] T 2ff914352c7c43de800f3d54ecf63072 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:24.399751 2118 ts_tablet_manager.cc:1905] T 2ff914352c7c43de800f3d54ecf63072 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:24.409596 2118 ts_tablet_manager.cc:1918] T 2ff914352c7c43de800f3d54ecf63072 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:24.409762 2118 log.cc:1199] T 2ff914352c7c43de800f3d54ecf63072 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/2ff914352c7c43de800f3d54ecf63072
I20250901 01:52:24.410176 2118 ts_tablet_manager.cc:1939] T 2ff914352c7c43de800f3d54ecf63072 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:24.411518 1150 tablet_bootstrap.cc:492] T b81a12b38ac64efabd44aeb192d181a0 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:24.414058 1150 tablet_bootstrap.cc:654] T b81a12b38ac64efabd44aeb192d181a0 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:24.416277 917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3c12d6a70ac24dce82e851782e6fe5aa" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:24.416486 917 raft_consensus.cc:3058] T 3c12d6a70ac24dce82e851782e6fe5aa P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:24.417991 1150 tablet_bootstrap.cc:492] T b81a12b38ac64efabd44aeb192d181a0 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:24.418099 1150 ts_tablet_manager.cc:1397] T b81a12b38ac64efabd44aeb192d181a0 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.000s
I20250901 01:52:24.418870 1150 raft_consensus.cc:357] T b81a12b38ac64efabd44aeb192d181a0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.419013 1150 raft_consensus.cc:383] T b81a12b38ac64efabd44aeb192d181a0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:24.419068 1150 raft_consensus.cc:738] T b81a12b38ac64efabd44aeb192d181a0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:24.419237 917 raft_consensus.cc:2466] T 3c12d6a70ac24dce82e851782e6fe5aa P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:24.419309 1150 consensus_queue.cc:260] T b81a12b38ac64efabd44aeb192d181a0 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.420701 1150 ts_tablet_manager.cc:1428] T b81a12b38ac64efabd44aeb192d181a0 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250901 01:52:24.420847 921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9b179dba2cc49428ba8090c962d6136" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:24.421016 921 raft_consensus.cc:3058] T e9b179dba2cc49428ba8090c962d6136 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:24.421628 1150 tablet_bootstrap.cc:492] T 48a211e174054866b030cdfb0cf947f4 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:24.422489 1151 tablet_bootstrap.cc:492] T aa424ac6ef1441e6b1cc2867b1c48186 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:24.423367 1150 tablet_bootstrap.cc:654] T 48a211e174054866b030cdfb0cf947f4 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:24.423643 859 leader_election.cc:343] T 7954393c8c1c4457ab8bb723c03a9c8b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:24.423758 859 leader_election.cc:304] T 7954393c8c1c4457ab8bb723c03a9c8b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:24.424202 2123 raft_consensus.cc:2747] T 7954393c8c1c4457ab8bb723c03a9c8b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:24.426396 921 raft_consensus.cc:2466] T e9b179dba2cc49428ba8090c962d6136 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:24.431893 1151 ts_tablet_manager.cc:1397] T aa424ac6ef1441e6b1cc2867b1c48186 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.057s user 0.003s sys 0.001s
W20250901 01:52:24.424113 992 leader_election.cc:336] T 3c12d6a70ac24dce82e851782e6fe5aa P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:24.438870 992 leader_election.cc:336] T e9b179dba2cc49428ba8090c962d6136 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:24.440033 992 consensus_peers.cc:489] T 72ba689f9230411995dc96e2b45601df P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250901 01:52:24.399392 1488 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 36e10328c34242ebb4dbd8f294162696 consensus state has no leader
I20250901 01:52:24.442652 1488 master_service.cc:948] ReplaceTablet: received request to replace tablet 26c8ed083cea4dc0acca80d00b11b29b from {username='slave'} at 127.0.0.1:33478
W20250901 01:52:24.400211 1484 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7f46e9ddadaf4ff986d8c991178daa0c consensus state has no leader
I20250901 01:52:24.443275 1484 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_0f959237963c4e33932e193285c0eacb" } new_table_name: "default.Table_302f6e7cf97d46b9a621eb852e6aefe0" modify_external_catalogs: true
I20250901 01:52:24.412669 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 2ff914352c7c43de800f3d54ecf63072 (table default.Table_1186b4165684445da50e713ed6a227f9 [id=74d89178d7594613ba29d2bb01358e80]) successfully deleted
I20250901 01:52:24.432350 1491 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 34042ea5c1b645969c1de41fe587c343
W20250901 01:52:24.446445 1491 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 34042ea5c1b645969c1de41fe587c343: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:24.447124 1491 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_95515169b000454aa95025573560278f" } modify_external_catalogs: true
I20250901 01:52:24.447341 1491 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_95515169b000454aa95025573560278f" } modify_external_catalogs: true
I20250901 01:52:24.432664 1489 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 879fa66e55544118aafe504587393c66
W20250901 01:52:24.447918 1489 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 879fa66e55544118aafe504587393c66: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:24.448566 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5aac2869d8d24b7796ea7a5841db9e5f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.445969 2113 tablet_replica.cc:331] T 2ff914352c7c43de800f3d54ecf63072 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:24.449662 2113 raft_consensus.cc:2241] T 2ff914352c7c43de800f3d54ecf63072 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:24.449597 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_947807feab7d4283ab9a59b5da4f5717"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.449986 2113 raft_consensus.cc:2270] T 2ff914352c7c43de800f3d54ecf63072 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:24.450553 2113 ts_tablet_manager.cc:1905] T 2ff914352c7c43de800f3d54ecf63072 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:24.433466 1482 master_service.cc:948] ReplaceTablet: received request to replace tablet 88ceba7f48b04d7885cc0164668700a5 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:24.452981 1036 tablet_service.cc:1515] Processing DeleteTablet for tablet 879fa66e55544118aafe504587393c66 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:23 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:24.453003 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_bdd188d13bf644ea9fd56e2e852d30eb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.453212 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 34042ea5c1b645969c1de41fe587c343 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:23 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:24.454402 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e0451b2fd2f74120b611cd00ef804d05"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:24.454608 1484 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 615b2306fca84003b264da9d5f95ddab consensus state has no leader
I20250901 01:52:24.455065 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4c01fc0b5b394f3daa4698650f7d561b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.455242 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_873702880d5642d5920ebe57a03dd203"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.455994 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_38c25197c53048cd83b10a0081ef4277"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.456722 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_29f1bb43b43a4a64915a1cc8fbfbb1b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.457501 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b9b442eed9c544a9ac371c64a5e3c9e9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.462378 2113 ts_tablet_manager.cc:1918] T 2ff914352c7c43de800f3d54ecf63072 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:24.462589 2113 log.cc:1199] T 2ff914352c7c43de800f3d54ecf63072 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/2ff914352c7c43de800f3d54ecf63072
I20250901 01:52:24.463152 2113 ts_tablet_manager.cc:1939] T 2ff914352c7c43de800f3d54ecf63072 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:24.455963 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_cdac30ee356c47f7a0638a5b18162741"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.464588 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 2ff914352c7c43de800f3d54ecf63072 (table default.Table_1186b4165684445da50e713ed6a227f9 [id=74d89178d7594613ba29d2bb01358e80]) successfully deleted
I20250901 01:52:24.464810 2113 tablet_replica.cc:331] T 34042ea5c1b645969c1de41fe587c343 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
W20250901 01:52:24.464919 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 879fa66e55544118aafe504587393c66 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 879fa66e55544118aafe504587393c66 already in progress: creating tablet
I20250901 01:52:24.465050 2113 raft_consensus.cc:2241] T 34042ea5c1b645969c1de41fe587c343 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:24.465569 2113 raft_consensus.cc:2270] T 34042ea5c1b645969c1de41fe587c343 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:24.466167 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c15f4b3be0ea4d5e8f1a90d66f27f21f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.466706 2113 ts_tablet_manager.cc:1905] T 34042ea5c1b645969c1de41fe587c343 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:24.467324 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9bc5642581704e5888e1339ed18329f5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.469519 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_0bbd3cba6ed04caeb67c9617cd8abed2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.469895 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "33f637f6a8134215bb1995ec38440073" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:24.470403 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7a583513c2fd49c7b2f974c07c29a1a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.471278 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_dc4596f8d4ca437b9308ca9737fef4a8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.472091 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1f010c26c5a24b93b5db8acdb0ce726f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.473559 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_0ea8f3997d3a407aa9bfce17c6ba2da3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.474422 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_0ce851b5a6d340d5b0f03cc4576f6a3d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.439662 1151 raft_consensus.cc:357] T aa424ac6ef1441e6b1cc2867b1c48186 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.480604 1151 raft_consensus.cc:383] T aa424ac6ef1441e6b1cc2867b1c48186 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:24.480710 1151 raft_consensus.cc:738] T aa424ac6ef1441e6b1cc2867b1c48186 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:24.481014 1151 consensus_queue.cc:260] T aa424ac6ef1441e6b1cc2867b1c48186 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.481885 1151 ts_tablet_manager.cc:1428] T aa424ac6ef1441e6b1cc2867b1c48186 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.050s user 0.001s sys 0.001s
I20250901 01:52:24.482252 1151 tablet_bootstrap.cc:492] T 42c1156e8dd84c6fa1396b46de13eb6f P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:24.484237 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_40539b6cc9ae4f2fbd0b86e4c9556b46"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.453977 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 34042ea5c1b645969c1de41fe587c343 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:23 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:24.454113 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 879fa66e55544118aafe504587393c66 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:23 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:24.485390 2118 tablet_replica.cc:331] T 34042ea5c1b645969c1de41fe587c343 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:24.485700 2118 raft_consensus.cc:2241] T 34042ea5c1b645969c1de41fe587c343 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:24.485929 2118 raft_consensus.cc:2270] T 34042ea5c1b645969c1de41fe587c343 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:24.485924 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d8853618c919418db33649c76f179e16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.486773 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_0731ae9be4ad45f5a499def4b466b006"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.486891 2118 ts_tablet_manager.cc:1905] T 34042ea5c1b645969c1de41fe587c343 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:24.487273 2123 raft_consensus.cc:491] T 1495de70f0844f82a0eea99c059f076b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:24.487393 2123 raft_consensus.cc:513] T 1495de70f0844f82a0eea99c059f076b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.487632 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_03f2e4e01add4dcd8aff780bd19ce761"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.487949 2123 leader_election.cc:290] T 1495de70f0844f82a0eea99c059f076b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:24.489585 858 leader_election.cc:336] T 1495de70f0844f82a0eea99c059f076b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:24.441947 1492 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: default.Table_66ad08ce0b6840d4848bcddb304cf88b [id=d1cd5288671540c0a95f7a037363014e] alter complete (version 3)
I20250901 01:52:24.495194 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_61cc1c9b258344c39b03e6d504c1c83b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.496222 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a20c7cc971224fd59d22ac992ed711e6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.496924 2118 ts_tablet_manager.cc:1918] T 34042ea5c1b645969c1de41fe587c343 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 01:52:24.497046 2118 log.cc:1199] T 34042ea5c1b645969c1de41fe587c343 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/34042ea5c1b645969c1de41fe587c343
I20250901 01:52:24.497198 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_edebce991d4f42eba86fe27e935c2265"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.497448 2118 ts_tablet_manager.cc:1939] T 34042ea5c1b645969c1de41fe587c343 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:24.498078 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_fccc4c7fb2914d6a8c2b37884891eb40"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.498827 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 34042ea5c1b645969c1de41fe587c343 (table default.Table_23fd89096be149b791975d89d65bc280 [id=edbc04078c904f29a16a1bbb354dad92]) successfully deleted
W20250901 01:52:24.499168 1472 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 879fa66e55544118aafe504587393c66 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 879fa66e55544118aafe504587393c66 already in progress: creating tablet
I20250901 01:52:24.502110 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7d548387074f42ccae0ab566fd7fd9e3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.503058 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2416a55760fb4daf86cbdf84705f67ab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.503928 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e9ef305090a441bb9a40f1a9a264de17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.507344 2123 raft_consensus.cc:491] T 1a60c10e42104502ac2d3b80b8126ceb P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:24.507514 2123 raft_consensus.cc:513] T 1a60c10e42104502ac2d3b80b8126ceb P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.508339 2123 leader_election.cc:290] T 1a60c10e42104502ac2d3b80b8126ceb P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:24.510851 858 leader_election.cc:336] T 1a60c10e42104502ac2d3b80b8126ceb P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:24.511919 1483 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 246b33338de647689d7404c70676e76f
W20250901 01:52:24.512207 1483 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 246b33338de647689d7404c70676e76f: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:24.512650 1483 catalog_manager.cc:6652] ReplaceTablet: tablet 246b33338de647689d7404c70676e76f deleted and replaced by tablet ae4844183a3c4a04a775e6fd878391a3
I20250901 01:52:24.513053 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_410e50d2ea3c4f8b813940c072bf551c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.516124 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 246b33338de647689d7404c70676e76f with delete_type TABLET_DATA_DELETED (replaced by tablet ae4844183a3c4a04a775e6fd878391a3) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:24.516311 2123 raft_consensus.cc:491] T 05630a72ddae4de0b890dece2feacc31 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:24.516455 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 246b33338de647689d7404c70676e76f with delete_type TABLET_DATA_DELETED (replaced by tablet ae4844183a3c4a04a775e6fd878391a3) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:24.516438 2123 raft_consensus.cc:513] T 05630a72ddae4de0b890dece2feacc31 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:24.516989 2123 leader_election.cc:290] T 05630a72ddae4de0b890dece2feacc31 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:24.517354 1472 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 246b33338de647689d7404c70676e76f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 246b33338de647689d7404c70676e76f already in progress: creating tablet
W20250901 01:52:24.518713 858 leader_election.cc:336] T 05630a72ddae4de0b890dece2feacc31 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:24.522732 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a3f194e95d7f47e89e85a47cbf0b8fcd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.524294 2171 raft_consensus.cc:491] T f4fd117c395e41c2ab679c684908c5b7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:24.524494 2171 raft_consensus.cc:513] T f4fd117c395e41c2ab679c684908c5b7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.523185 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b55c960f468b41d0954189560c9b7450"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.525261 2171 leader_election.cc:290] T f4fd117c395e41c2ab679c684908c5b7 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:24.525835 1761 consensus_queue.cc:1035] T 9127f9c5239e445ba86012951b7c75d9 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:24.527833 1761 consensus_queue.cc:1035] T 72ba689f9230411995dc96e2b45601df P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:24.529079 992 leader_election.cc:304] T 25ce0fe8c9544f828694184d19d13df9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:24.529456 1993 raft_consensus.cc:2802] T 25ce0fe8c9544f828694184d19d13df9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:24.529539 1993 raft_consensus.cc:491] T 25ce0fe8c9544f828694184d19d13df9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:24.529618 1993 raft_consensus.cc:3058] T 25ce0fe8c9544f828694184d19d13df9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:24.533547 1993 raft_consensus.cc:513] T 25ce0fe8c9544f828694184d19d13df9 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:24.537070 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_05ad78811b46469e86db2da7544bf6c6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.538762 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d3ebf84b9948470ab5ef63b5d16f99a5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.540210 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_85e9477e4b4642c7b8cfa19abab64dc6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.541368 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ad46c0a0596e42a1b826d77e0f5b7931"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.542773 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_324efa2edf8943138be6c89e4dd9fb38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.543748 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_93b17adc80d04f0ab183732b17102549"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.545012 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_34687f58943945b683583e3e7818346e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.546290 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_506b0310361d41fa9ac1beea19b182d1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.547708 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_24ec0af5c49d43d6bd2fc45978e4fc6a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.549728 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9eac9084fa06429baf7ea60958f50b29"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.558712 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_047b1543d9e8404c9ffccbf4e1f6f353"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.560177 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_458d9474fd944cecafdc5a9932835237"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.561440 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2fcf7a0da57c4d4cb89a49e8acaf0392"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.563199 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8a34d07e08b44d4684bebda760b3d86b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.564330 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_43ab04faa2af41c5a0bbf1d99a99ca6b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.566098 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f5dd57b6911f4c46a2c9fda0567ec1e2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.567385 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d4bacf563dac49efb8f4d1f562b4cb2e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.568601 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_451754558c054307894f49e8ac6e06ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.569840 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_cd243d54228c4384b29a2e9032a3b459"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.571241 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9d73fecbc5444575acf55f6bb82908e4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.571962 1482 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 88ceba7f48b04d7885cc0164668700a5
W20250901 01:52:24.572187 1482 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 88ceba7f48b04d7885cc0164668700a5: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:24.572945 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 88ceba7f48b04d7885cc0164668700a5 with delete_type TABLET_DATA_DELETED (replaced by tablet 9074543dea9949e7a085717b40913348) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:24.573338 1482 catalog_manager.cc:6652] ReplaceTablet: tablet 88ceba7f48b04d7885cc0164668700a5 deleted and replaced by tablet 9074543dea9949e7a085717b40913348
I20250901 01:52:24.573624 1486 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_ce1fd26ac65a4ed085d023e10a849e5c" } modify_external_catalogs: true
I20250901 01:52:24.573999 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 88ceba7f48b04d7885cc0164668700a5 with delete_type TABLET_DATA_DELETED (replaced by tablet 9074543dea9949e7a085717b40913348) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:24.573840 1486 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_ce1fd26ac65a4ed085d023e10a849e5c" } modify_external_catalogs: true
I20250901 01:52:24.575778 1491 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 438ae684af234c868a419fbab43b0b98
W20250901 01:52:24.576043 1491 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 438ae684af234c868a419fbab43b0b98: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
W20250901 01:52:24.578413 1472 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 88ceba7f48b04d7885cc0164668700a5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 88ceba7f48b04d7885cc0164668700a5 already in progress: creating tablet
I20250901 01:52:24.578523 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 438ae684af234c868a419fbab43b0b98 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:24 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:24.559556 992 leader_election.cc:304] T 3c12d6a70ac24dce82e851782e6fe5aa P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:24.578977 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 438ae684af234c868a419fbab43b0b98 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:24 UTC) from {username='slave'} at 127.0.0.1:35946
W20250901 01:52:24.579342 1472 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 438ae684af234c868a419fbab43b0b98 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 438ae684af234c868a419fbab43b0b98 already in progress: creating tablet
I20250901 01:52:24.579371 2170 raft_consensus.cc:2802] T 3c12d6a70ac24dce82e851782e6fe5aa P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:24.579532 2170 raft_consensus.cc:695] T 3c12d6a70ac24dce82e851782e6fe5aa P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:24.579856 2170 consensus_queue.cc:237] T 3c12d6a70ac24dce82e851782e6fe5aa P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.592630 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_dbf00a7af1a74c7fbaf2b2797cfeb2d0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.594254 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9f6c9973977149f5b40a45af25e045c2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.595767 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b9412cc46c4f43d88ddfabc064990fb6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.597203 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_954329934fb34365b24e850b488b8f67"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.598361 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c5ad725c6be44c15aeaf329d1e5224eb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.599958 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4a921a08b7884eca901f19c90735facf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.609408 1486 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 617cc33d0d23439cb5f1e0c31bf11281
W20250901 01:52:24.609638 1486 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 617cc33d0d23439cb5f1e0c31bf11281: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:24.610536 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 617cc33d0d23439cb5f1e0c31bf11281 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:24 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:24.612519 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8823b5244c214553b7503c20081f2ac3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:24.613885 1696 catalog_manager.cc:6205] Error selecting replicas for tablet be8dfbc466564842a2fdb4953eee939b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_75ba3ba70ca24a94920e7bce96190ff2'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:24.614145 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 7317eb7848ee44698038a4b404d4bf21, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b0e916c76a7a4766aa0887cc8691488f'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:24.614553 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3c23f2a21a154e8d8287bae51755355b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.616272 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 617cc33d0d23439cb5f1e0c31bf11281 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:24 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:24.615553 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8b73fb2208dd46e5ab71cd5147941805"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:24.617358 1472 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 617cc33d0d23439cb5f1e0c31bf11281 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 617cc33d0d23439cb5f1e0c31bf11281 already in progress: creating tablet
W20250901 01:52:24.617507 1474 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet 246b33338de647689d7404c70676e76f on TS 0cf599f74322478d91475df234e58965 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
E20250901 01:52:24.618026 1696 catalog_manager.cc:6205] Error selecting replicas for tablet e2ed32b3292141278af28026528e96c2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_50ae3908a7944f70949c54be8fbf1564'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:24.618216 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 15a4f99ae43745b9a3f60052d916a7bd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_23fb8cbcf1bd43fa8346f3c26c8b4c65'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:24.618366 1696 catalog_manager.cc:6205] Error selecting replicas for tablet a2ca69fabadd4ad69e1670cae7a0c503, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_97a5ec9ec34c4a89b0f917704b43e52b'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:24.618564 1696 catalog_manager.cc:6205] Error selecting replicas for tablet cc3b8736b9094d7e892c80da4d5b20f6, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_93afd37289414655980b22934695bd55'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:24.618992 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 18498c9cb09141d08c1c5c0854b63ebd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77f4810bc83c4d948531da2a45a3792e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:24.619202 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 6ce9521b925b4f97bb4a810ce77e147e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_9e04f976a93f4d30baccf90fa3622dfe'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:24.621802 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_da48fbf15d5e423aa2194a313a034bec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.622895 1482 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_219f0092f43743039ae1a61fedb3e4d1" } new_table_name: "default.Table_aee74198b51d4f089538c4f2bf3ce4dc" modify_external_catalogs: true
W20250901 01:52:24.622996 992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:40561, user_credentials={real_user=slave}} blocked reactor thread for 44126.8us
I20250901 01:52:24.623519 992 leader_election.cc:304] T e9b179dba2cc49428ba8090c962d6136 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:24.623914 1761 consensus_queue.cc:1035] T fae2779fac164b9f8052c3666db803b8 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:24.624176 2216 raft_consensus.cc:2802] T e9b179dba2cc49428ba8090c962d6136 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:24.624282 2216 raft_consensus.cc:695] T e9b179dba2cc49428ba8090c962d6136 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:24.624616 2216 consensus_queue.cc:237] T e9b179dba2cc49428ba8090c962d6136 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.625164 1993 leader_election.cc:290] T 25ce0fe8c9544f828694184d19d13df9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:24.627058 1491 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_ce31a83b98ea4b698acbbdc1379419f5" } modify_external_catalogs: true
I20250901 01:52:24.627377 1491 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_ce31a83b98ea4b698acbbdc1379419f5" } modify_external_catalogs: true
I20250901 01:52:24.637439 1491 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 756cfe84a517459ca015aecf5cd81571
W20250901 01:52:24.637699 1491 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 756cfe84a517459ca015aecf5cd81571: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
W20250901 01:52:24.638166 1482 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ae4844183a3c4a04a775e6fd878391a3 has no consensus state
I20250901 01:52:24.638363 2219 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:43871 every 8 connection(s)
I20250901 01:52:24.638567 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 756cfe84a517459ca015aecf5cd81571 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:24 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:24.638732 1922 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:43871
I20250901 01:52:24.638888 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 756cfe84a517459ca015aecf5cd81571 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:24 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:24.639158 2118 tablet_replica.cc:331] T 756cfe84a517459ca015aecf5cd81571 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:24.639381 2118 raft_consensus.cc:2241] T 756cfe84a517459ca015aecf5cd81571 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:24.639591 2118 raft_consensus.cc:2270] T 756cfe84a517459ca015aecf5cd81571 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:24.640175 2118 ts_tablet_manager.cc:1905] T 756cfe84a517459ca015aecf5cd81571 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:24.641459 1922 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/master-0/data/info.pb
I20250901 01:52:24.643174 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3070634ff49e422fbc79b35e9f029fd9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.645134 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_0ebadb968b494cae8a3a4d5fa71cf2d3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.646309 2118 ts_tablet_manager.cc:1918] T 756cfe84a517459ca015aecf5cd81571 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:24.646418 2118 log.cc:1199] T 756cfe84a517459ca015aecf5cd81571 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/756cfe84a517459ca015aecf5cd81571
I20250901 01:52:24.646500 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9cd7fad28c3447d6ad4ca3c30d58e491"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.646736 2118 ts_tablet_manager.cc:1939] T 756cfe84a517459ca015aecf5cd81571 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:24.647758 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 756cfe84a517459ca015aecf5cd81571 (table default.Table_ce31a83b98ea4b698acbbdc1379419f5 [id=f49c96ff2eaa44cc9b754225cb9e6fdf]) successfully deleted
I20250901 01:52:24.648653 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskddMNdm/build/asan/bin/kudu as pid 1922
I20250901 01:52:24.648650 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b386048cbf504489bf938014a4b881af"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.649668 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_df18e5f3b775438087a1b895f7c5a1d5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.650771 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e199fd7ba4c3474b82defee16d4e2c8c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.652956 1486 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_97a5ec9ec34c4a89b0f917704b43e52b" } new_table_name: "default.Table_32c4716e3aa24a87a6b1472984b255c4" modify_external_catalogs: true
I20250901 01:52:24.654291 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4b645f860db24989bebaae1d5e571a73"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.655779 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2e785ab8f55d421e9153520795b913ef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.656842 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d839e92aa5af4746a00492ba01debf33"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.657446 2216 raft_consensus.cc:491] T 60285f2ba037433999f5455fb9cc7acd P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:24.657636 2216 raft_consensus.cc:513] T 60285f2ba037433999f5455fb9cc7acd P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:24.661480 2216 leader_election.cc:290] T 60285f2ba037433999f5455fb9cc7acd P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:24.666410 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_cf965c9af85c45359d71e26bf70eb21f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.668650 1491 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_2f266fe78a3c41c2ab134faad48a377d" } new_table_name: "default.Table_33c84edd575e4969b264a684accb22e2" modify_external_catalogs: true
I20250901 01:52:24.669574 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8cf3a98c6151468a98651814b6d18d08"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.671094 1482 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_df96f1aa51644aed967bdfd4f75588f6" } new_table_name: "default.Table_345b7f9fe8214aa6878f20a166125364" modify_external_catalogs: true
I20250901 01:52:24.675069 1151 tablet_bootstrap.cc:654] T 42c1156e8dd84c6fa1396b46de13eb6f P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:24.676582 1485 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_e113aaaa38db48fdb16d32569b81b930" } modify_external_catalogs: true
W20250901 01:52:24.677073 992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:40561, user_credentials={real_user=slave}} blocked reactor thread for 53585.9us
I20250901 01:52:24.679502 1484 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_54422fcb2b624e7fa8d42cba3558a327" } modify_external_catalogs: true
I20250901 01:52:24.680809 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_dfa56468ae9044f7a6f17276469ba47c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.683904 1489 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_5ea919f20c6b4a4e91e599357a17a53a" } new_table_name: "default.Table_f44c609420054cebad09692be9aa6ecd" modify_external_catalogs: true
I20250901 01:52:24.684794 2216 raft_consensus.cc:491] T 12c04e0c1f804179ae23509cdb24d395 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:24.684928 2216 raft_consensus.cc:513] T 12c04e0c1f804179ae23509cdb24d395 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:24.685485 2216 leader_election.cc:290] T 12c04e0c1f804179ae23509cdb24d395 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:24.686977 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_20a7994b20ba4040b847b4887a41f8e4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.688596 1103 heartbeater.cc:499] Master 127.0.104.60:44231 was elected leader, sending a full tablet report...
I20250901 01:52:24.689589 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_56cc645d5f5643048b5b5b10c2165e5a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.696972 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_26c09910fc5e4ac898eeb80747bbdf63"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.701826 2123 raft_consensus.cc:491] T 56475ba491a24301a534304811f30140 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:24.702065 2123 raft_consensus.cc:513] T 56475ba491a24301a534304811f30140 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:24.702890 2123 leader_election.cc:290] T 56475ba491a24301a534304811f30140 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:24.704996 1485 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_e113aaaa38db48fdb16d32569b81b930" } modify_external_catalogs: true
W20250901 01:52:24.705310 858 leader_election.cc:336] T 56475ba491a24301a534304811f30140 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:24.705430 859 leader_election.cc:343] T 33f637f6a8134215bb1995ec38440073 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:24.705521 859 leader_election.cc:304] T 33f637f6a8134215bb1995ec38440073 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:24.706068 2123 raft_consensus.cc:2747] T 33f637f6a8134215bb1995ec38440073 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:24.708077 1151 tablet_bootstrap.cc:492] T 42c1156e8dd84c6fa1396b46de13eb6f P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:24.708232 1151 ts_tablet_manager.cc:1397] T 42c1156e8dd84c6fa1396b46de13eb6f P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.226s user 0.004s sys 0.000s
I20250901 01:52:24.708685 1484 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_54422fcb2b624e7fa8d42cba3558a327" } modify_external_catalogs: true
I20250901 01:52:24.709043 1151 raft_consensus.cc:357] T 42c1156e8dd84c6fa1396b46de13eb6f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:24.709196 1151 raft_consensus.cc:383] T 42c1156e8dd84c6fa1396b46de13eb6f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:24.709249 1151 raft_consensus.cc:738] T 42c1156e8dd84c6fa1396b46de13eb6f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:24.712180 969 heartbeater.cc:499] Master 127.0.104.60:44231 was elected leader, sending a full tablet report...
I20250901 01:52:24.717797 1484 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet dac01ff54b97443289d4f3d10f80cd41
W20250901 01:52:24.718001 1484 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet dac01ff54b97443289d4f3d10f80cd41: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:24.718693 1485 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet c754129977904f978f87fd89d7092e36
I20250901 01:52:24.718750 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet dac01ff54b97443289d4f3d10f80cd41 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:24 UTC) from {username='slave'} at 127.0.0.1:35946
W20250901 01:52:24.718906 1485 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet c754129977904f978f87fd89d7092e36: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:24.719462 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet c754129977904f978f87fd89d7092e36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:24 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:24.719599 894 tablet_service.cc:1515] Processing DeleteTablet for tablet dac01ff54b97443289d4f3d10f80cd41 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:24 UTC) from {username='slave'} at 127.0.0.1:34472
W20250901 01:52:24.720175 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet dac01ff54b97443289d4f3d10f80cd41 because the tablet was not found. No further retry: Not found: Tablet not found: dac01ff54b97443289d4f3d10f80cd41
I20250901 01:52:24.720300 894 tablet_service.cc:1515] Processing DeleteTablet for tablet c754129977904f978f87fd89d7092e36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:24 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:24.720520 2118 tablet_replica.cc:331] T c754129977904f978f87fd89d7092e36 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:24.720746 2118 raft_consensus.cc:2241] T c754129977904f978f87fd89d7092e36 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:24.720907 2118 raft_consensus.cc:2270] T c754129977904f978f87fd89d7092e36 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:24.721454 2118 ts_tablet_manager.cc:1905] T c754129977904f978f87fd89d7092e36 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:24.721582 1489 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7954393c8c1c4457ab8bb723c03a9c8b consensus state has no leader
I20250901 01:52:24.723985 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_fffc1ff0fd324e25816488b40918139c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.724642 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4d9a382db6324a6a8a8c58eaa9935c77"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.724819 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9300bfa6f0f649ef840752e6e2f5decc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.724344 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_181d913923f54e80b966bbcab839530a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.730243 2118 ts_tablet_manager.cc:1918] T c754129977904f978f87fd89d7092e36 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:24.732501 2118 log.cc:1199] T c754129977904f978f87fd89d7092e36 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/c754129977904f978f87fd89d7092e36
I20250901 01:52:24.732915 2118 ts_tablet_manager.cc:1939] T c754129977904f978f87fd89d7092e36 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:24.744565 2216 raft_consensus.cc:491] T aa424ac6ef1441e6b1cc2867b1c48186 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:24.744756 2216 raft_consensus.cc:513] T aa424ac6ef1441e6b1cc2867b1c48186 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.745503 2216 leader_election.cc:290] T aa424ac6ef1441e6b1cc2867b1c48186 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:24.745879 2216 raft_consensus.cc:491] T 4d27fb61943c45d4aca67689cc3a9ff3 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:24.745993 2216 raft_consensus.cc:513] T 4d27fb61943c45d4aca67689cc3a9ff3 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:24.746480 2216 leader_election.cc:290] T 4d27fb61943c45d4aca67689cc3a9ff3 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:24.753535 1482 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 508406601c994e42a5c5b1f665e74957 consensus state has no leader
W20250901 01:52:24.754837 1486 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a2ca69fabadd4ad69e1670cae7a0c503 has no consensus state
W20250901 01:52:24.761394 1491 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 06d5eeb70e664f73bf643922d445bc7a consensus state has no leader
I20250901 01:52:24.769610 917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f4fd117c395e41c2ab679c684908c5b7" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:24.769861 917 raft_consensus.cc:2466] T f4fd117c395e41c2ab679c684908c5b7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:24.771359 1981 raft_consensus.cc:491] T 14b466b2a5a249aa841f1b8b06301edc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:24.771481 1981 raft_consensus.cc:513] T 14b466b2a5a249aa841f1b8b06301edc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.772110 1981 leader_election.cc:290] T 14b466b2a5a249aa841f1b8b06301edc P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:24.773265 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "25ce0fe8c9544f828694184d19d13df9" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:24.773427 915 raft_consensus.cc:3058] T 25ce0fe8c9544f828694184d19d13df9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:24.780109 915 raft_consensus.cc:2466] T 25ce0fe8c9544f828694184d19d13df9 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
W20250901 01:52:24.798043 1113 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.CreateTable -> {remote=127.0.104.60:44231, user_credentials={real_user=slave}} blocked reactor thread for 93631.5us
I20250901 01:52:24.806658 1150 tablet_bootstrap.cc:492] T 48a211e174054866b030cdfb0cf947f4 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:24.806874 1150 ts_tablet_manager.cc:1397] T 48a211e174054866b030cdfb0cf947f4 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.386s user 0.062s sys 0.133s
I20250901 01:52:24.809185 2113 ts_tablet_manager.cc:1918] T 34042ea5c1b645969c1de41fe587c343 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 01:52:24.810220 1150 raft_consensus.cc:357] T 48a211e174054866b030cdfb0cf947f4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.815886 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_dec0f2389d9b4b268dee95ed845d567b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.816530 2113 log.cc:1199] T 34042ea5c1b645969c1de41fe587c343 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/34042ea5c1b645969c1de41fe587c343
I20250901 01:52:24.817023 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_37b8a58a761e478999244a9f599e3825"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.817399 1150 raft_consensus.cc:383] T 48a211e174054866b030cdfb0cf947f4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:24.817955 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_15cd3f95a0b74acdbd981c1a4cf0b954"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.820297 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_51af5744df584c7aa6fbbbea53039204"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.818312 1150 raft_consensus.cc:738] T 48a211e174054866b030cdfb0cf947f4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:24.821357 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_14e44b43fb494238b8eb08a3488bd4a0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.821565 1150 consensus_queue.cc:260] T 48a211e174054866b030cdfb0cf947f4 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.822120 1150 ts_tablet_manager.cc:1428] T 48a211e174054866b030cdfb0cf947f4 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
I20250901 01:52:24.818120 2113 ts_tablet_manager.cc:1939] T 34042ea5c1b645969c1de41fe587c343 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:24.822584 1486 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_d2049d0c20bb4949ae6c249b690e7f86" } modify_external_catalogs: true
I20250901 01:52:24.822829 1486 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_d2049d0c20bb4949ae6c249b690e7f86" } modify_external_catalogs: true
I20250901 01:52:24.823210 1054 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1495de70f0844f82a0eea99c059f076b" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:24.823438 1058 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05630a72ddae4de0b890dece2feacc31" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:24.823580 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "56475ba491a24301a534304811f30140" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:24.823680 1054 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14b466b2a5a249aa841f1b8b06301edc" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:24.823248 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1a60c10e42104502ac2d3b80b8126ceb" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:24.825079 2113 tablet_replica.cc:331] T 756cfe84a517459ca015aecf5cd81571 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:24.825098 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 34042ea5c1b645969c1de41fe587c343 (table default.Table_23fd89096be149b791975d89d65bc280 [id=edbc04078c904f29a16a1bbb354dad92]) successfully deleted
W20250901 01:52:24.825363 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 246b33338de647689d7404c70676e76f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 246b33338de647689d7404c70676e76f already in progress: creating tablet
W20250901 01:52:24.825556 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 88ceba7f48b04d7885cc0164668700a5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 88ceba7f48b04d7885cc0164668700a5 already in progress: creating tablet
W20250901 01:52:24.825711 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 438ae684af234c868a419fbab43b0b98 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 438ae684af234c868a419fbab43b0b98 already in progress: creating tablet
I20250901 01:52:24.825461 2216 raft_consensus.cc:491] T 616ff34e8d314e0aa0e11ce60be11789 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:24.825845 2216 raft_consensus.cc:513] T 616ff34e8d314e0aa0e11ce60be11789 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.825932 2113 raft_consensus.cc:2241] T 756cfe84a517459ca015aecf5cd81571 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:24.826138 2113 pending_rounds.cc:70] T 756cfe84a517459ca015aecf5cd81571 P fd27ab6665304aaea53adbd102dd4f58: Trying to abort 1 pending ops.
W20250901 01:52:24.826244 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 617cc33d0d23439cb5f1e0c31bf11281 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 617cc33d0d23439cb5f1e0c31bf11281 already in progress: creating tablet
I20250901 01:52:24.826417 2216 leader_election.cc:290] T 616ff34e8d314e0aa0e11ce60be11789 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:24.829903 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f9b1b9164bb34267bff5d6f3979e2916"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.831125 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f754195f05ea4368a268093796c295f6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.827750 2113 pending_rounds.cc:77] T 756cfe84a517459ca015aecf5cd81571 P fd27ab6665304aaea53adbd102dd4f58: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7195408566005084160 op_type: ALTER_SCHEMA_OP alter_schema_request { tablet_id: "756cfe84a517459ca015aecf5cd81571" schema { columns { id: 10 name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } schema_version: 1 new_table_name: "default.Table_ce31a83b98ea4b698acbbdc1379419f5" dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" new_extra_config { } }
I20250901 01:52:24.833552 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e4ad77f905694635aa80058088dac1ee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.835179 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_69fe81a2c3694ccfad6c5b19bcdef2a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:24.841643 2113 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35946: Aborted: Op aborted
I20250901 01:52:24.842084 2113 raft_consensus.cc:2270] T 756cfe84a517459ca015aecf5cd81571 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:24.833750 1151 consensus_queue.cc:260] T 42c1156e8dd84c6fa1396b46de13eb6f P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:24.843101 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_86289af8ceff46d185fd966046eb99a4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.843142 1151 ts_tablet_manager.cc:1428] T 42c1156e8dd84c6fa1396b46de13eb6f P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.135s user 0.000s sys 0.002s
I20250901 01:52:24.843551 1151 tablet_bootstrap.cc:492] T d0c3c363dda444519a17dca63ad9caa7 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:24.845438 1151 tablet_bootstrap.cc:654] T d0c3c363dda444519a17dca63ad9caa7 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:24.834268 2170 raft_consensus.cc:491] T 64fb46b942ff4e4c8749986ecc3dfe65 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:24.846457 2170 raft_consensus.cc:513] T 64fb46b942ff4e4c8749986ecc3dfe65 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.847258 2170 leader_election.cc:290] T 64fb46b942ff4e4c8749986ecc3dfe65 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:24.851017 1150 tablet_bootstrap.cc:492] T 4cde5f4f02044b7bbd7d43b49aac4ae8 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:24.854527 1150 tablet_bootstrap.cc:654] T 4cde5f4f02044b7bbd7d43b49aac4ae8 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:24.856519 1472 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.62:43871, user_credentials={real_user=slave}} blocked reactor thread for 114480us
I20250901 01:52:24.856997 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet c754129977904f978f87fd89d7092e36 (table default.Table_e113aaaa38db48fdb16d32569b81b930 [id=f09f20dc4db844a8abc825e9086093ba]) successfully deleted
I20250901 01:52:24.861838 1872 raft_consensus.cc:491] T dd694cfa09564d3da5d279c0861a4c32 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:24.862138 1872 raft_consensus.cc:513] T dd694cfa09564d3da5d279c0861a4c32 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:24.862861 1872 leader_election.cc:290] T dd694cfa09564d3da5d279c0861a4c32 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:24.863261 1482 master_service.cc:948] ReplaceTablet: received request to replace tablet 5e2472ee5d6e4d3bbf1f24027ba531ca from {username='slave'} at 127.0.0.1:33478
W20250901 01:52:24.863590 992 leader_election.cc:336] T e9b179dba2cc49428ba8090c962d6136 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:24.863880 992 leader_election.cc:336] T 3c12d6a70ac24dce82e851782e6fe5aa P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:24.864220 992 leader_election.cc:336] T 25ce0fe8c9544f828694184d19d13df9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:24.868430 1150 tablet_bootstrap.cc:492] T 4cde5f4f02044b7bbd7d43b49aac4ae8 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:24.868526 1491 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_23fb8cbcf1bd43fa8346f3c26c8b4c65" } modify_external_catalogs: true
I20250901 01:52:24.868696 1150 ts_tablet_manager.cc:1397] T 4cde5f4f02044b7bbd7d43b49aac4ae8 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.003s
I20250901 01:52:24.868878 1491 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_23fb8cbcf1bd43fa8346f3c26c8b4c65" } modify_external_catalogs: true
I20250901 01:52:24.869647 1485 master_service.cc:948] ReplaceTablet: received request to replace tablet 121803e79ab64ec0b948b448646b9d0a from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:24.870072 1150 raft_consensus.cc:357] T 4cde5f4f02044b7bbd7d43b49aac4ae8 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.870309 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4ebbcd730861402b8cacca215e713376"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.870409 1150 raft_consensus.cc:383] T 4cde5f4f02044b7bbd7d43b49aac4ae8 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:24.870558 1150 raft_consensus.cc:738] T 4cde5f4f02044b7bbd7d43b49aac4ae8 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:24.870927 1150 consensus_queue.cc:260] T 4cde5f4f02044b7bbd7d43b49aac4ae8 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.871790 1150 ts_tablet_manager.cc:1428] T 4cde5f4f02044b7bbd7d43b49aac4ae8 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250901 01:52:24.871995 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_529f3b64d3844d4b95224f73c5f2ca01"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.872265 1150 tablet_bootstrap.cc:492] T a9b7f2e6548043a7865e25387cdd06c9 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:24.873234 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f5bfef324df24b5ebc4f7bed911aefc8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.874292 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_bd5e97354f934672bf495b4c6da8cbe4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.875272 1150 tablet_bootstrap.cc:654] T a9b7f2e6548043a7865e25387cdd06c9 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:24.875317 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d0e0cd8d40fd4068825ace0bdec4a33c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.879930 2231 raft_consensus.cc:491] T ad1a6be9fcc045d3bb4e71a2f6bd5ccd P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:24.880052 2231 raft_consensus.cc:513] T ad1a6be9fcc045d3bb4e71a2f6bd5ccd P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.880707 2231 leader_election.cc:290] T ad1a6be9fcc045d3bb4e71a2f6bd5ccd P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:24.880827 1489 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_8d0db1143f1547c2a7d41775b4d9ef09" } new_table_name: "default.Table_4158d47f631042489d14100215dce98a" modify_external_catalogs: true
I20250901 01:52:24.883639 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_bbf81dad354f4409a4e5ca6d20013304"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.884658 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_91202ae3a0e64c219459987c6c0af1e1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.885919 1492 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: default.Table_420b0979d6fd42da9b320456f4ccc9ce [id=d2004563c4374790954327f05c305cc6] alter complete (version 1)
W20250901 01:52:24.886636 1492 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 1495de70f0844f82a0eea99c059f076b (table default.Table_02e911ce389e4ad581c0a2f976f99c4e [id=70a5ce71d16b47d4bc6dae97be4548e7], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 1495de70f0844f82a0eea99c059f076b consensus state has no leader
I20250901 01:52:24.887192 1482 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 5e2472ee5d6e4d3bbf1f24027ba531ca
I20250901 01:52:24.887266 2113 ts_tablet_manager.cc:1905] T 756cfe84a517459ca015aecf5cd81571 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:24.887302 1492 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet ed1b41340ce742a88c3e4fd85cd25ca6 (table default.Table_139f6d174d794a6e9a618e90587d0c0d [id=0baa4993b72e4a8d94ee3059c9d2db6c], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet ed1b41340ce742a88c3e4fd85cd25ca6 consensus state has no leader
W20250901 01:52:24.887353 1482 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 5e2472ee5d6e4d3bbf1f24027ba531ca: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
W20250901 01:52:24.887455 1492 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet d10c193452954c198bba686169b8025b (table default.Table_03fb48eaab9b42858ab7cefb722e43a8 [id=6c0576410d7340d6aa493a747e95565b], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet d10c193452954c198bba686169b8025b consensus state has no leader
W20250901 01:52:24.887604 1492 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 435b1cca5a7d4d49959a310cb250b2a0 (table default.Table_dc9c5d18c7ec478592a76dbd76424b88 [id=126b298fef6c4653952d672adff1db7e], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 435b1cca5a7d4d49959a310cb250b2a0 consensus state has no leader
I20250901 01:52:24.887681 1482 catalog_manager.cc:6652] ReplaceTablet: tablet 5e2472ee5d6e4d3bbf1f24027ba531ca deleted and replaced by tablet 0813cc68d4b0488ba60a523e4479971b
W20250901 01:52:24.887856 1492 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 410d44e5f1974ed69e5dee65e98856a4 (table default.Table_bec114a616504798bede9a9066e57192 [id=ae79f2c40fd7483ea400a36cf3c48888], current schema version=4): Not found: failed to reset TS proxy: No leader found: Tablet 410d44e5f1974ed69e5dee65e98856a4 consensus state has no leader
I20250901 01:52:24.890074 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 31614db4c53a4f3c96fd190d326ce601 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:24.890339 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 533d14f5903e49599a34ea353ffd8c00 with delete_type TABLET_DATA_DELETED (replaced by tablet a54af99bc10045e8b0b6868e5aed98b4) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:24.890504 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet 56475ba491a24301a534304811f30140 with delete_type TABLET_DATA_DELETED (replaced by tablet 7317eb7848ee44698038a4b404d4bf21) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:24.890704 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 5e2472ee5d6e4d3bbf1f24027ba531ca with delete_type TABLET_DATA_DELETED (replaced by tablet 0813cc68d4b0488ba60a523e4479971b) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:24.890748 1486 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 3c12d6a70ac24dce82e851782e6fe5aa
I20250901 01:52:24.890905 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 75e7a90815e449c39f386901a504143c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:18 UTC) from {username='slave'} at 127.0.0.1:35946
W20250901 01:52:24.890940 1486 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 3c12d6a70ac24dce82e851782e6fe5aa: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:24.891024 1036 tablet_service.cc:1515] Processing DeleteTablet for tablet 4902062977ea42149e3466e999117322 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:18 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:24.891099 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 7a607fac57994001b2304e5d81c9decc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:17 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:24.891220 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 48a211e174054866b030cdfb0cf947f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:18 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:24.891316 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet afc002ae2486476b9208e1cf8215157c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:17 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:24.891398 1034 tablet_service.cc:1515] Processing DeleteTablet for tablet b81a12b38ac64efabd44aeb192d181a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:18 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:24.891568 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 1a60c10e42104502ac2d3b80b8126ceb with delete_type TABLET_DATA_DELETED (replaced by tablet 88ceba7f48b04d7885cc0164668700a5) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:24.891628 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 4d27fb61943c45d4aca67689cc3a9ff3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:13 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:24.892205 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 3c12d6a70ac24dce82e851782e6fe5aa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:24 UTC) from {username='slave'} at 127.0.0.1:35946
W20250901 01:52:24.893393 1492 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet e87d1281f35648b0b115b67c95e244e7 (table default.Table_f840789a58984799bd8057128f6ea439 [id=4453959e273c473fadd737f56f7db3dd], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet e87d1281f35648b0b115b67c95e244e7 consensus state has no leader
I20250901 01:52:24.894719 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet e9b179dba2cc49428ba8090c962d6136 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:17 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:24.895705 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 05630a72ddae4de0b890dece2feacc31 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:18 UTC) from {username='slave'} at 127.0.0.1:35946
W20250901 01:52:24.896005 1492 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 6c0abbfc8d904dce989ce51961e4237d (table default.Table_3f9422b08b0349b9a5466218f0fe1b06 [id=f0dfb3811e744bd4ba6d59195cebde6f], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet 6c0abbfc8d904dce989ce51961e4237d consensus state has no leader
W20250901 01:52:24.896270 1492 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet c6f7c525f59b49938b697de98d4b5835 (table default.Table_db61e055dbd7450898e82c4c0c4763be [id=aabca67f6f754820977ac9c4c20f3e4f], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet c6f7c525f59b49938b697de98d4b5835 consensus state has no leader
I20250901 01:52:24.896852 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet c58edaa76ebf47dfb1d9d5fab6de7dfa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:17 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:24.897459 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet e249474dcbf642189560c91ec3a99fdf with delete_type TABLET_DATA_DELETED (replaced by tablet 7f46e9ddadaf4ff986d8c991178daa0c) from {username='slave'} at 127.0.0.1:35946
W20250901 01:52:24.897812 1492 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 081e2630e284498aa819f85bf8326315 (table default.Table_9e7ca61fcc324a8984bd1e470162c61f [id=8199118617cb4707bc3ae9b1363750f7], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet 081e2630e284498aa819f85bf8326315 consensus state has no leader
W20250901 01:52:24.898073 1492 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 2599f98abfdc4871b573edfc2ad97d22 (table default.Table_0e63d21acfcb4c028c9ea6406b6a04e4 [id=81323e6fbe424655b8e185f52657fadc], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 2599f98abfdc4871b573edfc2ad97d22 consensus state has no leader
I20250901 01:52:24.898638 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet eaa521c52a2f4408b59770bd2a05907c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:24.899497 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 17e8340dbd1a406cb782dad3fe88cc7e with delete_type TABLET_DATA_DELETED (replaced by tablet e2ed32b3292141278af28026528e96c2) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:24.900252 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet e25923b2daf9429683b2d44a75a56189 with delete_type TABLET_DATA_DELETED (replaced by tablet 2973b59bb1df41d18d9a5b6a29ae336f) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:24.901211 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 6741c56499b44f648fb032c8cf50205b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:35946
W20250901 01:52:24.901620 1492 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 81ce448b148447ddbc41875d863e0518 (table default.Table_6e461d9cb0ae4dd095975bbc21533794 [id=49673fa73f8c436e9b8026efc4f3cbbc], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 81ce448b148447ddbc41875d863e0518 consensus state has no leader
I20250901 01:52:24.902381 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 46bfbdff3f904715a8c24907b20fcab0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:24.903059 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 0dbdcb76eeba4fefa56e7835c23c0316 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:17 UTC) from {username='slave'} at 127.0.0.1:35946
W20250901 01:52:24.903486 1492 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 42c1156e8dd84c6fa1396b46de13eb6f (table default.Table_b5d681f51fc84395bbabe1e5b4cfb358 [id=40d05d90b0354334840ad17764f5b6bb], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 42c1156e8dd84c6fa1396b46de13eb6f consensus state has no leader
I20250901 01:52:24.904162 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet ded33e0950b544509af11739353c620d with delete_type TABLET_DATA_DELETED (replaced by tablet cc3b8736b9094d7e892c80da4d5b20f6) from {username='slave'} at 127.0.0.1:35946
W20250901 01:52:24.904462 1492 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 616ff34e8d314e0aa0e11ce60be11789 (table default.Table_0451d0afa26f4d5f8d4214a946eea04f [id=5ee13d46e06a4fa59b1070547573e523], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 616ff34e8d314e0aa0e11ce60be11789 consensus state has no leader
W20250901 01:52:24.904772 1492 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 3c12d6a70ac24dce82e851782e6fe5aa (table default.Table_d2049d0c20bb4949ae6c249b690e7f86 [id=26bba0e60b6b43eb87d084be403c556b], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 3c12d6a70ac24dce82e851782e6fe5aa consensus state has no leader
W20250901 01:52:24.905030 1492 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet e1d18e999b5e42b1873631e0f1445081 (table default.Table_fd17a2ad225546aa821489ce94a788a2 [id=033c76e0598342c495ce2be42a0df5dd], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet e1d18e999b5e42b1873631e0f1445081 consensus state has no leader
I20250901 01:52:24.909654 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet eac50cbae691416e903cb2cbc907f96d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:23 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:24.910394 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 7f64ebd080944190be2e64d02d70a8a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:15 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:24.911381 2170 raft_consensus.cc:491] T 20f7bfab18134e079e91c52c89468cc3 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:24.911620 2170 raft_consensus.cc:513] T 20f7bfab18134e079e91c52c89468cc3 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.912238 2170 leader_election.cc:290] T 20f7bfab18134e079e91c52c89468cc3 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:24.913206 1492 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 49aaaec611784e6bac131efd58dbca51 (table default.Table_95e1338a49a04f22b679e363e253262e [id=e24b91b4ac904ef0a8c4725f70bced2f], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 49aaaec611784e6bac131efd58dbca51 consensus state has no leader
I20250901 01:52:24.912986 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 25ce0fe8c9544f828694184d19d13df9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:16 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:24.914215 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet af2c4bcbb00741f789edee109252d8cb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:18 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:24.914844 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 6cf09e8f299548dba2cce16d7b565bad with delete_type TABLET_DATA_DELETED (replaced by tablet 0e4009963cfe42d6ba164fc828897f09) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:24.916316 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 2240302d9178445d8700e35f5ea274f7 with delete_type TABLET_DATA_DELETED (replaced by tablet b13afacd37474781877b9f2cd20b313a) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:24.916936 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet b7b802f976f3471faaaecc964ec87e48 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:16 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:24.917646 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 764bfeb2a7d14c40815b5ec1b07bdd62 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:23 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:24.918556 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 4855448edb4b40c184838aa7f81c112b with delete_type TABLET_DATA_DELETED (replaced by tablet 5e2472ee5d6e4d3bbf1f24027ba531ca) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:24.921952 2221 sys_catalog.cc:263] Verifying existing consensus state
W20250901 01:52:24.922757 858 leader_election.cc:336] T 14b466b2a5a249aa841f1b8b06301edc P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:24.929349 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet c49d1143d54b4c319739488906b0e271 with delete_type TABLET_DATA_DELETED (replaced by tablet 69f61ec283484f9d8b3eebcea27f741b) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:24.930131 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 0cb3262973934c8689b851adad60afe0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:16 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:24.931627 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet b6daeff0070a4db08e90020752d21c29 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:14 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:24.932435 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ee3f7912ecf14cc382b312086ddee050"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.933068 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 5e2472ee5d6e4d3bbf1f24027ba531ca with delete_type TABLET_DATA_DELETED (replaced by tablet 0813cc68d4b0488ba60a523e4479971b) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:24.933307 2118 tablet_replica.cc:331] T 5e2472ee5d6e4d3bbf1f24027ba531ca P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:24.933445 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 3c12d6a70ac24dce82e851782e6fe5aa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:24 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:24.933496 2118 raft_consensus.cc:2241] T 5e2472ee5d6e4d3bbf1f24027ba531ca P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:24.933461 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5dd64d4e66604b8e93705b6cc8113298"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.933640 2118 raft_consensus.cc:2270] T 5e2472ee5d6e4d3bbf1f24027ba531ca P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:24.933645 970 heartbeater.cc:344] Connected to a master server at 127.0.104.62:43871
I20250901 01:52:24.934181 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c1c42a5c9929425aa0e7056d89b78c21"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.934253 2118 ts_tablet_manager.cc:1905] T 5e2472ee5d6e4d3bbf1f24027ba531ca P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:24.934901 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_52cbfbb3aea8476493872f9bc5beb6c6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.934967 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a6781ac9adb14c8bbfe62860977940c3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.935611 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7318a75e01da4287aabc511d9bb5945d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.935819 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b41fc09a03124018a20a1d2e8cf5dce6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.936640 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d721b98e73484675a2f8b21b00471148"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.936772 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1a6fd6554e8044ed83494f05d9716595"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.937389 1484 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_420b0979d6fd42da9b320456f4ccc9ce" } modify_external_catalogs: true
I20250901 01:52:24.937558 1484 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_420b0979d6fd42da9b320456f4ccc9ce" } modify_external_catalogs: true
I20250901 01:52:24.941303 1485 master_service.cc:948] ReplaceTablet: received request to replace tablet ad1a6be9fcc045d3bb4e71a2f6bd5ccd from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:24.943336 1150 tablet_bootstrap.cc:492] T a9b7f2e6548043a7865e25387cdd06c9 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:24.943897 1150 ts_tablet_manager.cc:1397] T a9b7f2e6548043a7865e25387cdd06c9 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.072s user 0.018s sys 0.014s
I20250901 01:52:24.946288 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9d6cb2264e784ba4a7a887664fa3466a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.946421 2118 ts_tablet_manager.cc:1918] T 5e2472ee5d6e4d3bbf1f24027ba531ca P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:24.946822 2118 log.cc:1199] T 5e2472ee5d6e4d3bbf1f24027ba531ca P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/5e2472ee5d6e4d3bbf1f24027ba531ca
I20250901 01:52:24.947268 2118 ts_tablet_manager.cc:1939] T 5e2472ee5d6e4d3bbf1f24027ba531ca P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:24.947816 2118 tablet_replica.cc:331] T 3c12d6a70ac24dce82e851782e6fe5aa P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:24.947937 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8a88347a18784b328c376a8380d0baf9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.949074 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_631d9d54ff124c9db6968bd0d25b9e34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.947665 1150 raft_consensus.cc:357] T a9b7f2e6548043a7865e25387cdd06c9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.949687 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_62b9d67b065c48ebb4941894d8ed4320"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.949734 1150 raft_consensus.cc:383] T a9b7f2e6548043a7865e25387cdd06c9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:24.949841 2118 raft_consensus.cc:2241] T 3c12d6a70ac24dce82e851782e6fe5aa P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:24.949782 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e5092813c4344ad39421a17195e28a12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.949849 1150 raft_consensus.cc:738] T a9b7f2e6548043a7865e25387cdd06c9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:24.950023 2118 raft_consensus.cc:2270] T 3c12d6a70ac24dce82e851782e6fe5aa P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:24.950320 1150 consensus_queue.cc:260] T a9b7f2e6548043a7865e25387cdd06c9 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.950555 2118 ts_tablet_manager.cc:1905] T 3c12d6a70ac24dce82e851782e6fe5aa P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:24.950493 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_29adbf08cf174958a32b125b8c2f2393"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.950536 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2b3431e0ca2349bc9c53ca5c60b9095f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.951181 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_385d26f5379e4e59b994de78b314dff7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.951354 1150 ts_tablet_manager.cc:1428] T a9b7f2e6548043a7865e25387cdd06c9 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.007s user 0.000s sys 0.003s
I20250901 01:52:24.936352 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2d019fcbe6334179b1903edfdbfb2790"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.951720 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa424ac6ef1441e6b1cc2867b1c48186" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:24.952226 1150 tablet_bootstrap.cc:492] T abf6e3be60f74872b2308dc95bf9cd37 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:24.952816 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 8dd587304e6342edbe96c9897b40e5c9 with delete_type TABLET_DATA_DELETED (replaced by tablet 05630a72ddae4de0b890dece2feacc31) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:24.952253 914 raft_consensus.cc:2466] T aa424ac6ef1441e6b1cc2867b1c48186 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
W20250901 01:52:24.953723 1489 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f9b293bf6a17453b9ea9892a6970d0b0 consensus state has no leader
I20250901 01:52:24.954685 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f431c492458549558bde8a5f4fee1ef8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.954798 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_daeeaee5be6a402d8e6e2a5041604f59"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.955377 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_eff7ded72f5a4eba8ff24a66e04d07f1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.960785 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3b97701e28ea4c7a9dd46b2cf04d6cf1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.960942 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_131c4e5c092c48a99691f2157c668fc0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.961604 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_dcf710747cf04df08cc9dcc2427c3098"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.961783 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_626ba46df22a408e8e4ee8d71d460097"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.962659 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_43ddfc4db5bc4fdb93ed959f0e866add"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.962709 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3096d280533d4319b6342a7cfcf23da4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.963676 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 5e2472ee5d6e4d3bbf1f24027ba531ca (table default.Table_52733c9a363b4ed9b6ff11ad8571e421 [id=453153553bea48319e132857f628d48a]) successfully deleted
W20250901 01:52:24.965031 858 leader_election.cc:336] T ad1a6be9fcc045d3bb4e71a2f6bd5ccd P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:24.968442 859 leader_election.cc:343] T 1495de70f0844f82a0eea99c059f076b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
W20250901 01:52:24.968547 1492 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet d0c3c363dda444519a17dca63ad9caa7 (table default.Table_0b4ac0a1caa5472d909448e41318e679 [id=8c1b263824ec4c15a5ec424364c78895], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet d0c3c363dda444519a17dca63ad9caa7 consensus state has no leader
I20250901 01:52:24.968575 859 leader_election.cc:304] T 1495de70f0844f82a0eea99c059f076b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:24.968942 2231 raft_consensus.cc:2747] T 1495de70f0844f82a0eea99c059f076b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:24.969393 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1793305d6cdb41d3a42db42bb3dc0557"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.969527 1488 master_service.cc:948] ReplaceTablet: received request to replace tablet 1a60c10e42104502ac2d3b80b8126ceb from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:24.969919 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_04537c58d5744a57a317b761e6276887"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.970333 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e1ba4bfb7f5c4b5389d64bb5df0d168d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.972968 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 74b709cb5ec64006b2f12bead8217d80 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:35946
W20250901 01:52:24.974682 1492 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 75a703c8e20a4a2fa0ab6ec58d41ef92 (table default.Table_2f09538b6fbe401f909105decb56e1cb [id=a90d1f10531042c3bb7a196b8be61c5f], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 75a703c8e20a4a2fa0ab6ec58d41ef92 consensus state has no leader
I20250901 01:52:24.974761 1150 tablet_bootstrap.cc:654] T abf6e3be60f74872b2308dc95bf9cd37 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:24.975373 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2b628365b173463f82d87322fcda32f0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.975867 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_795328d5129742f492132fc3cadab3e6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.976212 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_0113d0c15ce64fb48d38d211744b5d83"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.977051 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9ebca78780204b76ae85021e4c16e5ee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.977458 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f05ff26d6234479ca3c56819d164a154"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.977773 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_12f31c93bfee4cb39a4ff635e27920e0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.969547 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_53a851fc25154afa83d269f65da853d6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.979511 2113 ts_tablet_manager.cc:1918] T 756cfe84a517459ca015aecf5cd81571 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 01:52:24.979789 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet f50f33de1b654663b9ef8a91e3436f11 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:13 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:24.984503 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 64fb46b942ff4e4c8749986ecc3dfe65 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:24.985503 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 66a48a8ebe154b2cac89c234eaa378ff with delete_type TABLET_DATA_DELETED (replaced by tablet 15e4d972e8254cf899c3e0b71b6e2bd9) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:24.986403 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet f3962b471bdb49aea6e9fa95f1f9dfe2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:17 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:24.987879 2113 log.cc:1199] T 756cfe84a517459ca015aecf5cd81571 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/756cfe84a517459ca015aecf5cd81571
I20250901 01:52:24.972658 992 leader_election.cc:304] T f4fd117c395e41c2ab679c684908c5b7 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:24.988866 2210 raft_consensus.cc:2802] T f4fd117c395e41c2ab679c684908c5b7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:24.988940 2210 raft_consensus.cc:491] T f4fd117c395e41c2ab679c684908c5b7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:24.989006 2210 raft_consensus.cc:3058] T f4fd117c395e41c2ab679c684908c5b7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:24.989876 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 3fda93767e494a4684978c8522b829dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:15 UTC) from {username='slave'} at 127.0.0.1:35946
W20250901 01:52:24.990196 1492 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet e9f1f892658b4736859ff507536920a1 (table default.Table_8825dd28afbe464193eb1b668cc9326a [id=9aa62cb634314a0a849612c54c1220d9], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet e9f1f892658b4736859ff507536920a1 consensus state has no leader
I20250901 01:52:24.991254 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet b85d64e02cc646e380581ac9f59dee9e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:24.991887 2210 raft_consensus.cc:513] T f4fd117c395e41c2ab679c684908c5b7 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.992583 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 088ef4861dc14f81ae470c3aaee5b4e1 with delete_type TABLET_DATA_DELETED (replaced by tablet 6ce9521b925b4f97bb4a810ce77e147e) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:24.992857 2210 leader_election.cc:290] T f4fd117c395e41c2ab679c684908c5b7 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:24.993007 2113 ts_tablet_manager.cc:1939] T 756cfe84a517459ca015aecf5cd81571 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:24.993311 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e0175edc36654d38bbd51e3e7546bcf4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.996498 1151 tablet_bootstrap.cc:492] T d0c3c363dda444519a17dca63ad9caa7 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:24.996737 1151 ts_tablet_manager.cc:1397] T d0c3c363dda444519a17dca63ad9caa7 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.153s user 0.005s sys 0.000s
I20250901 01:52:24.997267 2221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Bootstrap starting.
I20250901 01:52:24.998028 1151 raft_consensus.cc:357] T d0c3c363dda444519a17dca63ad9caa7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:24.998492 2231 raft_consensus.cc:491] T 615b2306fca84003b264da9d5f95ddab P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:24.998525 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a99a0e13c3ea45b0accae01eb835075f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.998626 2231 raft_consensus.cc:513] T 615b2306fca84003b264da9d5f95ddab P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:24.998805 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8e3811bd31c64a6586fac3da3debecba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:24.999264 2231 leader_election.cc:290] T 615b2306fca84003b264da9d5f95ddab P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:24.999801 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ba6c249d43a0498db62487116c59bfde"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.000438 1150 tablet_bootstrap.cc:492] T abf6e3be60f74872b2308dc95bf9cd37 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:25.000715 1150 ts_tablet_manager.cc:1397] T abf6e3be60f74872b2308dc95bf9cd37 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.049s user 0.004s sys 0.004s
I20250901 01:52:25.000741 1872 raft_consensus.cc:491] T 824229183e024feaa975c7f0387c9c1a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:25.000742 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9ab0e754279743c1b3495f95ba863f58"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.000867 1872 raft_consensus.cc:513] T 824229183e024feaa975c7f0387c9c1a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:25.001376 1872 leader_election.cc:290] T 824229183e024feaa975c7f0387c9c1a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:25.002354 2210 consensus_queue.cc:1035] T 7b16ca1daf394a6da65a4af8da82ec33 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:25.002704 1150 raft_consensus.cc:357] T abf6e3be60f74872b2308dc95bf9cd37 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:25.003103 1150 raft_consensus.cc:383] T abf6e3be60f74872b2308dc95bf9cd37 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:25.003230 1150 raft_consensus.cc:738] T abf6e3be60f74872b2308dc95bf9cd37 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:25.003629 1150 consensus_queue.cc:260] T abf6e3be60f74872b2308dc95bf9cd37 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:25.004199 858 leader_election.cc:336] T 615b2306fca84003b264da9d5f95ddab P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:25.005373 1872 consensus_queue.cc:1035] T c49d1143d54b4c319739488906b0e271 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:25.005605 2231 raft_consensus.cc:491] T 4cde5f4f02044b7bbd7d43b49aac4ae8 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:25.005659 1990 consensus_queue.cc:1035] T d58f69fad01f403eab74a47d3695ed83 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:25.005978 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_857b6239b3f94d2f84dc294e951c6102"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:25.006110 992 leader_election.cc:343] T 60285f2ba037433999f5455fb9cc7acd P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 60285f2ba037433999f5455fb9cc7acd
I20250901 01:52:25.005996 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_499f9b5c9106431aa019c21d2e7e3f42"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.006815 992 leader_election.cc:304] T 25ce0fe8c9544f828694184d19d13df9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:25.007149 2210 raft_consensus.cc:2802] T 25ce0fe8c9544f828694184d19d13df9 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:25.007259 2210 raft_consensus.cc:695] T 25ce0fe8c9544f828694184d19d13df9 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:25.007642 2210 consensus_queue.cc:237] T 25ce0fe8c9544f828694184d19d13df9 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
W20250901 01:52:25.008008 1492 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet fa7fb88ded21432a8757f37ec3db5e71 (table default.Table_22d8d38d0b624912a0d8968fb764420e [id=808a8849c99e485b8be7cd17397874ec], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet fa7fb88ded21432a8757f37ec3db5e71 consensus state has no leader
I20250901 01:52:25.008718 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet e2aa32450a584395ae07b49f76a7b26d with delete_type TABLET_DATA_DELETED (replaced by tablet 7304937048ca48a7bd153f3cdcfaeff0) from {username='slave'} at 127.0.0.1:35946
W20250901 01:52:25.009568 1492 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 5f5396045979450193d461ba1334a5c9 (table default.Table_a89cc8e48563460baabc34c821ceae35 [id=9c3930e9fcfe4b49ba4bea67a2b0ad07], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 5f5396045979450193d461ba1334a5c9 consensus state has no leader
I20250901 01:52:25.010216 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet e8e7d2a72f704f52a8f8d46911a8a9e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:35946
W20250901 01:52:25.010794 1492 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 246b33338de647689d7404c70676e76f (table default.Table_aee74198b51d4f089538c4f2bf3ce4dc [id=a7b1d2a8ace14ef8bcce0a3bed41e85d], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet 246b33338de647689d7404c70676e76f consensus state has no leader
I20250901 01:52:25.010921 2231 raft_consensus.cc:513] T 4cde5f4f02044b7bbd7d43b49aac4ae8 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:25.011715 2231 leader_election.cc:290] T 4cde5f4f02044b7bbd7d43b49aac4ae8 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:25.014008 1150 ts_tablet_manager.cc:1428] T abf6e3be60f74872b2308dc95bf9cd37 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20250901 01:52:25.016283 2118 ts_tablet_manager.cc:1918] T 3c12d6a70ac24dce82e851782e6fe5aa P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:25.016788 2113 tablet_replica.cc:331] T 533d14f5903e49599a34ea353ffd8c00 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:25.016989 2113 raft_consensus.cc:2241] T 533d14f5903e49599a34ea353ffd8c00 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:25.017120 2113 raft_consensus.cc:2270] T 533d14f5903e49599a34ea353ffd8c00 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:25.017616 2113 ts_tablet_manager.cc:1905] T 533d14f5903e49599a34ea353ffd8c00 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:25.017071 1150 tablet_bootstrap.cc:492] T df01d5df9fc2424eaf9de4e68e781803 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:24.998283 1151 raft_consensus.cc:383] T d0c3c363dda444519a17dca63ad9caa7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:25.018808 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 756cfe84a517459ca015aecf5cd81571 (table default.Table_ce31a83b98ea4b698acbbdc1379419f5 [id=f49c96ff2eaa44cc9b754225cb9e6fdf]) successfully deleted
I20250901 01:52:25.018707 1151 raft_consensus.cc:738] T d0c3c363dda444519a17dca63ad9caa7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:25.019397 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3a865810f0434292a90d0fa13b28d6af"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.019701 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 75345ea2a3dc40fc8c9f8090d8839d2d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:25.019877 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 5a8c21aee9f84216b6f90538a5bf4c17 with delete_type TABLET_DATA_DELETED (replaced by tablet 42c1156e8dd84c6fa1396b46de13eb6f) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:25.019541 1151 consensus_queue.cc:260] T d0c3c363dda444519a17dca63ad9caa7 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:25.018832 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_51d81b5a28384f72b5f92f4ef73874cd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.021386 1151 ts_tablet_manager.cc:1428] T d0c3c363dda444519a17dca63ad9caa7 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.025s user 0.002s sys 0.000s
W20250901 01:52:25.021415 1492 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 617cc33d0d23439cb5f1e0c31bf11281 (table default.Table_ce1fd26ac65a4ed085d023e10a849e5c [id=fb088d31bba64f04871fbfd355d6263b], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 617cc33d0d23439cb5f1e0c31bf11281 consensus state has no leader
I20250901 01:52:25.021833 1151 tablet_bootstrap.cc:492] T 15e4d972e8254cf899c3e0b71b6e2bd9 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
W20250901 01:52:25.022013 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet dac01ff54b97443289d4f3d10f80cd41 because the tablet was not found. No further retry: Not found: Tablet not found: dac01ff54b97443289d4f3d10f80cd41
I20250901 01:52:25.023630 1483 master_service.cc:948] ReplaceTablet: received request to replace tablet b11e8f77243e4764b96548d1132b8e5a from {username='slave'} at 127.0.0.1:33478
W20250901 01:52:25.023806 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet c754129977904f978f87fd89d7092e36 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c754129977904f978f87fd89d7092e36 already in progress: creating tablet
W20250901 01:52:25.024355 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 31614db4c53a4f3c96fd190d326ce601 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 31614db4c53a4f3c96fd190d326ce601 already in progress: creating tablet
I20250901 01:52:25.024571 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ede4cf2c2b9246b3aef0636a6c24f7fe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.025632 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d0c06fef88464774b9c743d6e660c896"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.026367 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d794094769344204b8e28d7efbbc3a04"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.027757 1151 tablet_bootstrap.cc:654] T 15e4d972e8254cf899c3e0b71b6e2bd9 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:25.028151 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2c05a2b0b55f4e9ea8ca5350bb6aa6be"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.029273 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_547e6fdf472c4e038079d1eedeac8de5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.030205 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d22f1a7e1bde4ea3af755f90fcc49b26"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:25.036619 992 leader_election.cc:343] T 12c04e0c1f804179ae23509cdb24d395 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 12c04e0c1f804179ae23509cdb24d395
I20250901 01:52:25.037621 1482 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_16ca44aa872f426f858aa71d8491b750" } new_table_name: "default.Table_11f4f2d9a7404f919fad5a1c628cf5cb" modify_external_catalogs: true
I20250901 01:52:25.039337 1034 tablet_service.cc:1515] Processing DeleteTablet for tablet 50472a7aa2f04e0fbd65ac6c86c34b3e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:20 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:25.039530 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet b541a6e14f114ab29a864cca11806316 with delete_type TABLET_DATA_DELETED (replaced by tablet 2599f98abfdc4871b573edfc2ad97d22) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:25.039686 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet b633b461dbf34e13bc6e1cc08e6d20c1 with delete_type TABLET_DATA_DELETED (replaced by tablet 4855448edb4b40c184838aa7f81c112b) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:25.040542 2118 log.cc:1199] T 3c12d6a70ac24dce82e851782e6fe5aa P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/3c12d6a70ac24dce82e851782e6fe5aa
I20250901 01:52:25.034034 1489 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_c1959bda12f14bc9995d832476abf670" } modify_external_catalogs: true
I20250901 01:52:25.042717 1489 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_c1959bda12f14bc9995d832476abf670" } modify_external_catalogs: true
W20250901 01:52:25.043766 992 leader_election.cc:343] T 4d27fb61943c45d4aca67689cc3a9ff3 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 4d27fb61943c45d4aca67689cc3a9ff3
W20250901 01:52:25.036096 1492 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet b164b4a67a9143ceaff69b0ce49d63fe (table default.Table_da2c70a8f2fe49a0a8184e88c0a9e0ea [id=c1b1e9e3876244f68d31c63ce7bdebf1], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet b164b4a67a9143ceaff69b0ce49d63fe consensus state has no leader
I20250901 01:52:25.044044 1993 consensus_queue.cc:1035] T 822382a1781b4304bce29355ce83730e P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250901 01:52:25.044137 1492 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet ae21b9ae2e3b472eb67fb5f96388a50f (table default.Table_046289f393ff432aae0b033bdec1fb17 [id=db57b18a9ce24e068b362f5fbe940531], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet ae21b9ae2e3b472eb67fb5f96388a50f consensus state has no leader
I20250901 01:52:25.044828 1993 consensus_queue.cc:1035] T 121803e79ab64ec0b948b448646b9d0a P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:25.045619 992 leader_election.cc:304] T aa424ac6ef1441e6b1cc2867b1c48186 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:25.046089 1150 tablet_bootstrap.cc:654] T df01d5df9fc2424eaf9de4e68e781803 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:25.048799 2118 ts_tablet_manager.cc:1939] T 3c12d6a70ac24dce82e851782e6fe5aa P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
W20250901 01:52:25.049149 858 leader_election.cc:336] T 4cde5f4f02044b7bbd7d43b49aac4ae8 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:25.044346 1492 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet d4c9aa723aee4dc4b6237cb5c3722b52 (table default.Table_5e65c5f068b84995a2d7a76fa65dda0f [id=b9d81ddd38fe47838877b010be2b40d0], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet d4c9aa723aee4dc4b6237cb5c3722b52 consensus state has no leader
I20250901 01:52:25.051057 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 3c12d6a70ac24dce82e851782e6fe5aa (table default.Table_d2049d0c20bb4949ae6c249b690e7f86 [id=26bba0e60b6b43eb87d084be403c556b]) successfully deleted
W20250901 01:52:25.052063 1492 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 93bcac1fd95d44dc927abfca81692921 (table default.Table_b8f8ee74648c406eaf589d3ab80934db [id=b8f69313439543be9c11eea88dfa69c5], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 93bcac1fd95d44dc927abfca81692921 consensus state has no leader
I20250901 01:52:25.033747 1484 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 4410da9e95454ecf95faf80221b0a71c
W20250901 01:52:25.054761 1484 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 4410da9e95454ecf95faf80221b0a71c: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
W20250901 01:52:25.056833 1492 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 3dae00aa38204ee3ac77325eb334be42 (table default.Table_faf7ef12324f454396e5ae64d64ebe51 [id=5812902df9ea406ba59ff0ff23f130b6], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 3dae00aa38204ee3ac77325eb334be42 consensus state has no leader
I20250901 01:52:25.042336 1483 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet b11e8f77243e4764b96548d1132b8e5a
W20250901 01:52:25.057883 1483 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet b11e8f77243e4764b96548d1132b8e5a: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:25.058557 1483 catalog_manager.cc:6652] ReplaceTablet: tablet b11e8f77243e4764b96548d1132b8e5a deleted and replaced by tablet 96824e68bbf24b8287dbb683038ffaba
I20250901 01:52:25.034091 1488 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_5610feff8a344dc9be4bc1f0ad8ddd51" } modify_external_catalogs: true
I20250901 01:52:25.060103 1488 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_5610feff8a344dc9be4bc1f0ad8ddd51" } modify_external_catalogs: true
I20250901 01:52:25.058943 1483 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_203547c9458e43b3b48ea051d56892c6" } new_table_name: "default.Table_8dd9c9817a5c4e3dbb0bbf1d5cf1b5db" modify_external_catalogs: true
I20250901 01:52:25.069164 2231 raft_consensus.cc:491] T 0dbdcb76eeba4fefa56e7835c23c0316 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:25.069352 2231 raft_consensus.cc:513] T 0dbdcb76eeba4fefa56e7835c23c0316 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:25.070024 2231 leader_election.cc:290] T 0dbdcb76eeba4fefa56e7835c23c0316 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:25.070868 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 4410da9e95454ecf95faf80221b0a71c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:24 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.071296 894 tablet_service.cc:1515] Processing DeleteTablet for tablet b11e8f77243e4764b96548d1132b8e5a with delete_type TABLET_DATA_DELETED (replaced by tablet 96824e68bbf24b8287dbb683038ffaba) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.072444 2118 tablet_replica.cc:331] T 4410da9e95454ecf95faf80221b0a71c P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:25.072688 2118 raft_consensus.cc:2241] T 4410da9e95454ecf95faf80221b0a71c P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:25.073019 2118 raft_consensus.cc:2270] T 4410da9e95454ecf95faf80221b0a71c P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:25.073666 2118 ts_tablet_manager.cc:1905] T 4410da9e95454ecf95faf80221b0a71c P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:25.076370 1150 tablet_bootstrap.cc:492] T df01d5df9fc2424eaf9de4e68e781803 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:25.076561 1150 ts_tablet_manager.cc:1397] T df01d5df9fc2424eaf9de4e68e781803 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.060s user 0.002s sys 0.003s
W20250901 01:52:25.077338 1492 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet cb7d7e1796c0456f8de98fb812b9e374 (table default.Table_c4148158ed7b48a3946559ba978e72fd [id=327c5a63904540508142ab600c308757], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet cb7d7e1796c0456f8de98fb812b9e374 consensus state has no leader
I20250901 01:52:25.077365 1150 raft_consensus.cc:357] T df01d5df9fc2424eaf9de4e68e781803 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:25.077566 1150 raft_consensus.cc:383] T df01d5df9fc2424eaf9de4e68e781803 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:25.057343 1484 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_aba07c28a3e248e4b102730106d6a2ba" } new_table_name: "default.Table_bbbde49530e34a76b7cb17988ac6baca" modify_external_catalogs: true
I20250901 01:52:25.077653 1150 raft_consensus.cc:738] T df01d5df9fc2424eaf9de4e68e781803 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:25.079802 1150 consensus_queue.cc:260] T df01d5df9fc2424eaf9de4e68e781803 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:25.081241 1150 ts_tablet_manager.cc:1428] T df01d5df9fc2424eaf9de4e68e781803 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
W20250901 01:52:25.080771 1492 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 33f637f6a8134215bb1995ec38440073 (table default.Table_2ece91972d494680a8ad00273c10dee4 [id=557afe29976d40f6bfb6ae4ca348f887], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 33f637f6a8134215bb1995ec38440073 consensus state has no leader
I20250901 01:52:25.081575 1150 tablet_bootstrap.cc:492] T 5afae316ff164413be0d25d4bc0f58dc P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
W20250901 01:52:25.087769 1492 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet f80f13db501c42c7950499df8fcb6fc8 (table default.Table_ce86974fd8974de9a5a737954fac2ad7 [id=ff772d20b287476f94716379b86d2a71], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet f80f13db501c42c7950499df8fcb6fc8 consensus state has no leader
I20250901 01:52:25.088088 1150 tablet_bootstrap.cc:654] T 5afae316ff164413be0d25d4bc0f58dc P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:25.049117 2113 ts_tablet_manager.cc:1918] T 533d14f5903e49599a34ea353ffd8c00 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:25.090240 2113 log.cc:1199] T 533d14f5903e49599a34ea353ffd8c00 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/533d14f5903e49599a34ea353ffd8c00
I20250901 01:52:25.090360 1993 raft_consensus.cc:2802] T aa424ac6ef1441e6b1cc2867b1c48186 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:25.090505 1993 raft_consensus.cc:491] T aa424ac6ef1441e6b1cc2867b1c48186 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:25.090603 2113 ts_tablet_manager.cc:1939] T 533d14f5903e49599a34ea353ffd8c00 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:25.090641 1993 raft_consensus.cc:3058] T aa424ac6ef1441e6b1cc2867b1c48186 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
W20250901 01:52:25.091030 1492 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 3618a3eb4f174c43805ce7dc0e5315cb (table default.Table_83f2c0acedbc4c6982caf5313646e90b [id=31a2e50c32644321825da40176127015], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 3618a3eb4f174c43805ce7dc0e5315cb consensus state has no leader
W20250901 01:52:25.091202 1492 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet d1f2d98712ff4ea6aec665512de068bd (table default.Table_82d91a1ba2eb4265896824028dec8d3a [id=3f7624d2901445e39155c12fa68e34e9], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet d1f2d98712ff4ea6aec665512de068bd consensus state has no leader
I20250901 01:52:25.091842 2113 tablet_replica.cc:331] T 75e7a90815e449c39f386901a504143c P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:25.092061 2113 raft_consensus.cc:2241] T 75e7a90815e449c39f386901a504143c P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:25.092350 2113 raft_consensus.cc:2270] T 75e7a90815e449c39f386901a504143c P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:25.051029 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 20f7bfab18134e079e91c52c89468cc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:17 UTC) from {username='slave'} at 127.0.0.1:35946
E20250901 01:52:25.039794 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae4844183a3c4a04a775e6fd878391a3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_aee74198b51d4f089538c4f2bf3ce4dc'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:25.052474 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 56f2c4d2e6ab4bb4bdd6d025843589b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:25.052881 1034 tablet_service.cc:1515] Processing DeleteTablet for tablet 879fa66e55544118aafe504587393c66 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:23 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:25.053220 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet ef58f477ebeb49639ed7d071e1a62c10 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:13 UTC) from {username='slave'} at 127.0.0.1:35946
E20250901 01:52:25.092800 1696 catalog_manager.cc:6205] Error selecting replicas for tablet be8dfbc466564842a2fdb4953eee939b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_75ba3ba70ca24a94920e7bce96190ff2'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:25.053584 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet f98d1d8d660443cb8066a547e5d9b77f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:23 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:25.054482 1036 tablet_service.cc:1515] Processing DeleteTablet for tablet 60285f2ba037433999f5455fb9cc7acd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:13 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:25.054639 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet e1c8fe96a6ba4edba4f09c00f7add75e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:25.055505 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet dd694cfa09564d3da5d279c0861a4c32 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:12 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:25.055704 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 4fca1435d82447f289fda4035aa2ed89 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:20 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:25.056604 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 4410da9e95454ecf95faf80221b0a71c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:24 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:25.059679 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet b11e8f77243e4764b96548d1132b8e5a with delete_type TABLET_DATA_DELETED (replaced by tablet 96824e68bbf24b8287dbb683038ffaba) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:25.079491 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 49d30da95b14400b845270122487a2a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:25.079957 1035 tablet_service.cc:1515] Processing DeleteTablet for tablet a774cc6ae3924d299127fd9ffb01ca57 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:16 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:25.093215 2113 ts_tablet_manager.cc:1905] T 75e7a90815e449c39f386901a504143c P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:25.080498 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet a7ff996482164776b468cf305989866b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:13 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:25.082020 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet a54af99bc10045e8b0b6868e5aed98b4 with delete_type TABLET_DATA_DELETED (replaced by tablet 50472a7aa2f04e0fbd65ac6c86c34b3e) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:25.089330 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet c95798f0daf4437db18a291c4464522e with delete_type TABLET_DATA_DELETED (replaced by tablet 7a2f6e2c7ee2402ea310c058e046416c) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:25.089777 1038 tablet_service.cc:1515] Processing DeleteTablet for tablet f06a77ecc841462396a8d5026c7327f4 with delete_type TABLET_DATA_DELETED (replaced by tablet f3962b471bdb49aea6e9fa95f1f9dfe2) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:25.080132 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 438cd35ff3f74c49b908d072560077db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:25.094017 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 533d14f5903e49599a34ea353ffd8c00 (table default.Table_489b87a28dac48d093c241f301d5cccc [id=5fdd352d5e5e47a3a2bd8fd608a263dd]) successfully deleted
W20250901 01:52:25.094710 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 56475ba491a24301a534304811f30140 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 56475ba491a24301a534304811f30140 already in progress: creating tablet
W20250901 01:52:25.095376 992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:40561, user_credentials={real_user=slave}} blocked reactor thread for 49737.7us
W20250901 01:52:25.095551 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 5e2472ee5d6e4d3bbf1f24027ba531ca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5e2472ee5d6e4d3bbf1f24027ba531ca already in progress: creating tablet
W20250901 01:52:25.097512 1482 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0f37a24e693847f98b574950379081f3 consensus state has no leader
I20250901 01:52:25.098688 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet b73e362321284df1b18ef755dfe930a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:15 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:25.100471 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet c763d252598745a088344aa0eb5fa3d3 with delete_type TABLET_DATA_DELETED (replaced by tablet a8483e7212d94daba7e5401256c33aef) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:25.101018 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 579b9cdf819f4609a532e96bec093fd5 with delete_type TABLET_DATA_DELETED (replaced by tablet 56475ba491a24301a534304811f30140) from {username='slave'} at 127.0.0.1:35946
W20250901 01:52:25.101151 859 leader_election.cc:343] T 05630a72ddae4de0b890dece2feacc31 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:25.101330 859 leader_election.cc:304] T 05630a72ddae4de0b890dece2feacc31 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:25.101589 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 822382a1781b4304bce29355ce83730e with delete_type TABLET_DATA_DELETED (replaced by tablet 0958b0a4068344abb0f9db601d7dd2ad) from {username='slave'} at 127.0.0.1:35946
W20250901 01:52:25.101817 859 leader_election.cc:343] T 56475ba491a24301a534304811f30140 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:25.101969 859 leader_election.cc:304] T 56475ba491a24301a534304811f30140 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:25.102439 859 leader_election.cc:343] T 14b466b2a5a249aa841f1b8b06301edc P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:25.102589 859 leader_election.cc:304] T 14b466b2a5a249aa841f1b8b06301edc P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
E20250901 01:52:25.102587 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 7317eb7848ee44698038a4b404d4bf21, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b0e916c76a7a4766aa0887cc8691488f'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:25.102741 1696 catalog_manager.cc:6205] Error selecting replicas for tablet e2ed32b3292141278af28026528e96c2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_50ae3908a7944f70949c54be8fbf1564'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:25.102846 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 15a4f99ae43745b9a3f60052d916a7bd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_23fb8cbcf1bd43fa8346f3c26c8b4c65'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:25.102885 1981 raft_consensus.cc:491] T 48a211e174054866b030cdfb0cf947f4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
E20250901 01:52:25.102972 1696 catalog_manager.cc:6205] Error selecting replicas for tablet a2ca69fabadd4ad69e1670cae7a0c503, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_32c4716e3aa24a87a6b1472984b255c4'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:25.103080 1696 catalog_manager.cc:6205] Error selecting replicas for tablet cc3b8736b9094d7e892c80da4d5b20f6, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_93afd37289414655980b22934695bd55'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:25.103194 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9074543dea9949e7a085717b40913348, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_53e30205c5fd488eb3b48c685a159d04'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:25.103080 1981 raft_consensus.cc:513] T 48a211e174054866b030cdfb0cf947f4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
E20250901 01:52:25.103294 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 18498c9cb09141d08c1c5c0854b63ebd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77f4810bc83c4d948531da2a45a3792e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:25.103403 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 6ce9521b925b4f97bb4a810ce77e147e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_9e04f976a93f4d30baccf90fa3622dfe'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:25.104710 1981 leader_election.cc:290] T 48a211e174054866b030cdfb0cf947f4 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:25.104871 1981 raft_consensus.cc:2747] T 05630a72ddae4de0b890dece2feacc31 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:25.105314 2123 raft_consensus.cc:2747] T 14b466b2a5a249aa841f1b8b06301edc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:25.105028 1981 raft_consensus.cc:2747] T 56475ba491a24301a534304811f30140 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:25.113382 1492 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet ad45c90ba0da43c9860745e534664a98 (table default.Table_1cd3f3fb6b244b3f8796bfca9f7622c8 [id=6fd739b0e725424ca844dbb652ea8028], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet ad45c90ba0da43c9860745e534664a98 consensus state has no leader
I20250901 01:52:25.114459 2216 raft_consensus.cc:491] T 5a8c21aee9f84216b6f90538a5bf4c17 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:25.114624 2216 raft_consensus.cc:513] T 5a8c21aee9f84216b6f90538a5bf4c17 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:25.115290 2216 leader_election.cc:290] T 5a8c21aee9f84216b6f90538a5bf4c17 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:25.115291 1492 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet fecf7d538b4c4c619182d5eccb44dd8b (table default.Table_1b5729d9768546fe99eaafc4e2604f42 [id=02d0056abe2146ac9af81ebf4447b99d], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet fecf7d538b4c4c619182d5eccb44dd8b consensus state has no leader
I20250901 01:52:25.115538 2216 raft_consensus.cc:491] T 4855448edb4b40c184838aa7f81c112b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:25.115681 2216 raft_consensus.cc:513] T 4855448edb4b40c184838aa7f81c112b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:25.116258 2216 leader_election.cc:290] T 4855448edb4b40c184838aa7f81c112b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:25.118041 1492 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 8ca5cf4e99014c0d84a3acad4ef713af (table default.Table_ed6fd284be6f4ae3a008e21e7ce17930 [id=303e76c918bc4c39aa47a4d24849a358], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 8ca5cf4e99014c0d84a3acad4ef713af consensus state has no leader
W20250901 01:52:25.121625 858 leader_election.cc:336] T 0dbdcb76eeba4fefa56e7835c23c0316 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:25.126827 992 consensus_peers.cc:489] T 7b16ca1daf394a6da65a4af8da82ec33 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:25.126838 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet a59ac57aebe24860aa02f53ce0485910 with delete_type TABLET_DATA_DELETED (replaced by tablet 93bcac1fd95d44dc927abfca81692921) from {username='slave'} at 127.0.0.1:35946
W20250901 01:52:25.127017 992 leader_election.cc:336] T f4fd117c395e41c2ab679c684908c5b7 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:25.127007 919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "616ff34e8d314e0aa0e11ce60be11789" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:25.127240 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet abe27e749fa0485195e0064a49d5431e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:15 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:25.127249 919 raft_consensus.cc:2466] T 616ff34e8d314e0aa0e11ce60be11789 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:25.127633 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet 824229183e024feaa975c7f0387c9c1a with delete_type TABLET_DATA_DELETED (replaced by tablet d4c9aa723aee4dc4b6237cb5c3722b52) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:25.127959 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet 2ff914352c7c43de800f3d54ecf63072 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:23 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:25.127941 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "64fb46b942ff4e4c8749986ecc3dfe65" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:25.128122 914 raft_consensus.cc:2466] T 64fb46b942ff4e4c8749986ecc3dfe65 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:25.128320 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet 40d74d7b5a3b4d1dae09eebd9e07a0c4 with delete_type TABLET_DATA_DELETED (replaced by tablet 6cf09e8f299548dba2cce16d7b565bad) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:25.128593 2251 tablet.cc:1722] T fcb7b5b7d63d4f0b906fea97cb548de2 P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250901 01:52:25.128680 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet c9a4c80bec1d41458974a0c449aecaf7 with delete_type TABLET_DATA_DELETED (replaced by tablet 121803e79ab64ec0b948b448646b9d0a) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:25.129076 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet 121803e79ab64ec0b948b448646b9d0a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:14 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:25.129388 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet 82d163681fda47f38d79863bd28f576d with delete_type TABLET_DATA_DELETED (replaced by tablet bf8f0237233a49219cd1c41cf808bdba) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:25.129722 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet 4a87e68850c4459eb2c8eafac89c6aa9 with delete_type TABLET_DATA_DELETED (replaced by tablet 615b2306fca84003b264da9d5f95ddab) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:25.130059 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet 12c04e0c1f804179ae23509cdb24d395 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:13 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:25.130385 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet 670d90a37b934579b353bb4e4820127b with delete_type TABLET_DATA_DELETED (replaced by tablet 75a703c8e20a4a2fa0ab6ec58d41ef92) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:25.130749 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet 50e3da53433f4c14b57db8a733fa53bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:18 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:25.133229 1151 tablet_bootstrap.cc:492] T 15e4d972e8254cf899c3e0b71b6e2bd9 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:25.133435 1151 ts_tablet_manager.cc:1397] T 15e4d972e8254cf899c3e0b71b6e2bd9 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.112s user 0.004s sys 0.012s
I20250901 01:52:25.134322 1151 raft_consensus.cc:357] T 15e4d972e8254cf899c3e0b71b6e2bd9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:25.134562 1151 raft_consensus.cc:383] T 15e4d972e8254cf899c3e0b71b6e2bd9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:25.134645 1151 raft_consensus.cc:738] T 15e4d972e8254cf899c3e0b71b6e2bd9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:25.134898 1151 consensus_queue.cc:260] T 15e4d972e8254cf899c3e0b71b6e2bd9 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:25.135744 1151 ts_tablet_manager.cc:1428] T 15e4d972e8254cf899c3e0b71b6e2bd9 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250901 01:52:25.136119 1151 tablet_bootstrap.cc:492] T a289a66b2dfa44dc8c8547ff83c1716e P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
W20250901 01:52:25.131117 1473 catalog_manager.cc:5001] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): alter failed for tablet 34042ea5c1b645969c1de41fe587c343 (table default.Table_23fd89096be149b791975d89d65bc280 [id=edbc04078c904f29a16a1bbb354dad92]),no further retry: Not found: Tablet not found: 34042ea5c1b645969c1de41fe587c343
W20250901 01:52:25.141942 1492 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 89d40f0331114490b51de3ba6d773571 (table default.Table_4dacb3b1d2e94e42ac8937eb6eb58375 [id=8ee0aa1ba24042dd8ded004644913b9c], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 89d40f0331114490b51de3ba6d773571 consensus state has no leader
W20250901 01:52:25.144870 1492 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.3:51295 (request call id 179) took 1469 ms (client timeout 1000 ms). Trace:
W20250901 01:52:25.144990 1492 rpcz_store.cc:269] 0901 01:52:23.675565 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:23.675618 (+ 53us) service_pool.cc:225] Handling call
0901 01:52:23.720478 (+ 44860us) spinlock_profiling.cc:243] Waited 38.1 ms on lock 0x616000040c54. stack: 00007f0d64ec57d6 00007f0d64ec5590 00007f0d63eb83ad 0000000000972d94 0000000000972d5d 00007f0d73c9014d 00007f0d73c90243 00007f0d7094f941 00007f0d784daa5d 00007f0d7865e6d8 00007f0d6f498c86 00007f0d6f4989b5 00007f0d6bc732ce 00007f0d6bc721ae 00007f0d6bc783e5 00007f0d6bc78d0f
0901 01:52:25.144805 (+1424327us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0901 01:52:24.214052 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:24.214170 (+ 118us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:24.214177 (+ 7us) write_op.cc:435] Acquired schema lock
0901 01:52:24.214179 (+ 2us) tablet.cc:660] Decoding operations
0901 01:52:24.214702 (+ 523us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:24.214717 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:24.214722 (+ 5us) tablet.cc:683] Acquiring locks for 18 operations
0901 01:52:24.215044 (+ 322us) tablet.cc:699] Row locks acquired
0901 01:52:24.215049 (+ 5us) write_op.cc:260] PREPARE: finished
0901 01:52:24.215095 (+ 46us) write_op.cc:270] Start()
0901 01:52:24.215133 (+ 38us) write_op.cc:276] Timestamp: P: 1756691544215037 usec, L: 0
0901 01:52:24.215138 (+ 5us) op_driver.cc:348] REPLICATION: starting
0901 01:52:24.215371 (+ 233us) log.cc:844] Serialized 6886 byte log entry
0901 01:52:24.225255 (+ 9884us) op_driver.cc:464] REPLICATION: finished
0901 01:52:24.225453 (+ 198us) write_op.cc:301] APPLY: starting
0901 01:52:24.225479 (+ 26us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:24.225555 (+ 76us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:24.225591 (+ 36us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:24.225832 (+ 241us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:24.225881 (+ 49us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=18
0901 01:52:24.225889 (+ 8us) write_op.cc:312] APPLY: finished
0901 01:52:24.226013 (+ 124us) log.cc:844] Serialized 125 byte log entry
0901 01:52:24.226080 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:24.226117 (+ 37us) write_op.cc:454] Released schema lock
0901 01:52:24.226125 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0901 01:52:24.226197 (+ 72us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:24.226201 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0901 01:52:24.431526 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:24.432389 (+ 863us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:24.432412 (+ 23us) write_op.cc:435] Acquired schema lock
0901 01:52:24.432415 (+ 3us) tablet.cc:660] Decoding operations
0901 01:52:24.432551 (+ 136us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:24.432570 (+ 19us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:24.432578 (+ 8us) tablet.cc:683] Acquiring locks for 1 operations
0901 01:52:24.432662 (+ 84us) tablet.cc:699] Row locks acquired
0901 01:52:24.432665 (+ 3us) write_op.cc:260] PREPARE: finished
0901 01:52:24.432711 (+ 46us) write_op.cc:270] Start()
0901 01:52:24.432759 (+ 48us) write_op.cc:276] Timestamp: P: 1756691544432633 usec, L: 0
0901 01:52:24.432765 (+ 6us) op_driver.cc:348] REPLICATION: starting
0901 01:52:24.432985 (+ 220us) log.cc:844] Serialized 356 byte log entry
0901 01:52:24.438940 (+ 5955us) op_driver.cc:464] REPLICATION: finished
0901 01:52:24.439102 (+ 162us) write_op.cc:301] APPLY: starting
0901 01:52:24.439120 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:24.439163 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:24.439216 (+ 53us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:24.439289 (+ 73us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:24.439324 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0901 01:52:24.439330 (+ 6us) write_op.cc:312] APPLY: finished
0901 01:52:24.439407 (+ 77us) log.cc:844] Serialized 23 byte log entry
0901 01:52:24.439471 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:24.439491 (+ 20us) write_op.cc:454] Released schema lock
0901 01:52:24.439498 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0901 01:52:24.439579 (+ 81us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:24.439583 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0901 01:52:24.865383 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:24.865495 (+ 112us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:24.865502 (+ 7us) write_op.cc:435] Acquired schema lock
0901 01:52:24.865504 (+ 2us) tablet.cc:660] Decoding operations
0901 01:52:24.865609 (+ 105us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:24.865622 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:24.865627 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0901 01:52:24.865703 (+ 76us) tablet.cc:699] Row locks acquired
0901 01:52:24.865706 (+ 3us) write_op.cc:260] PREPARE: finished
0901 01:52:24.865748 (+ 42us) write_op.cc:270] Start()
0901 01:52:24.865784 (+ 36us) write_op.cc:276] Timestamp: P: 1756691544865686 usec, L: 0
0901 01:52:24.865788 (+ 4us) op_driver.cc:348] REPLICATION: starting
0901 01:52:24.865999 (+ 211us) log.cc:844] Serialized 356 byte log entry
0901 01:52:24.884757 (+ 18758us) op_driver.cc:464] REPLICATION: finished
0901 01:52:24.885491 (+ 734us) write_op.cc:301] APPLY: starting
0901 01:52:24.885501 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:24.885531 (+ 30us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:24.885559 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:24.885594 (+ 35us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:24.885622 (+ 28us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0901 01:52:24.885629 (+ 7us) write_op.cc:312] APPLY: finished
0901 01:52:24.885677 (+ 48us) log.cc:844] Serialized 23 byte log entry
0901 01:52:24.885724 (+ 47us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:24.885735 (+ 11us) write_op.cc:454] Released schema lock
0901 01:52:24.885740 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0901 01:52:24.885795 (+ 55us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:24.885798 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":68119,"reported_tablets":248,"spinlock_wait_cycles":87612672,"child_traces":[["op",{"apply.queue_time_us":146,"apply.run_cpu_time_us":737,"apply.run_wall_time_us":737,"num_ops":18,"prepare.queue_time_us":49,"prepare.run_cpu_time_us":1444,"prepare.run_wall_time_us":1444,"raft.queue_time_us":1538,"raft.run_cpu_time_us":251,"raft.run_wall_time_us":252,"replication_time_us":10084}],["op",{"apply.queue_time_us":101,"apply.run_cpu_time_us":468,"apply.run_wall_time_us":467,"num_ops":1,"prepare.queue_time_us":103,"prepare.run_cpu_time_us":1028,"prepare.run_wall_time_us":1721,"raft.queue_time_us":1455,"raft.run_cpu_time_us":255,"raft.run_wall_time_us":256,"replication_time_us":6144}],["op",{"apply.queue_time_us":703,"apply.run_cpu_time_us":296,"apply.run_wall_time_us":295,"num_ops":1,"prepare.queue_time_us":2092,"prepare.run_cpu_time_us":723,"prepare.run_wall_time_us":1208,"replication_time_us":18955}]]}
I20250901 01:52:25.147424 1492 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_1a9de416bd864f7ba7ed302579ebda5a" } new_table_name: "default.Table_e67df63f08fb415293f39cad748a0bd5" modify_external_catalogs: true
I20250901 01:52:25.148041 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 6741c56499b44f648fb032c8cf50205b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.148586 894 tablet_service.cc:1515] Processing DeleteTablet for tablet e8e7d2a72f704f52a8f8d46911a8a9e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.148967 894 tablet_service.cc:1515] Processing DeleteTablet for tablet e2aa32450a584395ae07b49f76a7b26d with delete_type TABLET_DATA_DELETED (replaced by tablet 7304937048ca48a7bd153f3cdcfaeff0) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.149199 2210 raft_consensus.cc:491] T c9a4c80bec1d41458974a0c449aecaf7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:25.149646 894 tablet_service.cc:1515] Processing DeleteTablet for tablet becbaae4fd7e44faa679e9479465a7c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:23 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.150018 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 31614db4c53a4f3c96fd190d326ce601 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.155005 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 879fa66e55544118aafe504587393c66 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:23 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.156174 1491 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 15a4f99ae43745b9a3f60052d916a7bd
I20250901 01:52:25.160437 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 4902062977ea42149e3466e999117322 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:18 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.160436 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c1a03e22b8de4acfa2aa83acf845f7dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.161547 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_44f6b8edfb1e431eaf7e804273a5e0df"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.162289 894 tablet_service.cc:1515] Processing DeleteTablet for tablet b81a12b38ac64efabd44aeb192d181a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:18 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.162333 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5217d0b541014b6e818b0a7ee44d2ed1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.163050 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_abfc6eac629c4289a009457f7e3352b6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.163690 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d6910b7791b546dba528f209f96e1956"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.163919 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 48a211e174054866b030cdfb0cf947f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:18 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.164573 894 tablet_service.cc:1515] Processing DeleteTablet for tablet b85d64e02cc646e380581ac9f59dee9e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.165277 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_505c0897ea34484fa32041b9ae0bb964"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.167208 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 1a60c10e42104502ac2d3b80b8126ceb with delete_type TABLET_DATA_DELETED (replaced by tablet 88ceba7f48b04d7885cc0164668700a5) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.168493 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 56475ba491a24301a534304811f30140 with delete_type TABLET_DATA_DELETED (replaced by tablet 7317eb7848ee44698038a4b404d4bf21) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.168829 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_80011a5dccc74414b6cd38ed22cf2fb6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.169420 894 tablet_service.cc:1515] Processing DeleteTablet for tablet afc002ae2486476b9208e1cf8215157c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:17 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.169565 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_fdffc3cf8e9a4cb48358dce5cb864244"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.170255 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_dbd91b1bde2c4023b6510df1f557940b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.170928 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 75e7a90815e449c39f386901a504143c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:18 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.171162 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_6be0e73c102e4f6d8fa63ea12f3e6993"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.171983 1491 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_33c84edd575e4969b264a684accb22e2" } modify_external_catalogs: true
I20250901 01:52:25.172195 1491 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_33c84edd575e4969b264a684accb22e2" } modify_external_catalogs: true
I20250901 01:52:25.172693 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 05630a72ddae4de0b890dece2feacc31 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:18 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.173360 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 7a607fac57994001b2304e5d81c9decc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:17 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.173895 1038 tablet_service.cc:1515] Processing DeleteTablet for tablet f8a5886ddb644a86b76df44aab309057 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:14 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:25.174099 894 tablet_service.cc:1515] Processing DeleteTablet for tablet e9b179dba2cc49428ba8090c962d6136 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:17 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.174718 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 74b709cb5ec64006b2f12bead8217d80 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.178409 1696 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 15a4f99ae43745b9a3f60052d916a7bd
I20250901 01:52:25.192764 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 8dd587304e6342edbe96c9897b40e5c9 with delete_type TABLET_DATA_DELETED (replaced by tablet 05630a72ddae4de0b890dece2feacc31) from {username='slave'} at 127.0.0.1:34472
W20250901 01:52:25.194851 1490 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:54603 (request call id 179) took 1485 ms (client timeout 1000 ms). Trace:
W20250901 01:52:25.195086 1490 rpcz_store.cc:269] 0901 01:52:23.709019 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:23.717522 (+ 8503us) service_pool.cc:225] Handling call
0901 01:52:25.194802 (+1477280us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":36494,"reported_tablets":219}
I20250901 01:52:25.196467 1490 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_02e911ce389e4ad581c0a2f976f99c4e" } new_table_name: "default.Table_9f48256fcac7429ba7d7fdb86979eda6" modify_external_catalogs: true
I20250901 01:52:25.196607 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 40d74d7b5a3b4d1dae09eebd9e07a0c4 with delete_type TABLET_DATA_DELETED (replaced by tablet 6cf09e8f299548dba2cce16d7b565bad) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.197279 894 tablet_service.cc:1515] Processing DeleteTablet for tablet f3962b471bdb49aea6e9fa95f1f9dfe2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:17 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.201021 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 75345ea2a3dc40fc8c9f8090d8839d2d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.201370 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 088ef4861dc14f81ae470c3aaee5b4e1 with delete_type TABLET_DATA_DELETED (replaced by tablet 6ce9521b925b4f97bb4a810ce77e147e) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.201689 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 831e42de95854541a49ae6b3e039d3e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:16 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.202026 894 tablet_service.cc:1515] Processing DeleteTablet for tablet f06a77ecc841462396a8d5026c7327f4 with delete_type TABLET_DATA_DELETED (replaced by tablet f3962b471bdb49aea6e9fa95f1f9dfe2) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.202361 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 64fb46b942ff4e4c8749986ecc3dfe65 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.202684 894 tablet_service.cc:1515] Processing DeleteTablet for tablet c763d252598745a088344aa0eb5fa3d3 with delete_type TABLET_DATA_DELETED (replaced by tablet a8483e7212d94daba7e5401256c33aef) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.203109 894 tablet_service.cc:1515] Processing DeleteTablet for tablet bb0cf5a6a27045a08df6845d7473184d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.203405 894 tablet_service.cc:1515] Processing DeleteTablet for tablet a54af99bc10045e8b0b6868e5aed98b4 with delete_type TABLET_DATA_DELETED (replaced by tablet 50472a7aa2f04e0fbd65ac6c86c34b3e) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.204311 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 7a2f6e2c7ee2402ea310c058e046416c with delete_type TABLET_DATA_DELETED (replaced by tablet 5f5396045979450193d461ba1334a5c9) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.207898 2171 raft_consensus.cc:491] T b541a6e14f114ab29a864cca11806316 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:25.209347 2170 raft_consensus.cc:491] T 9473f0ef978c4bfaa41fb7b0729e8863 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:25.211572 1151 tablet_bootstrap.cc:654] T a289a66b2dfa44dc8c8547ff83c1716e P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:25.212491 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 26c8ed083cea4dc0acca80d00b11b29b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:23 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.212550 1993 raft_consensus.cc:513] T aa424ac6ef1441e6b1cc2867b1c48186 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:25.212700 2216 raft_consensus.cc:491] T a774cc6ae3924d299127fd9ffb01ca57 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:25.212824 2216 raft_consensus.cc:513] T a774cc6ae3924d299127fd9ffb01ca57 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:25.213253 2216 leader_election.cc:290] T a774cc6ae3924d299127fd9ffb01ca57 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:25.213310 2171 raft_consensus.cc:513] T b541a6e14f114ab29a864cca11806316 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:25.213866 2171 leader_election.cc:290] T b541a6e14f114ab29a864cca11806316 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:25.214020 1038 tablet_service.cc:1515] Processing DeleteTablet for tablet 1286c0ab85704837a99b6e5a62184d77 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:16 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:25.214298 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet 9473f0ef978c4bfaa41fb7b0729e8863 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:17 UTC) from {username='slave'} at 127.0.0.1:35946
W20250901 01:52:25.127269 992 consensus_peers.cc:489] T d58f69fad01f403eab74a47d3695ed83 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:25.215713 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 17e8340dbd1a406cb782dad3fe88cc7e with delete_type TABLET_DATA_DELETED (replaced by tablet e2ed32b3292141278af28026528e96c2) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.213254 1993 leader_election.cc:290] T aa424ac6ef1441e6b1cc2867b1c48186 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:25.216084 894 tablet_service.cc:1515] Processing DeleteTablet for tablet e25923b2daf9429683b2d44a75a56189 with delete_type TABLET_DATA_DELETED (replaced by tablet 2973b59bb1df41d18d9a5b6a29ae336f) from {username='slave'} at 127.0.0.1:34472
W20250901 01:52:25.216097 859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:39447, user_credentials={real_user=slave}} blocked reactor thread for 112988us
I20250901 01:52:25.216382 885 tablet_service.cc:1515] Processing DeleteTablet for tablet f98d1d8d660443cb8066a547e5d9b77f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:23 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.216675 1485 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet ad1a6be9fcc045d3bb4e71a2f6bd5ccd
W20250901 01:52:25.216914 1485 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet ad1a6be9fcc045d3bb4e71a2f6bd5ccd: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:25.217283 1485 catalog_manager.cc:6652] ReplaceTablet: tablet ad1a6be9fcc045d3bb4e71a2f6bd5ccd deleted and replaced by tablet ae9a4ad4fdfb4771ba23a91f7007447e
I20250901 01:52:25.217543 1150 tablet_bootstrap.cc:492] T 5afae316ff164413be0d25d4bc0f58dc P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:25.217577 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 50472a7aa2f04e0fbd65ac6c86c34b3e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:20 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.217686 1150 ts_tablet_manager.cc:1397] T 5afae316ff164413be0d25d4bc0f58dc P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.136s user 0.006s sys 0.001s
I20250901 01:52:25.217650 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_531e65bada1e471c85b1929a6b31db82"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.219550 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_63dbace04311490d80653cc3b3a0d35c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.220543 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_184dcb20c1f042b6b5f59370c1192843"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.220702 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 764bfeb2a7d14c40815b5ec1b07bdd62 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:23 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.222018 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c08be7db99c64dfc8feaa80b17821ff2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.223012 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 56f2c4d2e6ab4bb4bdd6d025843589b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.223562 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 4fca1435d82447f289fda4035aa2ed89 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:20 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.224282 904 tablet_service.cc:1515] Processing DeleteTablet for tablet 0dbdcb76eeba4fefa56e7835c23c0316 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:17 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.224575 901 tablet_service.cc:1515] Processing DeleteTablet for tablet 670d90a37b934579b353bb4e4820127b with delete_type TABLET_DATA_DELETED (replaced by tablet 75a703c8e20a4a2fa0ab6ec58d41ef92) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.224818 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 1286c0ab85704837a99b6e5a62184d77 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:16 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.225246 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 2ff914352c7c43de800f3d54ecf63072 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:23 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.225594 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_be0bf3bb1c634f6abc0ba297fba4ea55"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.225800 900 tablet_service.cc:1515] Processing DeleteTablet for tablet 822382a1781b4304bce29355ce83730e with delete_type TABLET_DATA_DELETED (replaced by tablet 0958b0a4068344abb0f9db601d7dd2ad) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.225984 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 579b9cdf819f4609a532e96bec093fd5 with delete_type TABLET_DATA_DELETED (replaced by tablet 56475ba491a24301a534304811f30140) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.226166 886 tablet_service.cc:1515] Processing DeleteTablet for tablet d77aadd7291647a0bae7c421f244c252 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:14 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.226728 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7073b0abb3dc4ec197195bc476a89af9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.227494 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a0e2bf5c9cfa46b3933407de533b0a09"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.228292 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_423d108d491744caa2d397b4ce26a7ae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:25.228699 1492 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 669d26c6551c4fc58283b67a501d9bf1 consensus state has no leader
I20250901 01:52:25.241901 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_503f15e10bdd400b975b94f3a325cff4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.243037 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_bf746a0f5691412f92b7d4e3ccc17118"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.243920 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d69046310b1347bcb5425fd5c586546a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.244818 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a3ab0cc853f34d909b8bb0e63e986207"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.246527 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a1b1b90408e245b1b1503d3893d57015"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.246538 2170 raft_consensus.cc:513] T 9473f0ef978c4bfaa41fb7b0729e8863 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:25.245754 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_038dc321ca1b49b6b5f5b7b6346e0e68"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.247597 2210 raft_consensus.cc:513] T c9a4c80bec1d41458974a0c449aecaf7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:25.248517 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c8c874ecda96479e9ab82862bfd86caf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.249336 2210 leader_election.cc:290] T c9a4c80bec1d41458974a0c449aecaf7 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:25.249224 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7c4fcd28292d4135b151ee4b8905e6f4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.251650 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a875296f24aa4aaa9e9dd89209cea5e2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.252473 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_da7c02b55eba47d0b82fb47cf63e2a59"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.256285 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e4f8057a525e412481d27aafc1a29911"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.256912 1993 raft_consensus.cc:491] T b633b461dbf34e13bc6e1cc08e6d20c1 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:25.257095 1993 raft_consensus.cc:513] T b633b461dbf34e13bc6e1cc08e6d20c1 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:25.256726 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_be523b1e411f469a8c0262c9d9415770"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.257758 1993 leader_election.cc:290] T b633b461dbf34e13bc6e1cc08e6d20c1 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:25.258853 2171 raft_consensus.cc:491] T 438cd35ff3f74c49b908d072560077db P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:25.259017 2171 raft_consensus.cc:513] T 438cd35ff3f74c49b908d072560077db P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:25.259819 2171 leader_election.cc:290] T 438cd35ff3f74c49b908d072560077db P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:25.260139 1038 tablet_service.cc:1515] Processing DeleteTablet for tablet 154574e17f214cc0ad56e6e6be3af899 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:13 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:25.260576 2170 leader_election.cc:290] T 9473f0ef978c4bfaa41fb7b0729e8863 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:25.260720 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet 9d886fa63f8f4996b087997b542e8bb1 with delete_type TABLET_DATA_DELETED (replaced by tablet eaa521c52a2f4408b59770bd2a05907c) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:25.262449 1486 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 81ce448b148447ddbc41875d863e0518 (table default.Table_6e461d9cb0ae4dd095975bbc21533794 [id=49673fa73f8c436e9b8026efc4f3cbbc]). Expected version 1 got 0
I20250901 01:52:25.262770 1486 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet cb7d7e1796c0456f8de98fb812b9e374 (table default.Table_c4148158ed7b48a3946559ba978e72fd [id=327c5a63904540508142ab600c308757]). Expected version 1 got 0
I20250901 01:52:25.263161 1035 tablet_service.cc:1515] Processing DeleteTablet for tablet 26c8ed083cea4dc0acca80d00b11b29b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:23 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:25.263342 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 98cca86164db4252b1b49de88747822a with delete_type TABLET_DATA_DELETED (replaced by tablet fa7fb88ded21432a8757f37ec3db5e71) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:25.263485 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet bb0cf5a6a27045a08df6845d7473184d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:25.263629 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet d58f69fad01f403eab74a47d3695ed83 with delete_type TABLET_DATA_DELETED (replaced by tablet 74b709cb5ec64006b2f12bead8217d80) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:25.263787 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet d77aadd7291647a0bae7c421f244c252 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:14 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:25.264081 2113 ts_tablet_manager.cc:1918] T 75e7a90815e449c39f386901a504143c P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:25.264206 2113 log.cc:1199] T 75e7a90815e449c39f386901a504143c P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/75e7a90815e449c39f386901a504143c
I20250901 01:52:25.264282 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet ad1a6be9fcc045d3bb4e71a2f6bd5ccd with delete_type TABLET_DATA_DELETED (replaced by tablet ae9a4ad4fdfb4771ba23a91f7007447e) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:25.264600 2113 ts_tablet_manager.cc:1939] T 75e7a90815e449c39f386901a504143c P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:25.264744 1038 tablet_service.cc:1515] Processing DeleteTablet for tablet becbaae4fd7e44faa679e9479465a7c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:23 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:25.264575 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet 23af154a6f1c4e2e9352fb17c9024f10 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:15 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:25.264956 1486 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet fcb7b5b7d63d4f0b906fea97cb548de2 (table default.Table_2ceaff93c8f44f00aab53e48d8466dfd [id=4d9ea39016ca4bf4859f4f80ea4e9ed6]). Expected version 4 got 0
I20250901 01:52:25.265013 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet 7a2f6e2c7ee2402ea310c058e046416c with delete_type TABLET_DATA_DELETED (replaced by tablet 5f5396045979450193d461ba1334a5c9) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:25.265116 1486 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet b164b4a67a9143ceaff69b0ce49d63fe (table default.Table_da2c70a8f2fe49a0a8184e88c0a9e0ea [id=c1b1e9e3876244f68d31c63ce7bdebf1]). Expected version 1 got 0
I20250901 01:52:25.265527 1486 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet d1f2d98712ff4ea6aec665512de068bd (table default.Table_82d91a1ba2eb4265896824028dec8d3a [id=3f7624d2901445e39155c12fa68e34e9]). Expected version 1 got 0
W20250901 01:52:25.265568 992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:40731, user_credentials={real_user=slave}} blocked reactor thread for 88202.2us
I20250901 01:52:25.265751 1486 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet fa7fb88ded21432a8757f37ec3db5e71 (table default.Table_22d8d38d0b624912a0d8968fb764420e [id=808a8849c99e485b8be7cd17397874ec]). Expected version 1 got 0
W20250901 01:52:25.266376 992 leader_election.cc:336] T 60285f2ba037433999f5455fb9cc7acd P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:25.266640 992 leader_election.cc:304] T 60285f2ba037433999f5455fb9cc7acd P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:25.270850 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet 831e42de95854541a49ae6b3e039d3e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:16 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:25.272518 1486 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 410d44e5f1974ed69e5dee65e98856a4 (table default.Table_bec114a616504798bede9a9066e57192 [id=ae79f2c40fd7483ea400a36cf3c48888]). Expected version 4 got 0
I20250901 01:52:25.272706 885 tablet_service.cc:1515] Processing DeleteTablet for tablet e1c8fe96a6ba4edba4f09c00f7add75e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.273057 1486 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 1495de70f0844f82a0eea99c059f076b (table default.Table_02e911ce389e4ad581c0a2f976f99c4e [id=70a5ce71d16b47d4bc6dae97be4548e7]). Expected version 1 got 0
I20250901 01:52:25.273417 1486 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet d10c193452954c198bba686169b8025b (table default.Table_03fb48eaab9b42858ab7cefb722e43a8 [id=6c0576410d7340d6aa493a747e95565b]). Expected version 1 got 0
I20250901 01:52:25.273463 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 46bfbdff3f904715a8c24907b20fcab0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.273500 902 tablet_service.cc:1515] Processing DeleteTablet for tablet 20f7bfab18134e079e91c52c89468cc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:17 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.273767 904 tablet_service.cc:1515] Processing DeleteTablet for tablet 50e3da53433f4c14b57db8a733fa53bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:18 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.272526 1872 raft_consensus.cc:491] T 49d30da95b14400b845270122487a2a0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:25.274434 1872 raft_consensus.cc:513] T 49d30da95b14400b845270122487a2a0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:25.275192 1872 leader_election.cc:290] T 49d30da95b14400b845270122487a2a0 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:25.275223 2113 tablet_replica.cc:331] T 4d27fb61943c45d4aca67689cc3a9ff3 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:25.275235 904 tablet_service.cc:1515] Processing DeleteTablet for tablet 4855448edb4b40c184838aa7f81c112b with delete_type TABLET_DATA_DELETED (replaced by tablet 5e2472ee5d6e4d3bbf1f24027ba531ca) from {username='slave'} at 127.0.0.1:34472
W20250901 01:52:25.274046 992 consensus_peers.cc:489] T a54af99bc10045e8b0b6868e5aed98b4 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:25.275344 885 tablet_service.cc:1515] Processing DeleteTablet for tablet c49d1143d54b4c319739488906b0e271 with delete_type TABLET_DATA_DELETED (replaced by tablet 69f61ec283484f9d8b3eebcea27f741b) from {username='slave'} at 127.0.0.1:34472
W20250901 01:52:25.275471 992 consensus_peers.cc:489] T c49d1143d54b4c319739488906b0e271 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250901 01:52:25.275614 992 leader_election.cc:336] T 25ce0fe8c9544f828694184d19d13df9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:25.275480 2113 raft_consensus.cc:2241] T 4d27fb61943c45d4aca67689cc3a9ff3 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:25.275983 2113 raft_consensus.cc:2270] T 4d27fb61943c45d4aca67689cc3a9ff3 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250901 01:52:25.276000 859 leader_election.cc:343] T 1a60c10e42104502ac2d3b80b8126ceb P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:25.276136 859 leader_election.cc:304] T 1a60c10e42104502ac2d3b80b8126ceb P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:25.276531 2113 ts_tablet_manager.cc:1905] T 4d27fb61943c45d4aca67689cc3a9ff3 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:25.274221 2210 raft_consensus.cc:2747] T 60285f2ba037433999f5455fb9cc7acd P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:25.278220 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20f7bfab18134e079e91c52c89468cc3" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:25.279853 1485 master_service.cc:948] ReplaceTablet: received request to replace tablet e249474dcbf642189560c91ec3a99fdf from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:25.280773 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 75e7a90815e449c39f386901a504143c (table default.Table_ca246ea8f871412d949125b3714d8b67 [id=85d86bb68a4b43c4a8d08dd9ce12a4fc]) successfully deleted
W20250901 01:52:25.281138 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 4902062977ea42149e3466e999117322 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4902062977ea42149e3466e999117322 already in progress: creating tablet
I20250901 01:52:25.278358 1150 raft_consensus.cc:357] T 5afae316ff164413be0d25d4bc0f58dc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:25.281363 1486 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 435b1cca5a7d4d49959a310cb250b2a0 (table default.Table_dc9c5d18c7ec478592a76dbd76424b88 [id=126b298fef6c4653952d672adff1db7e]). Expected version 1 got 0
W20250901 01:52:25.281411 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 7a607fac57994001b2304e5d81c9decc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7a607fac57994001b2304e5d81c9decc already in progress: creating tablet
I20250901 01:52:25.281607 1486 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet ed1b41340ce742a88c3e4fd85cd25ca6 (table default.Table_139f6d174d794a6e9a618e90587d0c0d [id=0baa4993b72e4a8d94ee3059c9d2db6c]). Expected version 1 got 0
W20250901 01:52:25.281639 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 48a211e174054866b030cdfb0cf947f4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 48a211e174054866b030cdfb0cf947f4 already in progress: creating tablet
I20250901 01:52:25.281746 1150 raft_consensus.cc:383] T 5afae316ff164413be0d25d4bc0f58dc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:25.281778 898 tablet_service.cc:1515] Processing DeleteTablet for tablet f8a5886ddb644a86b76df44aab309057 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:14 UTC) from {username='slave'} at 127.0.0.1:34472
W20250901 01:52:25.281863 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet afc002ae2486476b9208e1cf8215157c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet afc002ae2486476b9208e1cf8215157c already in progress: creating tablet
W20250901 01:52:25.282097 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet b81a12b38ac64efabd44aeb192d181a0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b81a12b38ac64efabd44aeb192d181a0 already in progress: creating tablet
W20250901 01:52:25.282418 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 1a60c10e42104502ac2d3b80b8126ceb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1a60c10e42104502ac2d3b80b8126ceb already in progress: creating tablet
I20250901 01:52:25.282588 1486 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet c6f7c525f59b49938b697de98d4b5835 (table default.Table_db61e055dbd7450898e82c4c0c4763be [id=aabca67f6f754820977ac9c4c20f3e4f]). Expected version 1 got 0
I20250901 01:52:25.283844 1486 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 7954393c8c1c4457ab8bb723c03a9c8b (table default.Table_f44c609420054cebad09692be9aa6ecd [id=543f2074598d4c29972e564be1123fca]). Expected version 2 got 0
I20250901 01:52:25.284193 1486 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 6c0abbfc8d904dce989ce51961e4237d (table default.Table_3f9422b08b0349b9a5466218f0fe1b06 [id=f0dfb3811e744bd4ba6d59195cebde6f]). Expected version 2 got 0
I20250901 01:52:25.284602 1486 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet f9b293bf6a17453b9ea9892a6970d0b0 (table default.Table_4158d47f631042489d14100215dce98a [id=b67bfe518a094e0781ef931524b55990]). Expected version 1 got 0
I20250901 01:52:25.285158 1486 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet d0c3c363dda444519a17dca63ad9caa7 (table default.Table_0b4ac0a1caa5472d909448e41318e679 [id=8c1b263824ec4c15a5ec424364c78895]). Expected version 2 got 0
I20250901 01:52:25.285532 1486 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet e6c10426f60f43e8bde69d9b1b9c8eb6 (table default.Table_eecfab7239cc40c5a2f1fb6ac6d35dc9 [id=d3dbeb4258424133844e20b181ee44e4]). Expected version 1 got 0
I20250901 01:52:25.285826 1486 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 615b2306fca84003b264da9d5f95ddab (table default.Table_302f6e7cf97d46b9a621eb852e6aefe0 [id=cecdd8d6ca6d445c9fd82decd2a95366]). Expected version 2 got 0
I20250901 01:52:25.286144 1486 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet fecf7d538b4c4c619182d5eccb44dd8b (table default.Table_1b5729d9768546fe99eaafc4e2604f42 [id=02d0056abe2146ac9af81ebf4447b99d]). Expected version 1 got 0
I20250901 01:52:25.286545 1486 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 629db80a8ab444e582628578155638fd (table default.Table_f0e56dba1e8a4ed49a760e204c1ff96e [id=c3865e058daa4cea9a89f5046fca712a]). Expected version 1 got 0
I20250901 01:52:25.286820 1486 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 75a703c8e20a4a2fa0ab6ec58d41ef92 (table default.Table_2f09538b6fbe401f909105decb56e1cb [id=a90d1f10531042c3bb7a196b8be61c5f]). Expected version 1 got 0
I20250901 01:52:25.287257 1486 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 06d5eeb70e664f73bf643922d445bc7a (table default.Table_33c84edd575e4969b264a684accb22e2 [id=86e2bc0201e04503a1f60452605eaf27]). Expected version 1 got 0
I20250901 01:52:25.287530 1486 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 89d40f0331114490b51de3ba6d773571 (table default.Table_4dacb3b1d2e94e42ac8937eb6eb58375 [id=8ee0aa1ba24042dd8ded004644913b9c]). Expected version 1 got 0
I20250901 01:52:25.287873 1486 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet ae21b9ae2e3b472eb67fb5f96388a50f (table default.Table_046289f393ff432aae0b033bdec1fb17 [id=db57b18a9ce24e068b362f5fbe940531]). Expected version 1 got 0
I20250901 01:52:25.289649 917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f4fd117c395e41c2ab679c684908c5b7" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:25.289232 1486 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 3618a3eb4f174c43805ce7dc0e5315cb (table default.Table_83f2c0acedbc4c6982caf5313646e90b [id=31a2e50c32644321825da40176127015]). Expected version 1 got 0
I20250901 01:52:25.290040 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 34042ea5c1b645969c1de41fe587c343 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:23 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.290220 1150 raft_consensus.cc:738] T 5afae316ff164413be0d25d4bc0f58dc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:25.282002 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9a614090d0c042c0a57c01ecdb796d31"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.290840 1150 consensus_queue.cc:260] T 5afae316ff164413be0d25d4bc0f58dc P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:25.291237 887 tablet_service.cc:1515] Processing DeleteTablet for tablet b6daeff0070a4db08e90020752d21c29 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:14 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.290846 897 tablet_service.cc:1515] Processing DeleteTablet for tablet 121803e79ab64ec0b948b448646b9d0a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:14 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.291376 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_0b165485b3e9427fae67435e343afd24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.291741 1150 ts_tablet_manager.cc:1428] T 5afae316ff164413be0d25d4bc0f58dc P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.074s user 0.002s sys 0.000s
I20250901 01:52:25.292055 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8dcd9a2d7f164e64935912461b9541d9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.292455 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 0cb3262973934c8689b851adad60afe0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:16 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.292563 893 tablet_service.cc:1515] Processing DeleteTablet for tablet b7b802f976f3471faaaecc964ec87e48 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:16 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.292771 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 2240302d9178445d8700e35f5ea274f7 with delete_type TABLET_DATA_DELETED (replaced by tablet b13afacd37474781877b9f2cd20b313a) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.292853 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_53e1204b94ba469593e7fee76f5ae58a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.293221 1486 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet f80f13db501c42c7950499df8fcb6fc8 (table default.Table_ce86974fd8974de9a5a737954fac2ad7 [id=ff772d20b287476f94716379b86d2a71]). Expected version 1 got 0
I20250901 01:52:25.293521 1486 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet da5f69c5b0ca4257ad2ff7918ee2af2e (table default.Table_ecb9e8f638a043cc87c38afbebce250f [id=61f1e8a842844ed899e5913c76f362ff]). Expected version 1 got 0
I20250901 01:52:25.295374 1150 tablet_bootstrap.cc:492] T 3239e58531d445fa80165cf678bc277c P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:25.297513 1486 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 508406601c994e42a5c5b1f665e74957 (table default.Table_345b7f9fe8214aa6878f20a166125364 [id=4aabc328f6374ec5b1dea26f778208b2]). Expected version 1 got 0
I20250901 01:52:25.298600 1150 tablet_bootstrap.cc:654] T 3239e58531d445fa80165cf678bc277c P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:25.298943 1486 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 36e10328c34242ebb4dbd8f294162696 (table default.Table_8445fcd774874cadaf4acc3581eee417 [id=958e51a6bbee47638e504e1a44094737]). Expected version 1 got 0
I20250901 01:52:25.299376 1486 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet d4c9aa723aee4dc4b6237cb5c3722b52 (table default.Table_5e65c5f068b84995a2d7a76fa65dda0f [id=b9d81ddd38fe47838877b010be2b40d0]). Expected version 1 got 0
I20250901 01:52:25.302971 1486 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet b009cc95c819431ea9827a3e453b8da3 (table default.Table_203547c9458e43b3b48ea051d56892c6 [id=cc26c89db2484a6fad919231116285f2]). Expected version 1 got 0
I20250901 01:52:25.304394 1486 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 93bcac1fd95d44dc927abfca81692921 (table default.Table_b8f8ee74648c406eaf589d3ab80934db [id=b8f69313439543be9c11eea88dfa69c5]). Expected version 1 got 0
I20250901 01:52:25.305084 1486 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet e9f1f892658b4736859ff507536920a1 (table default.Table_8825dd28afbe464193eb1b668cc9326a [id=9aa62cb634314a0a849612c54c1220d9]). Expected version 1 got 0
I20250901 01:52:25.305672 1486 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 7f46e9ddadaf4ff986d8c991178daa0c (table default.Table_9978497447394e5694ae928412634ecb [id=ea928501fe954cd0b81d38add4ef7128]). Expected version 2 got 0
I20250901 01:52:25.306116 1486 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 5f5396045979450193d461ba1334a5c9 (table default.Table_a89cc8e48563460baabc34c821ceae35 [id=9c3930e9fcfe4b49ba4bea67a2b0ad07]). Expected version 1 got 0
I20250901 01:52:25.307514 1486 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 3dae00aa38204ee3ac77325eb334be42 (table default.Table_faf7ef12324f454396e5ae64d64ebe51 [id=5812902df9ea406ba59ff0ff23f130b6]). Expected version 1 got 0
I20250901 01:52:25.307806 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad1a6be9fcc045d3bb4e71a2f6bd5ccd" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:25.293545 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d498e6fa598c42f68052fa0587552fb4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.308476 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "615b2306fca84003b264da9d5f95ddab" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:25.308965 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4cde5f4f02044b7bbd7d43b49aac4ae8" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:25.309413 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0dbdcb76eeba4fefa56e7835c23c0316" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:25.311532 923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4855448edb4b40c184838aa7f81c112b" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:25.311754 923 raft_consensus.cc:2466] T 4855448edb4b40c184838aa7f81c112b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:25.313632 1151 tablet_bootstrap.cc:492] T a289a66b2dfa44dc8c8547ff83c1716e P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:25.314465 1151 ts_tablet_manager.cc:1397] T a289a66b2dfa44dc8c8547ff83c1716e P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.178s user 0.006s sys 0.010s
I20250901 01:52:25.315948 1151 raft_consensus.cc:357] T a289a66b2dfa44dc8c8547ff83c1716e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:25.317471 1151 raft_consensus.cc:383] T a289a66b2dfa44dc8c8547ff83c1716e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:25.317642 1151 raft_consensus.cc:738] T a289a66b2dfa44dc8c8547ff83c1716e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:25.318002 1151 consensus_queue.cc:260] T a289a66b2dfa44dc8c8547ff83c1716e P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:25.308959 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_db2d7694cd1947008397c638b9c7da76"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.320585 921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa424ac6ef1441e6b1cc2867b1c48186" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:25.320845 921 raft_consensus.cc:3058] T aa424ac6ef1441e6b1cc2867b1c48186 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:25.322425 1151 ts_tablet_manager.cc:1428] T a289a66b2dfa44dc8c8547ff83c1716e P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250901 01:52:25.319904 1486 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 33f637f6a8134215bb1995ec38440073 (table default.Table_2ece91972d494680a8ad00273c10dee4 [id=557afe29976d40f6bfb6ae4ca348f887]). Expected version 1 got 0
W20250901 01:52:25.322661 992 consensus_peers.cc:489] T 121803e79ab64ec0b948b448646b9d0a P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250901 01:52:25.323038 992 leader_election.cc:336] T dd694cfa09564d3da5d279c0861a4c32 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:25.323223 1486 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet f62d96c37cca4143a4b098b7c69979e6 (table default.Table_0877ac8202a24e9da650a589bc6dbfcc [id=1f65b3e09c9748f189ecb2b56c5d1132]). Expected version 1 got 0
I20250901 01:52:25.323287 1151 tablet_bootstrap.cc:492] T 46bfbdff3f904715a8c24907b20fcab0 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:25.323412 1486 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 5adf6b492640417ca071c94702b48cb9 (table default.Table_589c302e02474fc3bc5d3a290a8bd54a [id=5d14706774d345c48ca3575feabb17b0]). Expected version 2 got 0
I20250901 01:52:25.323773 1486 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 61ddcf86c9f34726939cc116434f0822 (table default.Table_63d0f2ead9d946d39ce40162ebd769d6 [id=f9dcf4937b1143248bfa9d88053fce17]). Expected version 3 got 0
W20250901 01:52:25.323447 992 leader_election.cc:336] T 64fb46b942ff4e4c8749986ecc3dfe65 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:25.324039 992 consensus_peers.cc:489] T 822382a1781b4304bce29355ce83730e P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:25.324038 1486 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 3a8231590ea643b0a1f83eec4d9a8191 (table default.Table_a4ebd8ab87c842f082da5cd04d1bddac [id=fd6d8e92d6e64a89a7d38aa761376955]). Expected version 1 got 0
W20250901 01:52:25.324247 992 leader_election.cc:336] T 4d27fb61943c45d4aca67689cc3a9ff3 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:25.324347 1486 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 8ca5cf4e99014c0d84a3acad4ef713af (table default.Table_ed6fd284be6f4ae3a008e21e7ce17930 [id=303e76c918bc4c39aa47a4d24849a358]). Expected version 1 got 0
I20250901 01:52:25.324898 1486 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet ad45c90ba0da43c9860745e534664a98 (table default.Table_1cd3f3fb6b244b3f8796bfca9f7622c8 [id=6fd739b0e725424ca844dbb652ea8028]). Expected version 2 got 0
I20250901 01:52:25.325358 1486 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 72ba689f9230411995dc96e2b45601df (table default.Table_604f2c5c58d74154bacf2a065a467f32 [id=351a05134f4948039c920cafbe050914]). Expected version 1 got 0
I20250901 01:52:25.325565 2118 ts_tablet_manager.cc:1918] T 4410da9e95454ecf95faf80221b0a71c P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 01:52:25.325717 2118 log.cc:1199] T 4410da9e95454ecf95faf80221b0a71c P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/4410da9e95454ecf95faf80221b0a71c
I20250901 01:52:25.325734 1486 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 3c1fd0820d8a4995abe027f685da7f7b (table default.Table_bcd502e1360b48b1953fa97eadce7006 [id=b3914f33e856466e8479d0a2263ec56f]). Expected version 1 got 0
I20250901 01:52:25.326102 992 leader_election.cc:304] T 4d27fb61943c45d4aca67689cc3a9ff3 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:25.326185 2118 ts_tablet_manager.cc:1939] T 4410da9e95454ecf95faf80221b0a71c P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
W20250901 01:52:25.326331 992 raft_consensus.cc:2684] T 4d27fb61943c45d4aca67689cc3a9ff3 P fd27ab6665304aaea53adbd102dd4f58: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250901 01:52:25.326566 992 leader_election.cc:336] T aa424ac6ef1441e6b1cc2867b1c48186 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:25.326777 1486 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet ddbc15685fca4c708329420109ad0c57 (table default.Table_942175dadf4b496195b2fa53f39f6ddc [id=2bd332b452f84f25af4225af8e6c6740]). Expected version 1 got 0
W20250901 01:52:25.326828 992 leader_election.cc:336] T 12c04e0c1f804179ae23509cdb24d395 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:25.326948 992 leader_election.cc:304] T 12c04e0c1f804179ae23509cdb24d395 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:25.327142 1486 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 49aaaec611784e6bac131efd58dbca51 (table default.Table_95e1338a49a04f22b679e363e253262e [id=e24b91b4ac904ef0a8c4725f70bced2f]). Expected version 1 got 0
I20250901 01:52:25.327391 1486 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet e87d1281f35648b0b115b67c95e244e7 (table default.Table_f840789a58984799bd8057128f6ea439 [id=4453959e273c473fadd737f56f7db3dd]). Expected version 1 got 0
I20250901 01:52:25.327906 1486 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet b2e8b2cd10154de08ddd30c214592a4a (table default.Table_4a412a59b6db421eb0187ffd02ecf541 [id=fd72e46c84634b6c9ea74970480ecc62]). Expected version 4 got 0
I20250901 01:52:25.328155 1486 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet e1d18e999b5e42b1873631e0f1445081 (table default.Table_fd17a2ad225546aa821489ce94a788a2 [id=033c76e0598342c495ce2be42a0df5dd]). Expected version 1 got 0
I20250901 01:52:25.328361 1486 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 616ff34e8d314e0aa0e11ce60be11789 (table default.Table_0451d0afa26f4d5f8d4214a946eea04f [id=5ee13d46e06a4fa59b1070547573e523]). Expected version 1 got 0
I20250901 01:52:25.328722 1486 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 42c1156e8dd84c6fa1396b46de13eb6f (table default.Table_b5d681f51fc84395bbabe1e5b4cfb358 [id=40d05d90b0354334840ad17764f5b6bb]). Expected version 1 got 0
I20250901 01:52:25.329290 1486 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 2599f98abfdc4871b573edfc2ad97d22 (table default.Table_0e63d21acfcb4c028c9ea6406b6a04e4 [id=81323e6fbe424655b8e185f52657fadc]). Expected version 1 got 0
I20250901 01:52:25.329533 1486 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 081e2630e284498aa819f85bf8326315 (table default.Table_9e7ca61fcc324a8984bd1e470162c61f [id=8199118617cb4707bc3ae9b1363750f7]). Expected version 2 got 0
I20250901 01:52:25.329901 1486 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 8ed6616ce54548ba83ba299c4ecf3c38 (table default.Table_1bdd9041d74045cab6f23de11bf4dd09 [id=946ec3e53dcd4de4b8501b3e87e99072]). Expected version 2 got 0
I20250901 01:52:25.331121 1872 raft_consensus.cc:2747] T 12c04e0c1f804179ae23509cdb24d395 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:25.326006 1151 tablet_bootstrap.cc:654] T 46bfbdff3f904715a8c24907b20fcab0 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:25.335302 1872 raft_consensus.cc:491] T 3fda93767e494a4684978c8522b829dd P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:25.335475 1872 raft_consensus.cc:513] T 3fda93767e494a4684978c8522b829dd P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:25.336066 1872 leader_election.cc:290] T 3fda93767e494a4684978c8522b829dd P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:25.336525 992 leader_election.cc:304] T 616ff34e8d314e0aa0e11ce60be11789 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:25.336839 1872 raft_consensus.cc:2802] T 616ff34e8d314e0aa0e11ce60be11789 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:25.336938 1872 raft_consensus.cc:491] T 616ff34e8d314e0aa0e11ce60be11789 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:25.337049 1872 raft_consensus.cc:3058] T 616ff34e8d314e0aa0e11ce60be11789 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
W20250901 01:52:25.340080 992 leader_election.cc:343] T dd694cfa09564d3da5d279c0861a4c32 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: dd694cfa09564d3da5d279c0861a4c32
I20250901 01:52:25.340302 992 leader_election.cc:304] T dd694cfa09564d3da5d279c0861a4c32 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:25.340866 992 leader_election.cc:304] T 64fb46b942ff4e4c8749986ecc3dfe65 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:25.341619 1993 raft_consensus.cc:2802] T 64fb46b942ff4e4c8749986ecc3dfe65 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:25.341750 1993 raft_consensus.cc:491] T 64fb46b942ff4e4c8749986ecc3dfe65 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:25.341837 1993 raft_consensus.cc:3058] T 64fb46b942ff4e4c8749986ecc3dfe65 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:25.341632 2210 raft_consensus.cc:2747] T dd694cfa09564d3da5d279c0861a4c32 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:25.351855 891 tablet_service.cc:1515] Processing DeleteTablet for tablet af2c4bcbb00741f789edee109252d8cb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:18 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.352063 886 tablet_service.cc:1515] Processing DeleteTablet for tablet ded33e0950b544509af11739353c620d with delete_type TABLET_DATA_DELETED (replaced by tablet cc3b8736b9094d7e892c80da4d5b20f6) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.352165 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 7f64ebd080944190be2e64d02d70a8a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:15 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.352182 897 tablet_service.cc:1515] Processing DeleteTablet for tablet 25ce0fe8c9544f828694184d19d13df9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:16 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.352528 886 tablet_service.cc:1515] Processing DeleteTablet for tablet eaa521c52a2f4408b59770bd2a05907c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.352728 897 tablet_service.cc:1515] Processing DeleteTablet for tablet c58edaa76ebf47dfb1d9d5fab6de7dfa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:17 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.352967 886 tablet_service.cc:1515] Processing DeleteTablet for tablet e249474dcbf642189560c91ec3a99fdf with delete_type TABLET_DATA_DELETED (replaced by tablet 7f46e9ddadaf4ff986d8c991178daa0c) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.353024 897 tablet_service.cc:1515] Processing DeleteTablet for tablet ad1a6be9fcc045d3bb4e71a2f6bd5ccd with delete_type TABLET_DATA_DELETED (replaced by tablet ae9a4ad4fdfb4771ba23a91f7007447e) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.354557 1981 raft_consensus.cc:491] T b81a12b38ac64efabd44aeb192d181a0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:25.354763 1981 raft_consensus.cc:513] T b81a12b38ac64efabd44aeb192d181a0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:25.355538 1981 leader_election.cc:290] T b81a12b38ac64efabd44aeb192d181a0 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:25.362516 2113 ts_tablet_manager.cc:1918] T 4d27fb61943c45d4aca67689cc3a9ff3 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:25.362659 2113 log.cc:1199] T 4d27fb61943c45d4aca67689cc3a9ff3 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/4d27fb61943c45d4aca67689cc3a9ff3
I20250901 01:52:25.364001 2113 ts_tablet_manager.cc:1939] T 4d27fb61943c45d4aca67689cc3a9ff3 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:25.365197 1872 raft_consensus.cc:513] T 616ff34e8d314e0aa0e11ce60be11789 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:25.366052 1872 leader_election.cc:290] T 616ff34e8d314e0aa0e11ce60be11789 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:25.370900 2210 consensus_queue.cc:1035] T 579b9cdf819f4609a532e96bec093fd5 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:25.371011 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_137502bd1e934fd79621dca82a8fe7d2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.371414 2171 raft_consensus.cc:491] T 9d886fa63f8f4996b087997b542e8bb1 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:25.371582 2171 raft_consensus.cc:513] T 9d886fa63f8f4996b087997b542e8bb1 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:25.372072 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_92a494ae6dec489d9c5ae1380117b2de"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.372161 2171 leader_election.cc:290] T 9d886fa63f8f4996b087997b542e8bb1 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:25.372756 1485 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_99afdac28a4e4ac9ae93b4fbbe704a4d" } modify_external_catalogs: true
I20250901 01:52:25.372891 1485 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_99afdac28a4e4ac9ae93b4fbbe704a4d" } modify_external_catalogs: true
I20250901 01:52:25.374531 2210 consensus_queue.cc:1035] T af2c4bcbb00741f789edee109252d8cb P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:25.376148 2210 consensus_queue.cc:1035] T 7f64ebd080944190be2e64d02d70a8a7 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250901 01:52:25.376683 992 leader_election.cc:343] T 824229183e024feaa975c7f0387c9c1a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 824229183e024feaa975c7f0387c9c1a
W20250901 01:52:25.377161 859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:39447, user_credentials={real_user=slave}} blocked reactor thread for 56396.8us
I20250901 01:52:25.377744 2123 raft_consensus.cc:491] T d10c193452954c198bba686169b8025b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:25.377875 2123 raft_consensus.cc:513] T d10c193452954c198bba686169b8025b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:25.378600 2123 leader_election.cc:290] T d10c193452954c198bba686169b8025b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:25.379416 859 leader_election.cc:343] T ad1a6be9fcc045d3bb4e71a2f6bd5ccd P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:25.379526 859 leader_election.cc:304] T ad1a6be9fcc045d3bb4e71a2f6bd5ccd P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:25.379712 2123 raft_consensus.cc:2747] T ad1a6be9fcc045d3bb4e71a2f6bd5ccd P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:25.380081 859 leader_election.cc:343] T 615b2306fca84003b264da9d5f95ddab P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:25.380172 859 leader_election.cc:304] T 615b2306fca84003b264da9d5f95ddab P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:25.380352 916 raft_consensus.cc:2466] T 20f7bfab18134e079e91c52c89468cc3 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:25.380580 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 6cf09e8f299548dba2cce16d7b565bad with delete_type TABLET_DATA_DELETED (replaced by tablet 0e4009963cfe42d6ba164fc828897f09) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.380740 901 tablet_service.cc:1515] Processing DeleteTablet for tablet d58f69fad01f403eab74a47d3695ed83 with delete_type TABLET_DATA_DELETED (replaced by tablet 74b709cb5ec64006b2f12bead8217d80) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.380865 917 raft_consensus.cc:3058] T f4fd117c395e41c2ab679c684908c5b7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:25.380877 900 tablet_service.cc:1515] Processing DeleteTablet for tablet eac50cbae691416e903cb2cbc907f96d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:23 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.381237 2253 raft_consensus.cc:2747] T 1a60c10e42104502ac2d3b80b8126ceb P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:25.385210 2251 tablet.cc:1722] T 72ba689f9230411995dc96e2b45601df P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:25.386317 1872 consensus_queue.cc:1035] T e1d18e999b5e42b1873631e0f1445081 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250901 01:52:25.392371 992 leader_election.cc:343] T 5a8c21aee9f84216b6f90538a5bf4c17 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 5a8c21aee9f84216b6f90538a5bf4c17
I20250901 01:52:25.392765 992 leader_election.cc:304] T 4855448edb4b40c184838aa7f81c112b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:25.393461 1872 raft_consensus.cc:2802] T 4855448edb4b40c184838aa7f81c112b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:25.393604 1872 raft_consensus.cc:491] T 4855448edb4b40c184838aa7f81c112b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:25.393765 1872 raft_consensus.cc:3058] T 4855448edb4b40c184838aa7f81c112b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:25.393884 2113 tablet_replica.cc:331] T 3c12d6a70ac24dce82e851782e6fe5aa P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:25.394192 2113 raft_consensus.cc:2241] T 3c12d6a70ac24dce82e851782e6fe5aa P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:25.394490 2113 pending_rounds.cc:70] T 3c12d6a70ac24dce82e851782e6fe5aa P fd27ab6665304aaea53adbd102dd4f58: Trying to abort 1 pending ops.
I20250901 01:52:25.394594 2113 pending_rounds.cc:77] T 3c12d6a70ac24dce82e851782e6fe5aa P fd27ab6665304aaea53adbd102dd4f58: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7195408566602207232 op_type: NO_OP noop_request { }
I20250901 01:52:25.394763 2113 raft_consensus.cc:2887] T 3c12d6a70ac24dce82e851782e6fe5aa P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250901 01:52:25.394923 2113 raft_consensus.cc:2270] T 3c12d6a70ac24dce82e851782e6fe5aa P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:25.395560 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 4d27fb61943c45d4aca67689cc3a9ff3 (table default.Table_0e0c69a04fea4f5fab174e11a35bf628 [id=78de546347e74cc89be668054e292f04]) successfully deleted
I20250901 01:52:25.396520 1151 tablet_bootstrap.cc:492] T 46bfbdff3f904715a8c24907b20fcab0 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:25.396770 1151 ts_tablet_manager.cc:1397] T 46bfbdff3f904715a8c24907b20fcab0 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.074s user 0.002s sys 0.005s
I20250901 01:52:25.399672 2276 tablet.cc:1722] T 3c1fd0820d8a4995abe027f685da7f7b P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:25.400815 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_14ef69849ba34767be0a442f5593ea68"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.402143 1151 raft_consensus.cc:357] T 46bfbdff3f904715a8c24907b20fcab0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:25.403405 992 leader_election.cc:343] T a774cc6ae3924d299127fd9ffb01ca57 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: a774cc6ae3924d299127fd9ffb01ca57
I20250901 01:52:25.403431 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_240f964095dd432fa4812e4b306fc733"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.405712 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_0ab72f31cdfa479882ea88e2b2c034da"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.408631 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_721ccbfac4d14b2aa27a0b85703a1029"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.408872 2277 tablet.cc:1722] T ddbc15685fca4c708329420109ad0c57 P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:25.409103 2123 raft_consensus.cc:2747] T 615b2306fca84003b264da9d5f95ddab P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:25.408916 992 leader_election.cc:343] T 9473f0ef978c4bfaa41fb7b0729e8863 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 9473f0ef978c4bfaa41fb7b0729e8863
W20250901 01:52:25.409310 859 leader_election.cc:343] T 4cde5f4f02044b7bbd7d43b49aac4ae8 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:25.409415 859 leader_election.cc:304] T 4cde5f4f02044b7bbd7d43b49aac4ae8 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:25.409019 1151 raft_consensus.cc:383] T 46bfbdff3f904715a8c24907b20fcab0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:25.409626 2123 raft_consensus.cc:2747] T 4cde5f4f02044b7bbd7d43b49aac4ae8 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:25.409801 859 leader_election.cc:343] T 0dbdcb76eeba4fefa56e7835c23c0316 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:25.409893 859 leader_election.cc:304] T 0dbdcb76eeba4fefa56e7835c23c0316 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:25.410086 2253 raft_consensus.cc:2747] T 0dbdcb76eeba4fefa56e7835c23c0316 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:25.410058 1151 raft_consensus.cc:738] T 46bfbdff3f904715a8c24907b20fcab0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
W20250901 01:52:25.410732 992 leader_election.cc:343] T c9a4c80bec1d41458974a0c449aecaf7 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: c9a4c80bec1d41458974a0c449aecaf7
I20250901 01:52:25.411907 1151 consensus_queue.cc:260] T 46bfbdff3f904715a8c24907b20fcab0 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:25.412532 992 leader_election.cc:343] T 438cd35ff3f74c49b908d072560077db P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 438cd35ff3f74c49b908d072560077db
W20250901 01:52:25.414341 992 leader_election.cc:343] T b633b461dbf34e13bc6e1cc08e6d20c1 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: b633b461dbf34e13bc6e1cc08e6d20c1
W20250901 01:52:25.415436 992 leader_election.cc:343] T b541a6e14f114ab29a864cca11806316 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: b541a6e14f114ab29a864cca11806316
W20250901 01:52:25.415488 858 leader_election.cc:336] T 48a211e174054866b030cdfb0cf947f4 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:25.416628 2253 raft_consensus.cc:491] T 5afae316ff164413be0d25d4bc0f58dc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:25.417101 2253 raft_consensus.cc:513] T 5afae316ff164413be0d25d4bc0f58dc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:25.416033 1151 ts_tablet_manager.cc:1428] T 46bfbdff3f904715a8c24907b20fcab0 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20250901 01:52:25.419353 2253 leader_election.cc:290] T 5afae316ff164413be0d25d4bc0f58dc P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:25.419452 2170 consensus_queue.cc:1035] T 0cb3262973934c8689b851adad60afe0 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250901 01:52:25.420364 992 leader_election.cc:343] T 49d30da95b14400b845270122487a2a0 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 49d30da95b14400b845270122487a2a0
I20250901 01:52:25.421087 1151 tablet_bootstrap.cc:492] T 6741c56499b44f648fb032c8cf50205b P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:25.422014 2170 consensus_queue.cc:1035] T b7b802f976f3471faaaecc964ec87e48 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:25.417137 2271 tablet.cc:1722] T fcb7b5b7d63d4f0b906fea97cb548de2 P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
W20250901 01:52:25.426286 992 consensus_peers.cc:489] T 4410da9e95454ecf95faf80221b0a71c P fd27ab6665304aaea53adbd102dd4f58 -> Peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Couldn't send request to peer 00c282f779c1477d920f24dc0dc16628. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:25.427819 2170 consensus_queue.cc:1035] T 8dd587304e6342edbe96c9897b40e5c9 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:25.429121 992 leader_election.cc:304] T 20f7bfab18134e079e91c52c89468cc3 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:25.430509 2170 raft_consensus.cc:2802] T 20f7bfab18134e079e91c52c89468cc3 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:25.430609 2170 raft_consensus.cc:491] T 20f7bfab18134e079e91c52c89468cc3 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:25.430972 2170 raft_consensus.cc:3058] T 20f7bfab18134e079e91c52c89468cc3 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:25.434991 1151 tablet_bootstrap.cc:654] T 6741c56499b44f648fb032c8cf50205b P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:25.436825 992 leader_election.cc:336] T 438cd35ff3f74c49b908d072560077db P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:25.436957 992 leader_election.cc:304] T 438cd35ff3f74c49b908d072560077db P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:25.437203 2210 raft_consensus.cc:2747] T 438cd35ff3f74c49b908d072560077db P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:25.437309 992 leader_election.cc:336] T b633b461dbf34e13bc6e1cc08e6d20c1 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:25.437422 992 leader_election.cc:304] T b633b461dbf34e13bc6e1cc08e6d20c1 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:25.437664 2210 raft_consensus.cc:2747] T b633b461dbf34e13bc6e1cc08e6d20c1 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:25.437737 992 leader_election.cc:336] T c9a4c80bec1d41458974a0c449aecaf7 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:25.437832 992 leader_election.cc:304] T c9a4c80bec1d41458974a0c449aecaf7 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:25.438036 2210 raft_consensus.cc:2747] T c9a4c80bec1d41458974a0c449aecaf7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:25.438122 992 leader_election.cc:336] T 9473f0ef978c4bfaa41fb7b0729e8863 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:25.438217 992 leader_election.cc:304] T 9473f0ef978c4bfaa41fb7b0729e8863 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:25.438458 2210 raft_consensus.cc:2747] T 9473f0ef978c4bfaa41fb7b0729e8863 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:25.438534 992 leader_election.cc:336] T b541a6e14f114ab29a864cca11806316 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:25.438632 992 leader_election.cc:304] T b541a6e14f114ab29a864cca11806316 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:25.438840 2210 raft_consensus.cc:2747] T b541a6e14f114ab29a864cca11806316 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:25.438925 992 leader_election.cc:336] T a774cc6ae3924d299127fd9ffb01ca57 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:25.439025 992 leader_election.cc:304] T a774cc6ae3924d299127fd9ffb01ca57 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:25.439244 2210 raft_consensus.cc:2747] T a774cc6ae3924d299127fd9ffb01ca57 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:25.439320 992 leader_election.cc:336] T aa424ac6ef1441e6b1cc2867b1c48186 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:25.439453 992 consensus_peers.cc:489] T 49aaaec611784e6bac131efd58dbca51 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250901 01:52:25.439568 992 leader_election.cc:336] T 4855448edb4b40c184838aa7f81c112b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:25.439771 992 leader_election.cc:336] T 49d30da95b14400b845270122487a2a0 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:25.441325 858 leader_election.cc:336] T b81a12b38ac64efabd44aeb192d181a0 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:25.441453 858 leader_election.cc:336] T d10c193452954c198bba686169b8025b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:25.442111 858 leader_election.cc:336] T 5afae316ff164413be0d25d4bc0f58dc P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:25.442747 1150 tablet_bootstrap.cc:492] T 3239e58531d445fa80165cf678bc277c P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:25.442870 1150 ts_tablet_manager.cc:1397] T 3239e58531d445fa80165cf678bc277c P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.151s user 0.003s sys 0.014s
I20250901 01:52:25.444025 1150 raft_consensus.cc:357] T 3239e58531d445fa80165cf678bc277c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:25.444183 1150 raft_consensus.cc:383] T 3239e58531d445fa80165cf678bc277c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:25.444228 1150 raft_consensus.cc:738] T 3239e58531d445fa80165cf678bc277c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:25.444388 1150 consensus_queue.cc:260] T 3239e58531d445fa80165cf678bc277c P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:25.444896 1150 ts_tablet_manager.cc:1428] T 3239e58531d445fa80165cf678bc277c P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250901 01:52:25.445152 1150 tablet_bootstrap.cc:492] T fa7fb88ded21432a8757f37ec3db5e71 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:25.448144 992 leader_election.cc:304] T 49d30da95b14400b845270122487a2a0 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:25.448486 2210 raft_consensus.cc:2747] T 49d30da95b14400b845270122487a2a0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:25.449174 992 leader_election.cc:336] T 616ff34e8d314e0aa0e11ce60be11789 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:25.449649 992 consensus_peers.cc:489] T 7f64ebd080944190be2e64d02d70a8a7 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:25.446139 2253 raft_consensus.cc:491] T a9b7f2e6548043a7865e25387cdd06c9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250901 01:52:25.449926 992 consensus_peers.cc:489] T 4410da9e95454ecf95faf80221b0a71c P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250901 01:52:25.449851 2253 raft_consensus.cc:513] T a9b7f2e6548043a7865e25387cdd06c9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:25.450047 992 consensus_peers.cc:489] T b7b802f976f3471faaaecc964ec87e48 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250901 01:52:25.450153 992 consensus_peers.cc:489] T fcb7b5b7d63d4f0b906fea97cb548de2 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250901 01:52:25.450263 992 consensus_peers.cc:489] T 8dd587304e6342edbe96c9897b40e5c9 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250901 01:52:25.450371 992 consensus_peers.cc:489] T af2c4bcbb00741f789edee109252d8cb P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:25.450757 2253 leader_election.cc:290] T a9b7f2e6548043a7865e25387cdd06c9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:25.450476 992 leader_election.cc:336] T 824229183e024feaa975c7f0387c9c1a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:25.451126 992 leader_election.cc:304] T 824229183e024feaa975c7f0387c9c1a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:25.451397 2210 raft_consensus.cc:2747] T 824229183e024feaa975c7f0387c9c1a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:25.451627 1993 raft_consensus.cc:513] T 64fb46b942ff4e4c8749986ecc3dfe65 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:25.452157 1993 leader_election.cc:290] T 64fb46b942ff4e4c8749986ecc3dfe65 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:25.455505 858 leader_election.cc:336] T a9b7f2e6548043a7865e25387cdd06c9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:25.456476 992 leader_election.cc:336] T f4fd117c395e41c2ab679c684908c5b7 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:25.456739 992 leader_election.cc:336] T 20f7bfab18134e079e91c52c89468cc3 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:25.456998 992 leader_election.cc:336] T 5a8c21aee9f84216b6f90538a5bf4c17 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:25.457100 992 leader_election.cc:304] T 5a8c21aee9f84216b6f90538a5bf4c17 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:25.457367 921 raft_consensus.cc:2466] T aa424ac6ef1441e6b1cc2867b1c48186 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:25.464557 1993 raft_consensus.cc:2747] T 5a8c21aee9f84216b6f90538a5bf4c17 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:25.465631 2118 tablet_replica.cc:331] T b11e8f77243e4764b96548d1132b8e5a P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:25.465844 2118 raft_consensus.cc:2241] T b11e8f77243e4764b96548d1132b8e5a P 00c282f779c1477d920f24dc0dc16628 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:25.466212 2118 raft_consensus.cc:2270] T b11e8f77243e4764b96548d1132b8e5a P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:25.466717 2118 ts_tablet_manager.cc:1905] T b11e8f77243e4764b96548d1132b8e5a P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:25.469606 921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "616ff34e8d314e0aa0e11ce60be11789" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:25.469771 921 raft_consensus.cc:3058] T 616ff34e8d314e0aa0e11ce60be11789 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:25.476943 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5a94b8aff28047e98c397548e22e2d3a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.477862 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_35b915dafca54d8b95115764c628b737"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.478613 1492 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_0877ac8202a24e9da650a589bc6dbfcc" } new_table_name: "default.Table_2ddf898959684cdc995c1351d84a9661" modify_external_catalogs: true
I20250901 01:52:25.491153 1151 tablet_bootstrap.cc:492] T 6741c56499b44f648fb032c8cf50205b P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:25.491264 2113 ts_tablet_manager.cc:1905] T 3c12d6a70ac24dce82e851782e6fe5aa P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:25.491329 1151 ts_tablet_manager.cc:1397] T 6741c56499b44f648fb032c8cf50205b P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.072s user 0.003s sys 0.002s
I20250901 01:52:25.492093 1151 raft_consensus.cc:357] T 6741c56499b44f648fb032c8cf50205b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:25.492230 1151 raft_consensus.cc:383] T 6741c56499b44f648fb032c8cf50205b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:25.492271 1151 raft_consensus.cc:738] T 6741c56499b44f648fb032c8cf50205b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:25.492456 1151 consensus_queue.cc:260] T 6741c56499b44f648fb032c8cf50205b P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:25.492940 1151 ts_tablet_manager.cc:1428] T 6741c56499b44f648fb032c8cf50205b P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250901 01:52:25.493177 1151 tablet_bootstrap.cc:492] T f80f13db501c42c7950499df8fcb6fc8 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:25.498167 1151 tablet_bootstrap.cc:654] T f80f13db501c42c7950499df8fcb6fc8 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:25.498332 1150 tablet_bootstrap.cc:654] T fa7fb88ded21432a8757f37ec3db5e71 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:25.500551 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 4410da9e95454ecf95faf80221b0a71c (table default.Table_420b0979d6fd42da9b320456f4ccc9ce [id=d2004563c4374790954327f05c305cc6]) successfully deleted
I20250901 01:52:25.515479 1150 tablet_bootstrap.cc:492] T fa7fb88ded21432a8757f37ec3db5e71 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:25.515972 1150 ts_tablet_manager.cc:1397] T fa7fb88ded21432a8757f37ec3db5e71 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.071s user 0.002s sys 0.004s
I20250901 01:52:25.516119 1993 raft_consensus.cc:491] T 42c1156e8dd84c6fa1396b46de13eb6f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:25.516253 1993 raft_consensus.cc:513] T 42c1156e8dd84c6fa1396b46de13eb6f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:25.516856 1993 leader_election.cc:290] T 42c1156e8dd84c6fa1396b46de13eb6f P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:25.517482 1150 raft_consensus.cc:357] T fa7fb88ded21432a8757f37ec3db5e71 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:25.517836 1150 raft_consensus.cc:383] T fa7fb88ded21432a8757f37ec3db5e71 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:25.517980 1150 raft_consensus.cc:738] T fa7fb88ded21432a8757f37ec3db5e71 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:25.518389 1150 consensus_queue.cc:260] T fa7fb88ded21432a8757f37ec3db5e71 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:25.518658 1151 tablet_bootstrap.cc:492] T f80f13db501c42c7950499df8fcb6fc8 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:25.518971 1151 ts_tablet_manager.cc:1397] T f80f13db501c42c7950499df8fcb6fc8 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.007s
I20250901 01:52:25.519515 1150 ts_tablet_manager.cc:1428] T fa7fb88ded21432a8757f37ec3db5e71 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250901 01:52:25.519671 1151 raft_consensus.cc:357] T f80f13db501c42c7950499df8fcb6fc8 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:25.519904 1151 raft_consensus.cc:383] T f80f13db501c42c7950499df8fcb6fc8 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:25.520005 1151 raft_consensus.cc:738] T f80f13db501c42c7950499df8fcb6fc8 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:25.520314 1151 consensus_queue.cc:260] T f80f13db501c42c7950499df8fcb6fc8 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:25.520646 1150 tablet_bootstrap.cc:492] T 4902062977ea42149e3466e999117322 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:25.521044 1151 ts_tablet_manager.cc:1428] T f80f13db501c42c7950499df8fcb6fc8 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250901 01:52:25.521395 1151 tablet_bootstrap.cc:492] T 3618a3eb4f174c43805ce7dc0e5315cb P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:25.522552 1150 tablet_bootstrap.cc:654] T 4902062977ea42149e3466e999117322 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:25.522957 1151 tablet_bootstrap.cc:654] T 3618a3eb4f174c43805ce7dc0e5315cb P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:25.523592 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 1495de70f0844f82a0eea99c059f076b (table default.Table_02e911ce389e4ad581c0a2f976f99c4e [id=70a5ce71d16b47d4bc6dae97be4548e7]). Expected version 1 got 0
I20250901 01:52:25.526767 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet ed1b41340ce742a88c3e4fd85cd25ca6 (table default.Table_139f6d174d794a6e9a618e90587d0c0d [id=0baa4993b72e4a8d94ee3059c9d2db6c]). Expected version 1 got 0
I20250901 01:52:25.532825 917 raft_consensus.cc:2466] T f4fd117c395e41c2ab679c684908c5b7 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:25.532447 2170 raft_consensus.cc:513] T 20f7bfab18134e079e91c52c89468cc3 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:25.534446 2170 leader_election.cc:290] T 20f7bfab18134e079e91c52c89468cc3 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:25.534703 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet d10c193452954c198bba686169b8025b (table default.Table_03fb48eaab9b42858ab7cefb722e43a8 [id=6c0576410d7340d6aa493a747e95565b]). Expected version 1 got 0
I20250901 01:52:25.536064 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 435b1cca5a7d4d49959a310cb250b2a0 (table default.Table_dc9c5d18c7ec478592a76dbd76424b88 [id=126b298fef6c4653952d672adff1db7e]). Expected version 1 got 0
I20250901 01:52:25.538254 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 410d44e5f1974ed69e5dee65e98856a4 (table default.Table_bec114a616504798bede9a9066e57192 [id=ae79f2c40fd7483ea400a36cf3c48888]). Expected version 4 got 0
I20250901 01:52:25.540892 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet f9b293bf6a17453b9ea9892a6970d0b0 (table default.Table_4158d47f631042489d14100215dce98a [id=b67bfe518a094e0781ef931524b55990]). Expected version 1 got 0
I20250901 01:52:25.542974 2271 tablet.cc:1722] T 72ba689f9230411995dc96e2b45601df P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:25.543059 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet e87d1281f35648b0b115b67c95e244e7 (table default.Table_f840789a58984799bd8057128f6ea439 [id=4453959e273c473fadd737f56f7db3dd]). Expected version 1 got 0
I20250901 01:52:25.545742 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 7954393c8c1c4457ab8bb723c03a9c8b (table default.Table_f44c609420054cebad09692be9aa6ecd [id=543f2074598d4c29972e564be1123fca]). Expected version 2 got 0
I20250901 01:52:25.546007 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48a211e174054866b030cdfb0cf947f4" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:25.546509 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 6c0abbfc8d904dce989ce51961e4237d (table default.Table_3f9422b08b0349b9a5466218f0fe1b06 [id=f0dfb3811e744bd4ba6d59195cebde6f]). Expected version 2 got 0
I20250901 01:52:25.547565 1150 tablet_bootstrap.cc:492] T 4902062977ea42149e3466e999117322 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:25.548097 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet c6f7c525f59b49938b697de98d4b5835 (table default.Table_db61e055dbd7450898e82c4c0c4763be [id=aabca67f6f754820977ac9c4c20f3e4f]). Expected version 1 got 0
I20250901 01:52:25.550576 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 8ed6616ce54548ba83ba299c4ecf3c38 (table default.Table_1bdd9041d74045cab6f23de11bf4dd09 [id=946ec3e53dcd4de4b8501b3e87e99072]). Expected version 2 got 0
I20250901 01:52:25.551602 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 081e2630e284498aa819f85bf8326315 (table default.Table_9e7ca61fcc324a8984bd1e470162c61f [id=8199118617cb4707bc3ae9b1363750f7]). Expected version 2 got 0
I20250901 01:52:25.552282 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 2599f98abfdc4871b573edfc2ad97d22 (table default.Table_0e63d21acfcb4c028c9ea6406b6a04e4 [id=81323e6fbe424655b8e185f52657fadc]). Expected version 1 got 0
I20250901 01:52:25.553071 1150 ts_tablet_manager.cc:1397] T 4902062977ea42149e3466e999117322 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.032s user 0.001s sys 0.002s
I20250901 01:52:25.554625 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 81ce448b148447ddbc41875d863e0518 (table default.Table_6e461d9cb0ae4dd095975bbc21533794 [id=49673fa73f8c436e9b8026efc4f3cbbc]). Expected version 1 got 0
I20250901 01:52:25.554988 1981 raft_consensus.cc:491] T df01d5df9fc2424eaf9de4e68e781803 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:25.555111 1981 raft_consensus.cc:513] T df01d5df9fc2424eaf9de4e68e781803 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:25.557789 1055 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b81a12b38ac64efabd44aeb192d181a0" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:25.557750 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 42c1156e8dd84c6fa1396b46de13eb6f (table default.Table_b5d681f51fc84395bbabe1e5b4cfb358 [id=40d05d90b0354334840ad17764f5b6bb]). Expected version 1 got 0
I20250901 01:52:25.558573 1055 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d10c193452954c198bba686169b8025b" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:25.559510 1981 leader_election.cc:290] T df01d5df9fc2424eaf9de4e68e781803 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:25.560597 858 leader_election.cc:336] T df01d5df9fc2424eaf9de4e68e781803 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:25.561110 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 616ff34e8d314e0aa0e11ce60be11789 (table default.Table_0451d0afa26f4d5f8d4214a946eea04f [id=5ee13d46e06a4fa59b1070547573e523]). Expected version 1 got 0
I20250901 01:52:25.562698 1482 catalog_manager.cc:5582] T e1d18e999b5e42b1873631e0f1445081 P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: UNKNOWN } } }
I20250901 01:52:25.565938 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet e1d18e999b5e42b1873631e0f1445081 (table default.Table_fd17a2ad225546aa821489ce94a788a2 [id=033c76e0598342c495ce2be42a0df5dd]). Expected version 1 got 0
I20250901 01:52:25.567811 1151 tablet_bootstrap.cc:492] T 3618a3eb4f174c43805ce7dc0e5315cb P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:25.567843 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet b2e8b2cd10154de08ddd30c214592a4a (table default.Table_4a412a59b6db421eb0187ffd02ecf541 [id=fd72e46c84634b6c9ea74970480ecc62]). Expected version 4 got 0
I20250901 01:52:25.568989 1151 ts_tablet_manager.cc:1397] T 3618a3eb4f174c43805ce7dc0e5315cb P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.001s
I20250901 01:52:25.570199 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a9b7f2e6548043a7865e25387cdd06c9" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:25.570179 1054 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5afae316ff164413be0d25d4bc0f58dc" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:25.571043 1058 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "df01d5df9fc2424eaf9de4e68e781803" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:25.571274 1150 raft_consensus.cc:357] T 4902062977ea42149e3466e999117322 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:25.571390 1104 heartbeater.cc:344] Connected to a master server at 127.0.104.62:43871
I20250901 01:52:25.570156 1482 catalog_manager.cc:5582] T 49aaaec611784e6bac131efd58dbca51 P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: UNKNOWN } } }
I20250901 01:52:25.572870 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 49aaaec611784e6bac131efd58dbca51 (table default.Table_95e1338a49a04f22b679e363e253262e [id=e24b91b4ac904ef0a8c4725f70bced2f]). Expected version 1 got 0
I20250901 01:52:25.573836 1150 raft_consensus.cc:383] T 4902062977ea42149e3466e999117322 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:25.574393 1151 raft_consensus.cc:357] T 3618a3eb4f174c43805ce7dc0e5315cb P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:25.575313 1151 raft_consensus.cc:383] T 3618a3eb4f174c43805ce7dc0e5315cb P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:25.575860 1151 raft_consensus.cc:738] T 3618a3eb4f174c43805ce7dc0e5315cb P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
W20250901 01:52:25.576460 992 consensus_peers.cc:489] T 579b9cdf819f4609a532e96bec093fd5 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:25.576503 1150 raft_consensus.cc:738] T 4902062977ea42149e3466e999117322 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
W20250901 01:52:25.576920 992 consensus_peers.cc:489] T 186ec810d02e4748aad15228b5866ac1 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250901 01:52:25.576831 1150 consensus_queue.cc:260] T 4902062977ea42149e3466e999117322 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:25.576226 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet ddbc15685fca4c708329420109ad0c57 (table default.Table_942175dadf4b496195b2fa53f39f6ddc [id=2bd332b452f84f25af4225af8e6c6740]). Expected version 1 got 0
W20250901 01:52:25.577018 992 consensus_peers.cc:489] T 50e3da53433f4c14b57db8a733fa53bc P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250901 01:52:25.577085 992 consensus_peers.cc:489] T 7442c6ff699141e1aff208ebdc9d192a P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250901 01:52:25.577111 1151 consensus_queue.cc:260] T 3618a3eb4f174c43805ce7dc0e5315cb P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:25.578548 859 leader_election.cc:343] T 48a211e174054866b030cdfb0cf947f4 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:25.579232 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet d0c3c363dda444519a17dca63ad9caa7 (table default.Table_0b4ac0a1caa5472d909448e41318e679 [id=8c1b263824ec4c15a5ec424364c78895]). Expected version 2 got 0
I20250901 01:52:25.581578 1151 ts_tablet_manager.cc:1428] T 3618a3eb4f174c43805ce7dc0e5315cb P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.012s user 0.000s sys 0.003s
I20250901 01:52:25.582372 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 629db80a8ab444e582628578155638fd (table default.Table_f0e56dba1e8a4ed49a760e204c1ff96e [id=c3865e058daa4cea9a89f5046fca712a]). Expected version 1 got 0
I20250901 01:52:25.583138 1151 tablet_bootstrap.cc:492] T 17e8340dbd1a406cb782dad3fe88cc7e P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:25.583273 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 75a703c8e20a4a2fa0ab6ec58d41ef92 (table default.Table_2f09538b6fbe401f909105decb56e1cb [id=a90d1f10531042c3bb7a196b8be61c5f]). Expected version 1 got 0
I20250901 01:52:25.584214 2251 tablet.cc:1722] T b2e8b2cd10154de08ddd30c214592a4a P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250901 01:52:25.578682 859 leader_election.cc:304] T 48a211e174054866b030cdfb0cf947f4 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:25.585506 1981 raft_consensus.cc:2747] T 48a211e174054866b030cdfb0cf947f4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:25.585682 859 leader_election.cc:343] T b81a12b38ac64efabd44aeb192d181a0 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:25.585786 992 leader_election.cc:304] T aa424ac6ef1441e6b1cc2867b1c48186 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:25.585858 859 leader_election.cc:304] T b81a12b38ac64efabd44aeb192d181a0 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:25.586079 1981 raft_consensus.cc:2747] T b81a12b38ac64efabd44aeb192d181a0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:25.586225 859 leader_election.cc:343] T d10c193452954c198bba686169b8025b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:25.586320 859 leader_election.cc:304] T d10c193452954c198bba686169b8025b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:25.586529 1981 raft_consensus.cc:2747] T d10c193452954c198bba686169b8025b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:25.586668 859 leader_election.cc:343] T a9b7f2e6548043a7865e25387cdd06c9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:25.586422 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet e9f1f892658b4736859ff507536920a1 (table default.Table_8825dd28afbe464193eb1b668cc9326a [id=9aa62cb634314a0a849612c54c1220d9]). Expected version 1 got 0
I20250901 01:52:25.586786 859 leader_election.cc:304] T a9b7f2e6548043a7865e25387cdd06c9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:25.586972 1981 raft_consensus.cc:2747] T a9b7f2e6548043a7865e25387cdd06c9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:25.587101 859 leader_election.cc:343] T 5afae316ff164413be0d25d4bc0f58dc P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:25.587196 859 leader_election.cc:304] T 5afae316ff164413be0d25d4bc0f58dc P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:25.587383 1981 raft_consensus.cc:2747] T 5afae316ff164413be0d25d4bc0f58dc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:25.587515 859 leader_election.cc:343] T df01d5df9fc2424eaf9de4e68e781803 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:25.587610 859 leader_election.cc:304] T df01d5df9fc2424eaf9de4e68e781803 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:25.587841 1981 raft_consensus.cc:2747] T df01d5df9fc2424eaf9de4e68e781803 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:25.573176 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "64fb46b942ff4e4c8749986ecc3dfe65" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:25.589038 915 raft_consensus.cc:3058] T 64fb46b942ff4e4c8749986ecc3dfe65 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:25.589041 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet fa7fb88ded21432a8757f37ec3db5e71 (table default.Table_22d8d38d0b624912a0d8968fb764420e [id=808a8849c99e485b8be7cd17397874ec]). Expected version 1 got 0
I20250901 01:52:25.589597 1981 raft_consensus.cc:491] T abf6e3be60f74872b2308dc95bf9cd37 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:25.589748 1981 raft_consensus.cc:513] T abf6e3be60f74872b2308dc95bf9cd37 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:25.590610 2170 raft_consensus.cc:2802] T aa424ac6ef1441e6b1cc2867b1c48186 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:25.590754 2170 raft_consensus.cc:695] T aa424ac6ef1441e6b1cc2867b1c48186 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:25.590670 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 5f5396045979450193d461ba1334a5c9 (table default.Table_a89cc8e48563460baabc34c821ceae35 [id=9c3930e9fcfe4b49ba4bea67a2b0ad07]). Expected version 1 got 0
I20250901 01:52:25.590991 2170 consensus_queue.cc:237] T aa424ac6ef1441e6b1cc2867b1c48186 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:25.592784 1151 tablet_bootstrap.cc:654] T 17e8340dbd1a406cb782dad3fe88cc7e P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:25.596854 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet b164b4a67a9143ceaff69b0ce49d63fe (table default.Table_da2c70a8f2fe49a0a8184e88c0a9e0ea [id=c1b1e9e3876244f68d31c63ce7bdebf1]). Expected version 1 got 0
I20250901 01:52:25.598606 1872 raft_consensus.cc:513] T 4855448edb4b40c184838aa7f81c112b P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:25.599234 1872 leader_election.cc:290] T 4855448edb4b40c184838aa7f81c112b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:25.599083 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet ae21b9ae2e3b472eb67fb5f96388a50f (table default.Table_046289f393ff432aae0b033bdec1fb17 [id=db57b18a9ce24e068b362f5fbe940531]). Expected version 1 got 0
I20250901 01:52:25.599553 921 raft_consensus.cc:2466] T 616ff34e8d314e0aa0e11ce60be11789 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:25.600072 2118 ts_tablet_manager.cc:1918] T b11e8f77243e4764b96548d1132b8e5a P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250901 01:52:25.600179 2118 log.cc:1199] T b11e8f77243e4764b96548d1132b8e5a P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/b11e8f77243e4764b96548d1132b8e5a
I20250901 01:52:25.600438 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 36e10328c34242ebb4dbd8f294162696 (table default.Table_8445fcd774874cadaf4acc3581eee417 [id=958e51a6bbee47638e504e1a44094737]). Expected version 1 got 0
I20250901 01:52:25.601538 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet d4c9aa723aee4dc4b6237cb5c3722b52 (table default.Table_5e65c5f068b84995a2d7a76fa65dda0f [id=b9d81ddd38fe47838877b010be2b40d0]). Expected version 1 got 0
W20250901 01:52:25.602993 992 leader_election.cc:343] T 3fda93767e494a4684978c8522b829dd P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 3fda93767e494a4684978c8522b829dd
W20250901 01:52:25.603163 992 leader_election.cc:343] T 9d886fa63f8f4996b087997b542e8bb1 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 9d886fa63f8f4996b087997b542e8bb1
I20250901 01:52:25.605530 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 93bcac1fd95d44dc927abfca81692921 (table default.Table_b8f8ee74648c406eaf589d3ab80934db [id=b8f69313439543be9c11eea88dfa69c5]). Expected version 1 got 0
I20250901 01:52:25.606642 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 508406601c994e42a5c5b1f665e74957 (table default.Table_345b7f9fe8214aa6878f20a166125364 [id=4aabc328f6374ec5b1dea26f778208b2]). Expected version 1 got 0
I20250901 01:52:25.607438 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet da5f69c5b0ca4257ad2ff7918ee2af2e (table default.Table_ecb9e8f638a043cc87c38afbebce250f [id=61f1e8a842844ed899e5913c76f362ff]). Expected version 1 got 0
I20250901 01:52:25.608096 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 7f46e9ddadaf4ff986d8c991178daa0c (table default.Table_9978497447394e5694ae928412634ecb [id=ea928501fe954cd0b81d38add4ef7128]). Expected version 2 got 0
I20250901 01:52:25.610476 2277 tablet.cc:1722] T 5adf6b492640417ca071c94702b48cb9 P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250901 01:52:25.612845 1150 ts_tablet_manager.cc:1428] T 4902062977ea42149e3466e999117322 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.060s user 0.001s sys 0.001s
I20250901 01:52:25.615274 1151 tablet_bootstrap.cc:492] T 17e8340dbd1a406cb782dad3fe88cc7e P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:25.615643 1151 ts_tablet_manager.cc:1397] T 17e8340dbd1a406cb782dad3fe88cc7e P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.002s
I20250901 01:52:25.617257 1151 raft_consensus.cc:357] T 17e8340dbd1a406cb782dad3fe88cc7e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:25.617689 1151 raft_consensus.cc:383] T 17e8340dbd1a406cb782dad3fe88cc7e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:25.617851 1151 raft_consensus.cc:738] T 17e8340dbd1a406cb782dad3fe88cc7e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:25.618333 1151 consensus_queue.cc:260] T 17e8340dbd1a406cb782dad3fe88cc7e P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:25.614054 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 3dae00aa38204ee3ac77325eb334be42 (table default.Table_faf7ef12324f454396e5ae64d64ebe51 [id=5812902df9ea406ba59ff0ff23f130b6]). Expected version 1 got 0
W20250901 01:52:25.628532 1473 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet 34042ea5c1b645969c1de41fe587c343 on TS 0cf599f74322478d91475df234e58965 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:25.629101 1151 ts_tablet_manager.cc:1428] T 17e8340dbd1a406cb782dad3fe88cc7e P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.013s user 0.002s sys 0.001s
I20250901 01:52:25.628257 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet fcb7b5b7d63d4f0b906fea97cb548de2 (table default.Table_2ceaff93c8f44f00aab53e48d8466dfd [id=4d9ea39016ca4bf4859f4f80ea4e9ed6]). Expected version 4 got 0
I20250901 01:52:25.630870 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet cb7d7e1796c0456f8de98fb812b9e374 (table default.Table_c4148158ed7b48a3946559ba978e72fd [id=327c5a63904540508142ab600c308757]). Expected version 1 got 0
I20250901 01:52:25.635156 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet b009cc95c819431ea9827a3e453b8da3 (table default.Table_203547c9458e43b3b48ea051d56892c6 [id=cc26c89db2484a6fad919231116285f2]). Expected version 1 got 0
I20250901 01:52:25.636540 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "42c1156e8dd84c6fa1396b46de13eb6f" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:25.636724 916 raft_consensus.cc:2466] T 42c1156e8dd84c6fa1396b46de13eb6f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:25.637257 2290 tablet.cc:1722] T 3c1fd0820d8a4995abe027f685da7f7b P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:25.637982 911 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20f7bfab18134e079e91c52c89468cc3" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:25.638130 911 raft_consensus.cc:3058] T 20f7bfab18134e079e91c52c89468cc3 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:25.637773 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 33f637f6a8134215bb1995ec38440073 (table default.Table_2ece91972d494680a8ad00273c10dee4 [id=557afe29976d40f6bfb6ae4ca348f887]). Expected version 1 got 0
I20250901 01:52:25.639060 1981 leader_election.cc:290] T abf6e3be60f74872b2308dc95bf9cd37 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:25.639823 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet f80f13db501c42c7950499df8fcb6fc8 (table default.Table_ce86974fd8974de9a5a737954fac2ad7 [id=ff772d20b287476f94716379b86d2a71]). Expected version 1 got 0
I20250901 01:52:25.641724 2118 ts_tablet_manager.cc:1939] T b11e8f77243e4764b96548d1132b8e5a P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:25.641667 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 3618a3eb4f174c43805ce7dc0e5315cb (table default.Table_83f2c0acedbc4c6982caf5313646e90b [id=31a2e50c32644321825da40176127015]). Expected version 1 got 0
I20250901 01:52:25.642831 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet d1f2d98712ff4ea6aec665512de068bd (table default.Table_82d91a1ba2eb4265896824028dec8d3a [id=3f7624d2901445e39155c12fa68e34e9]). Expected version 1 got 0
I20250901 01:52:25.643891 2301 tablet.cc:1722] T e6c10426f60f43e8bde69d9b1b9c8eb6 P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:25.644357 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet e6c10426f60f43e8bde69d9b1b9c8eb6 (table default.Table_eecfab7239cc40c5a2f1fb6ac6d35dc9 [id=d3dbeb4258424133844e20b181ee44e4]). Expected version 1 got 0
I20250901 01:52:25.645831 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 615b2306fca84003b264da9d5f95ddab (table default.Table_302f6e7cf97d46b9a621eb852e6aefe0 [id=cecdd8d6ca6d445c9fd82decd2a95366]). Expected version 2 got 0
I20250901 01:52:25.647874 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 3c1fd0820d8a4995abe027f685da7f7b (table default.Table_bcd502e1360b48b1953fa97eadce7006 [id=b3914f33e856466e8479d0a2263ec56f]). Expected version 1 got 0
W20250901 01:52:25.648509 2285 log.cc:927] Time spent T f62d96c37cca4143a4b098b7c69979e6 P 00c282f779c1477d920f24dc0dc16628: Append to log took a long time: real 0.086s user 0.000s sys 0.000s
I20250901 01:52:25.649438 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet f62d96c37cca4143a4b098b7c69979e6 (table default.Table_0877ac8202a24e9da650a589bc6dbfcc [id=1f65b3e09c9748f189ecb2b56c5d1132]). Expected version 1 got 0
W20250901 01:52:25.649981 858 leader_election.cc:336] T abf6e3be60f74872b2308dc95bf9cd37 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:25.650746 1150 tablet_bootstrap.cc:492] T ff013ea889d44cba8797834e4611b489 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:25.650487 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 5adf6b492640417ca071c94702b48cb9 (table default.Table_589c302e02474fc3bc5d3a290a8bd54a [id=5d14706774d345c48ca3575feabb17b0]). Expected version 2 got 0
I20250901 01:52:25.651921 915 raft_consensus.cc:2466] T 64fb46b942ff4e4c8749986ecc3dfe65 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:25.654130 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet ad45c90ba0da43c9860745e534664a98 (table default.Table_1cd3f3fb6b244b3f8796bfca9f7622c8 [id=6fd739b0e725424ca844dbb652ea8028]). Expected version 2 got 0
I20250901 01:52:25.654410 911 raft_consensus.cc:2466] T 20f7bfab18134e079e91c52c89468cc3 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:25.655284 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet fecf7d538b4c4c619182d5eccb44dd8b (table default.Table_1b5729d9768546fe99eaafc4e2604f42 [id=02d0056abe2146ac9af81ebf4447b99d]). Expected version 1 got 0
I20250901 01:52:25.651621 2171 raft_consensus.cc:491] T abe27e749fa0485195e0064a49d5431e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:25.656569 2171 raft_consensus.cc:513] T abe27e749fa0485195e0064a49d5431e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:25.657089 2171 leader_election.cc:290] T abe27e749fa0485195e0064a49d5431e P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:25.657068 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 8ca5cf4e99014c0d84a3acad4ef713af (table default.Table_ed6fd284be6f4ae3a008e21e7ce17930 [id=303e76c918bc4c39aa47a4d24849a358]). Expected version 1 got 0
I20250901 01:52:25.657836 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 61ddcf86c9f34726939cc116434f0822 (table default.Table_63d0f2ead9d946d39ce40162ebd769d6 [id=f9dcf4937b1143248bfa9d88053fce17]). Expected version 3 got 0
I20250901 01:52:25.658418 2171 raft_consensus.cc:491] T 15e4d972e8254cf899c3e0b71b6e2bd9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:25.658394 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 3a8231590ea643b0a1f83eec4d9a8191 (table default.Table_a4ebd8ab87c842f082da5cd04d1bddac [id=fd6d8e92d6e64a89a7d38aa761376955]). Expected version 1 got 0
I20250901 01:52:25.658555 2171 raft_consensus.cc:513] T 15e4d972e8254cf899c3e0b71b6e2bd9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:25.659009 2171 leader_election.cc:290] T 15e4d972e8254cf899c3e0b71b6e2bd9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:25.659587 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 89d40f0331114490b51de3ba6d773571 (table default.Table_4dacb3b1d2e94e42ac8937eb6eb58375 [id=8ee0aa1ba24042dd8ded004644913b9c]). Expected version 1 got 0
I20250901 01:52:25.661083 992 leader_election.cc:304] T f4fd117c395e41c2ab679c684908c5b7 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:25.661877 2276 tablet.cc:1722] T 3a8231590ea643b0a1f83eec4d9a8191 P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:25.661899 2271 tablet.cc:1722] T ddbc15685fca4c708329420109ad0c57 P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:25.663916 2170 raft_consensus.cc:2802] T f4fd117c395e41c2ab679c684908c5b7 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:25.664161 1761 consensus_queue.cc:1035] T e9b179dba2cc49428ba8090c962d6136 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:25.668386 2118 tablet_replica.cc:331] T 6741c56499b44f648fb032c8cf50205b P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:25.668687 2118 raft_consensus.cc:2241] T 6741c56499b44f648fb032c8cf50205b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:25.668881 2118 raft_consensus.cc:2270] T 6741c56499b44f648fb032c8cf50205b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:25.669492 2118 ts_tablet_manager.cc:1905] T 6741c56499b44f648fb032c8cf50205b P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:25.671437 2170 raft_consensus.cc:695] T f4fd117c395e41c2ab679c684908c5b7 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:25.671730 2170 consensus_queue.cc:237] T f4fd117c395e41c2ab679c684908c5b7 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:25.672920 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet b11e8f77243e4764b96548d1132b8e5a (table default.Table_3c7c628f35fb4d02bb79a34995f12938 [id=a39e8b171a874df1890d072732e55df9]) successfully deleted
I20250901 01:52:25.673851 1151 tablet_bootstrap.cc:492] T 81c36693199945d98465579b509852e2 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:25.675958 1151 tablet_bootstrap.cc:654] T 81c36693199945d98465579b509852e2 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:25.677742 1150 tablet_bootstrap.cc:654] T ff013ea889d44cba8797834e4611b489 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:25.688182 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 72ba689f9230411995dc96e2b45601df (table default.Table_604f2c5c58d74154bacf2a065a467f32 [id=351a05134f4948039c920cafbe050914]). Expected version 1 got 0
I20250901 01:52:25.690182 1489 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 6c05eff658924ca4a9a1e7a37f917d67
W20250901 01:52:25.690383 1489 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 6c05eff658924ca4a9a1e7a37f917d67: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:25.690557 1485 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet eee34044c39d4013baf78d3705489626
W20250901 01:52:25.690737 1485 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet eee34044c39d4013baf78d3705489626: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:25.691072 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_58f65ebe7c0f447dbfa18d2f9643c078"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.691282 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3e795da496804821856b91c311e9b39c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.691928 1489 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_bec114a616504798bede9a9066e57192" } modify_external_catalogs: true
I20250901 01:52:25.692034 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ef6230005a3a48d7955271abaee2a455"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.693585 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5230fa12e5b24839b368a82894c079d5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.694248 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1d89a57da2034b33a105234952bc2ae8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.695015 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_27781b4441f04f8c86bdf7cccfe81959"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.695695 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_6add76ab9a1044dd8731a77cdd9564b2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.695905 1150 tablet_bootstrap.cc:492] T ff013ea889d44cba8797834e4611b489 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:25.696059 1150 ts_tablet_manager.cc:1397] T ff013ea889d44cba8797834e4611b489 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.083s user 0.003s sys 0.003s
I20250901 01:52:25.696310 1485 master_service.cc:948] ReplaceTablet: received request to replace tablet fa7fb88ded21432a8757f37ec3db5e71 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:25.696885 1150 raft_consensus.cc:357] T ff013ea889d44cba8797834e4611b489 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:25.697057 1150 raft_consensus.cc:383] T ff013ea889d44cba8797834e4611b489 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:25.697116 1150 raft_consensus.cc:738] T ff013ea889d44cba8797834e4611b489 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:25.697306 1150 consensus_queue.cc:260] T ff013ea889d44cba8797834e4611b489 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:25.697909 1150 ts_tablet_manager.cc:1428] T ff013ea889d44cba8797834e4611b489 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250901 01:52:25.698180 1150 tablet_bootstrap.cc:492] T 5f52698fb56d472d9e91fb10cb10c69a P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:25.692098 1489 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_bec114a616504798bede9a9066e57192" } modify_external_catalogs: true
I20250901 01:52:25.699911 1150 tablet_bootstrap.cc:654] T 5f52698fb56d472d9e91fb10cb10c69a P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:25.700313 1490 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1495de70f0844f82a0eea99c059f076b consensus state has no leader
I20250901 01:52:25.700713 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet 6c05eff658924ca4a9a1e7a37f917d67 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:25 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:25.700719 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5ee01e47a15c44e5ac172494848ad05e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.700790 1872 raft_consensus.cc:491] T 98cca86164db4252b1b49de88747822a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:25.700896 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet eee34044c39d4013baf78d3705489626 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:25 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:25.701552 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4d5f1d90d9ae4457a1cc8897737e28c5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.702994 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7f70ffd2d7874aa794121210d269f177"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.703686 1490 master_service.cc:948] ReplaceTablet: received request to replace tablet d4c9aa723aee4dc4b6237cb5c3722b52 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:25.706693 900 tablet_service.cc:1515] Processing DeleteTablet for tablet 6c05eff658924ca4a9a1e7a37f917d67 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:25 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.706681 1492 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_4158d47f631042489d14100215dce98a" } new_table_name: "default.Table_a7fa05538bbd4766aa24d557677a692c" modify_external_catalogs: true
I20250901 01:52:25.706802 901 tablet_service.cc:1515] Processing DeleteTablet for tablet eee34044c39d4013baf78d3705489626 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:25 UTC) from {username='slave'} at 127.0.0.1:34472
W20250901 01:52:25.707410 1484 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet aa424ac6ef1441e6b1cc2867b1c48186 consensus state has no leader
I20250901 01:52:25.707796 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_78fc59d49c8b482780e2570d4aae1d29"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.708748 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c9b3b053703443e78894724bf6f32617"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.709538 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_cd363006dead4086aca1db0f30bba090"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.710312 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3c3a50d3c6374d0db2bbe54c7d181bc0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.711125 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_972d6353c7b845718ab18e5eac8098e5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.711905 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ced4264e18cd461988cf0a837adbfc79"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.716415 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5619c3cd21114edd8cb634d7bb1a7f1d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.717483 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f36f9588c1724adab2bb7d7e9d5339f7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.719611 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a04e1856d9de40f383a20dc9d33fec93"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.722553 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_11e005407d424412b72dd143ace3f964"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.724820 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_6d0de75ff41046ba9a0abc535bd0107c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.726583 1993 raft_consensus.cc:491] T 4a87e68850c4459eb2c8eafac89c6aa9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:25.726728 1993 raft_consensus.cc:513] T 4a87e68850c4459eb2c8eafac89c6aa9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:25.727273 1993 leader_election.cc:290] T 4a87e68850c4459eb2c8eafac89c6aa9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:25.728226 1488 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 7cc1efcb6bff4aba8015d2d3b7f4c923
W20250901 01:52:25.728474 1488 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 7cc1efcb6bff4aba8015d2d3b7f4c923: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:25.729099 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_376b9bfa6d8143628a5cdb7b3728210e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.729933 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ac7b70563ae3458c8da66e1f3c9abd65"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.730616 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e1658ec5c3d340d6bfd1911ac8fe5472"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.731393 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5397c5fbdb974e7aa38c4d9cdf9a0db0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.732249 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f2d9403ae39b463eb7b31f1ce5f47116"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.733054 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet 7cc1efcb6bff4aba8015d2d3b7f4c923 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:25 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:25.700904 1872 raft_consensus.cc:513] T 98cca86164db4252b1b49de88747822a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:25.735999 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_244bcf32dcd54b4c809906f2e0979a43"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.736575 1872 leader_election.cc:290] T 98cca86164db4252b1b49de88747822a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:25.736812 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_cb826e69082b4b6e826656160d506c34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.738473 901 tablet_service.cc:1515] Processing DeleteTablet for tablet 7cc1efcb6bff4aba8015d2d3b7f4c923 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:25 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.745539 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_01c7e4728e8142b1a20eab318249fb92"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.746619 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b9797bc0e5544250a1c28686dc516c63"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.751343 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_61056c0a88df4fa28cfa6634ce16a155"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.753324 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_af68a6ec30a649f3ba0228c91cbcc93d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.756707 1491 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 06d5eeb70e664f73bf643922d445bc7a
W20250901 01:52:25.756947 1491 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 06d5eeb70e664f73bf643922d445bc7a: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
W20250901 01:52:25.757120 1492 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f9b293bf6a17453b9ea9892a6970d0b0 consensus state has no leader
I20250901 01:52:25.757508 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5677d900dd6e442d817792545d24b7f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.757597 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_23401182f2b2477f8891bd177ae3ac08"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.758173 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_6ee28bee028343f9b2862255b07fa6ed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.758483 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_0a46e680e69f4bd08b2a1aedad580386"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.758817 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8d2972efdddc4b79b17bd261c38f6e26"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.759269 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2c522533c8484c37a1c39b498b874f94"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.759472 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4cb727480045484ebd0339647e367e49"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.760937 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet 06d5eeb70e664f73bf643922d445bc7a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:25 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:25.760087 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_953934477708490b8017877d64cd2342"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.762650 1486 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: default.Table_7c12d66048a6470e8c227dc1d9f45fa4 [id=91472f220fe44297b79d3eec8b9b80e8] alter complete (version 1)
I20250901 01:52:25.764760 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_eb8d1273fc6c4e1982fbfc7470dec4d1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.766883 1993 raft_consensus.cc:491] T ef58f477ebeb49639ed7d071e1a62c10 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:25.766857 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7622539e27604e46a8e9c3924d146b1e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.767010 1993 raft_consensus.cc:513] T ef58f477ebeb49639ed7d071e1a62c10 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:25.767627 1993 leader_election.cc:290] T ef58f477ebeb49639ed7d071e1a62c10 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:25.767987 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_09f600123f7540f783c2c576d0032506"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.768847 901 tablet_service.cc:1515] Processing DeleteTablet for tablet 06d5eeb70e664f73bf643922d445bc7a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:25 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.770881 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d1a3661776b141438a2736db61899648"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.772146 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7adb902676d04fd0910e94544efb4cc0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.773586 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2c3f9314db854e98ae7aea76cf2a9753"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.774627 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ea8ba1735b5e447cbaef66b237251e36"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.777122 1993 raft_consensus.cc:491] T 6741c56499b44f648fb032c8cf50205b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:25.777264 1993 raft_consensus.cc:513] T 6741c56499b44f648fb032c8cf50205b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:25.777877 1993 leader_election.cc:290] T 6741c56499b44f648fb032c8cf50205b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:25.781168 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_63ecbde24314447cbbb9c964df627bce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.778199 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_df1362de86db4209abd80b5f89aee7bc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:25.790357 992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:40561, user_credentials={real_user=slave}} blocked reactor thread for 128866us
I20250901 01:52:25.792588 1151 tablet_bootstrap.cc:492] T 81c36693199945d98465579b509852e2 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:25.792744 1151 ts_tablet_manager.cc:1397] T 81c36693199945d98465579b509852e2 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.119s user 0.007s sys 0.002s
I20250901 01:52:25.793694 1151 raft_consensus.cc:357] T 81c36693199945d98465579b509852e2 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:25.793880 1151 raft_consensus.cc:383] T 81c36693199945d98465579b509852e2 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:25.793936 1151 raft_consensus.cc:738] T 81c36693199945d98465579b509852e2 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:25.794457 1151 consensus_queue.cc:260] T 81c36693199945d98465579b509852e2 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:25.795277 1151 ts_tablet_manager.cc:1428] T 81c36693199945d98465579b509852e2 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250901 01:52:25.795665 1151 tablet_bootstrap.cc:492] T ad45c90ba0da43c9860745e534664a98 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:25.797600 1151 tablet_bootstrap.cc:654] T ad45c90ba0da43c9860745e534664a98 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:25.798401 992 leader_election.cc:304] T 616ff34e8d314e0aa0e11ce60be11789 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:25.799616 2171 raft_consensus.cc:2802] T 616ff34e8d314e0aa0e11ce60be11789 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:25.799777 2171 raft_consensus.cc:695] T 616ff34e8d314e0aa0e11ce60be11789 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:25.799993 2171 consensus_queue.cc:237] T 616ff34e8d314e0aa0e11ce60be11789 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:25.800573 1492 master_service.cc:948] ReplaceTablet: received request to replace tablet 50472a7aa2f04e0fbd65ac6c86c34b3e from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:25.801024 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5567f22efa64409a86b7491973a6e3a5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.803205 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3f6837603f3243309e84773edd04b86f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.806999 2113 ts_tablet_manager.cc:1918] T 3c12d6a70ac24dce82e851782e6fe5aa P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:25.807127 2113 log.cc:1199] T 3c12d6a70ac24dce82e851782e6fe5aa P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/3c12d6a70ac24dce82e851782e6fe5aa
I20250901 01:52:25.807538 2113 ts_tablet_manager.cc:1939] T 3c12d6a70ac24dce82e851782e6fe5aa P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:25.809743 992 leader_election.cc:304] T 42c1156e8dd84c6fa1396b46de13eb6f P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:25.811378 1150 tablet_bootstrap.cc:492] T 5f52698fb56d472d9e91fb10cb10c69a P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:25.811517 1150 ts_tablet_manager.cc:1397] T 5f52698fb56d472d9e91fb10cb10c69a P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.113s user 0.013s sys 0.012s
I20250901 01:52:25.812355 2307 raft_consensus.cc:2802] T 42c1156e8dd84c6fa1396b46de13eb6f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:25.812449 2307 raft_consensus.cc:491] T 42c1156e8dd84c6fa1396b46de13eb6f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:25.812512 2307 raft_consensus.cc:3058] T 42c1156e8dd84c6fa1396b46de13eb6f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:25.812693 2118 ts_tablet_manager.cc:1918] T 6741c56499b44f648fb032c8cf50205b P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:25.812816 2118 log.cc:1199] T 6741c56499b44f648fb032c8cf50205b P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/6741c56499b44f648fb032c8cf50205b
I20250901 01:52:25.813102 2118 ts_tablet_manager.cc:1939] T 6741c56499b44f648fb032c8cf50205b P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:25.813555 1489 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 410d44e5f1974ed69e5dee65e98856a4
W20250901 01:52:25.813735 1489 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 410d44e5f1974ed69e5dee65e98856a4: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:25.815119 2307 raft_consensus.cc:513] T 42c1156e8dd84c6fa1396b46de13eb6f P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:25.815781 2307 leader_election.cc:290] T 42c1156e8dd84c6fa1396b46de13eb6f P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:25.816025 2307 consensus_queue.cc:1035] T 49aaaec611784e6bac131efd58dbca51 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:25.817052 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet 410d44e5f1974ed69e5dee65e98856a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:25 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:25.819310 992 leader_election.cc:304] T 64fb46b942ff4e4c8749986ecc3dfe65 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:25.819494 2301 tablet.cc:1722] T f62d96c37cca4143a4b098b7c69979e6 P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:25.819404 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ab7cb5b6a37d4709b18bdffde3e8aa2b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.820016 2307 raft_consensus.cc:2802] T 64fb46b942ff4e4c8749986ecc3dfe65 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:25.820134 2307 raft_consensus.cc:695] T 64fb46b942ff4e4c8749986ecc3dfe65 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:25.820366 2307 consensus_queue.cc:237] T 64fb46b942ff4e4c8749986ecc3dfe65 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:25.824112 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_052abf0a1b204bf6b7120901c1f90fab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.825057 901 tablet_service.cc:1515] Processing DeleteTablet for tablet 410d44e5f1974ed69e5dee65e98856a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:25 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.825398 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_daa078ce9a274de997e3bca81b0e49e3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.826611 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e63aa0e204ed41cf9c8c4ee613d84429"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.827731 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e8526eddefd44885b097ffc5d88b6eae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.827822 992 leader_election.cc:304] T 20f7bfab18134e079e91c52c89468cc3 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:25.828255 2210 raft_consensus.cc:2802] T 20f7bfab18134e079e91c52c89468cc3 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:25.828359 2210 raft_consensus.cc:695] T 20f7bfab18134e079e91c52c89468cc3 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:25.829519 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_0447afd9ac93484883ec1a14ce02c435"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.829602 2210 consensus_queue.cc:237] T 20f7bfab18134e079e91c52c89468cc3 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:25.830657 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5a863dccac094d16a83f05b66f8c943a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.832533 1490 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet d4c9aa723aee4dc4b6237cb5c3722b52
W20250901 01:52:25.832764 1490 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet d4c9aa723aee4dc4b6237cb5c3722b52: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:25.832918 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 3c12d6a70ac24dce82e851782e6fe5aa (table default.Table_d2049d0c20bb4949ae6c249b690e7f86 [id=26bba0e60b6b43eb87d084be403c556b]) successfully deleted
I20250901 01:52:25.833132 1490 catalog_manager.cc:6652] ReplaceTablet: tablet d4c9aa723aee4dc4b6237cb5c3722b52 deleted and replaced by tablet f8f923ddcd2740bcb7e521f4f57c4034
I20250901 01:52:25.833508 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet d4c9aa723aee4dc4b6237cb5c3722b52 with delete_type TABLET_DATA_DELETED (replaced by tablet f8f923ddcd2740bcb7e521f4f57c4034) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:25.833995 901 tablet_service.cc:1515] Processing DeleteTablet for tablet d4c9aa723aee4dc4b6237cb5c3722b52 with delete_type TABLET_DATA_DELETED (replaced by tablet f8f923ddcd2740bcb7e521f4f57c4034) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.834815 2113 tablet_replica.cc:331] T e9b179dba2cc49428ba8090c962d6136 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:25.835830 2113 raft_consensus.cc:2241] T e9b179dba2cc49428ba8090c962d6136 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:25.836920 2113 pending_rounds.cc:70] T e9b179dba2cc49428ba8090c962d6136 P fd27ab6665304aaea53adbd102dd4f58: Trying to abort 1 pending ops.
I20250901 01:52:25.837062 2113 pending_rounds.cc:77] T e9b179dba2cc49428ba8090c962d6136 P fd27ab6665304aaea53adbd102dd4f58: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7195408566788640768 op_type: NO_OP noop_request { }
I20250901 01:52:25.837263 2113 raft_consensus.cc:2887] T e9b179dba2cc49428ba8090c962d6136 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250901 01:52:25.837411 2113 raft_consensus.cc:2270] T e9b179dba2cc49428ba8090c962d6136 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:25.838296 2113 ts_tablet_manager.cc:1905] T e9b179dba2cc49428ba8090c962d6136 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:25.838442 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_50a54da3d479454e800ba27973fb39ac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.839895 1485 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet fa7fb88ded21432a8757f37ec3db5e71
W20250901 01:52:25.840106 1485 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet fa7fb88ded21432a8757f37ec3db5e71: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:25.840461 1485 catalog_manager.cc:6652] ReplaceTablet: tablet fa7fb88ded21432a8757f37ec3db5e71 deleted and replaced by tablet 81319e373c9a4475abceeeedb507aabc
I20250901 01:52:25.840711 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet fa7fb88ded21432a8757f37ec3db5e71 with delete_type TABLET_DATA_DELETED (replaced by tablet 81319e373c9a4475abceeeedb507aabc) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:25.841112 901 tablet_service.cc:1515] Processing DeleteTablet for tablet fa7fb88ded21432a8757f37ec3db5e71 with delete_type TABLET_DATA_DELETED (replaced by tablet 81319e373c9a4475abceeeedb507aabc) from {username='slave'} at 127.0.0.1:34472
E20250901 01:52:25.841868 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae4844183a3c4a04a775e6fd878391a3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_aee74198b51d4f089538c4f2bf3ce4dc'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:25.842052 1696 catalog_manager.cc:6205] Error selecting replicas for tablet be8dfbc466564842a2fdb4953eee939b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_75ba3ba70ca24a94920e7bce96190ff2'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:25.842232 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 96824e68bbf24b8287dbb683038ffaba, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_3c7c628f35fb4d02bb79a34995f12938'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:25.842388 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 7317eb7848ee44698038a4b404d4bf21, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b0e916c76a7a4766aa0887cc8691488f'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:25.843183 1696 catalog_manager.cc:6205] Error selecting replicas for tablet e2ed32b3292141278af28026528e96c2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_50ae3908a7944f70949c54be8fbf1564'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:25.843348 1696 catalog_manager.cc:6205] Error selecting replicas for tablet a2ca69fabadd4ad69e1670cae7a0c503, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_32c4716e3aa24a87a6b1472984b255c4'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:25.843515 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 0813cc68d4b0488ba60a523e4479971b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_52733c9a363b4ed9b6ff11ad8571e421'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:25.843652 1696 catalog_manager.cc:6205] Error selecting replicas for tablet cc3b8736b9094d7e892c80da4d5b20f6, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_93afd37289414655980b22934695bd55'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:25.843798 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9074543dea9949e7a085717b40913348, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_53e30205c5fd488eb3b48c685a159d04'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:25.843927 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 18498c9cb09141d08c1c5c0854b63ebd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77f4810bc83c4d948531da2a45a3792e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:25.844115 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 6ce9521b925b4f97bb4a810ce77e147e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_9e04f976a93f4d30baccf90fa3622dfe'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:25.847894 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_70cde3ceb892491a86b8e7e6d32b9ecd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.850514 1151 tablet_bootstrap.cc:492] T ad45c90ba0da43c9860745e534664a98 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:25.851174 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9ffe37b878024df4b68a842dcd05795c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.853145 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_84a4f595a7cd4438a858a247e529c25d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.854861 1151 ts_tablet_manager.cc:1397] T ad45c90ba0da43c9860745e534664a98 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.059s user 0.004s sys 0.000s
I20250901 01:52:25.855430 1485 master_service.cc:948] ReplaceTablet: received request to replace tablet 40d74d7b5a3b4d1dae09eebd9e07a0c4 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:25.860303 2113 ts_tablet_manager.cc:1918] T e9b179dba2cc49428ba8090c962d6136 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:25.860728 2113 log.cc:1199] T e9b179dba2cc49428ba8090c962d6136 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/e9b179dba2cc49428ba8090c962d6136
I20250901 01:52:25.861619 1151 raft_consensus.cc:357] T ad45c90ba0da43c9860745e534664a98 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:25.864728 1150 raft_consensus.cc:357] T 5f52698fb56d472d9e91fb10cb10c69a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:25.864930 1150 raft_consensus.cc:383] T 5f52698fb56d472d9e91fb10cb10c69a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:25.864986 1150 raft_consensus.cc:738] T 5f52698fb56d472d9e91fb10cb10c69a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:25.865178 1150 consensus_queue.cc:260] T 5f52698fb56d472d9e91fb10cb10c69a P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:25.865926 1150 ts_tablet_manager.cc:1428] T 5f52698fb56d472d9e91fb10cb10c69a P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.054s user 0.003s sys 0.000s
I20250901 01:52:25.866851 1151 raft_consensus.cc:383] T ad45c90ba0da43c9860745e534664a98 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:25.867370 1151 raft_consensus.cc:738] T ad45c90ba0da43c9860745e534664a98 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:25.870136 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_762b9b27d3ba4a89a10fa3d26a9632f6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.869597 1151 consensus_queue.cc:260] T ad45c90ba0da43c9860745e534664a98 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:25.873248 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 6741c56499b44f648fb032c8cf50205b (table default.Table_eb9eaa97a468420bb672b720834c6ddd [id=c5667c5a65d045cea3cb52deb5801015]) successfully deleted
I20250901 01:52:25.873054 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_75e7aa29fff54731808a73d1301b8036"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.873811 2118 tablet_replica.cc:331] T 4902062977ea42149e3466e999117322 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:25.874004 2118 raft_consensus.cc:2241] T 4902062977ea42149e3466e999117322 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:25.874147 2118 raft_consensus.cc:2270] T 4902062977ea42149e3466e999117322 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250901 01:52:25.875028 1472 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet e8e7d2a72f704f52a8f8d46911a8a9e4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e8e7d2a72f704f52a8f8d46911a8a9e4 already in progress: creating tablet
W20250901 01:52:25.875275 1472 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet e2aa32450a584395ae07b49f76a7b26d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e2aa32450a584395ae07b49f76a7b26d already in progress: creating tablet
W20250901 01:52:25.875453 1472 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet becbaae4fd7e44faa679e9479465a7c4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet becbaae4fd7e44faa679e9479465a7c4 already in progress: creating tablet
W20250901 01:52:25.875670 1472 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 31614db4c53a4f3c96fd190d326ce601 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 31614db4c53a4f3c96fd190d326ce601 already in progress: creating tablet
W20250901 01:52:25.875852 1472 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 879fa66e55544118aafe504587393c66 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 879fa66e55544118aafe504587393c66 already in progress: creating tablet
I20250901 01:52:25.876286 2118 ts_tablet_manager.cc:1905] T 4902062977ea42149e3466e999117322 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:25.878173 992 leader_election.cc:336] T 3fda93767e494a4684978c8522b829dd P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:25.878293 992 leader_election.cc:304] T 3fda93767e494a4684978c8522b829dd P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:25.878595 2290 tablet.cc:1722] T 629db80a8ab444e582628578155638fd P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250901 01:52:25.878654 992 leader_election.cc:336] T 616ff34e8d314e0aa0e11ce60be11789 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:25.878865 992 leader_election.cc:336] T 9d886fa63f8f4996b087997b542e8bb1 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:25.878957 992 leader_election.cc:304] T 9d886fa63f8f4996b087997b542e8bb1 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:25.879104 2271 tablet.cc:1722] T b009cc95c819431ea9827a3e453b8da3 P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250901 01:52:25.879251 992 consensus_peers.cc:489] T 0cb3262973934c8689b851adad60afe0 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:25.879712 2305 raft_consensus.cc:2747] T 3fda93767e494a4684978c8522b829dd P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:25.879890 2305 raft_consensus.cc:2747] T 9d886fa63f8f4996b087997b542e8bb1 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:25.878031 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9018daeb007942d5b2c0a373788f1f37"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.883013 1981 raft_consensus.cc:491] T d0c3c363dda444519a17dca63ad9caa7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:25.883157 1981 raft_consensus.cc:513] T d0c3c363dda444519a17dca63ad9caa7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:25.883734 1981 leader_election.cc:290] T d0c3c363dda444519a17dca63ad9caa7 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:25.887709 992 leader_election.cc:336] T 64fb46b942ff4e4c8749986ecc3dfe65 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:25.889017 858 leader_election.cc:336] T d0c3c363dda444519a17dca63ad9caa7 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:25.889549 1151 ts_tablet_manager.cc:1428] T ad45c90ba0da43c9860745e534664a98 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.033s user 0.003s sys 0.000s
I20250901 01:52:25.891144 2271 tablet.cc:1722] T b2e8b2cd10154de08ddd30c214592a4a P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250901 01:52:25.891749 1151 tablet_bootstrap.cc:492] T e249474dcbf642189560c91ec3a99fdf P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:25.892134 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3a05594ff2884f40a2e0890471a032af"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.893631 2118 ts_tablet_manager.cc:1918] T 4902062977ea42149e3466e999117322 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:25.893798 2118 log.cc:1199] T 4902062977ea42149e3466e999117322 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/4902062977ea42149e3466e999117322
I20250901 01:52:25.894207 2118 ts_tablet_manager.cc:1939] T 4902062977ea42149e3466e999117322 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:25.894173 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b18bf94c70e3439e80ed6f4a4d58c6de"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.895495 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5da9a17fea564285920b19edc32d3c71"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.898784 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_451359fb06594161aa807b053f0148aa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.861889 2113 ts_tablet_manager.cc:1939] T e9b179dba2cc49428ba8090c962d6136 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:25.899878 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 39) took 1005 ms. Trace:
I20250901 01:52:25.900806 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_19375df5991b4fcb9ca544b18ceae32c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.901064 2113 rpcz_store.cc:276] 0901 01:52:24.894627 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:24.894675 (+ 48us) service_pool.cc:225] Handling call
0901 01:52:25.834557 (+939882us) ts_tablet_manager.cc:1145] Deleting tablet e9b179dba2cc49428ba8090c962d6136
0901 01:52:25.834661 (+ 104us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:25.849539 (+ 14878us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:25.860039 (+ 10500us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:25.899855 (+ 39816us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":939722}
I20250901 01:52:25.901574 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet e9b179dba2cc49428ba8090c962d6136 (table default.Table_271fd01373bf49f2ab8b4faf82db7393 [id=d409bc5c1aa548c69feacab89bd3d3ef]) successfully deleted
I20250901 01:52:25.901999 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 40) took 1006 ms. Trace:
I20250901 01:52:25.902110 2113 rpcz_store.cc:276] 0901 01:52:24.895633 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:24.895669 (+ 36us) service_pool.cc:225] Handling call
0901 01:52:25.901871 (+1006202us) ts_tablet_manager.cc:1145] Deleting tablet 05630a72ddae4de0b890dece2feacc31
0901 01:52:25.901883 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:25.901981 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1006077}
W20250901 01:52:25.902668 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 05630a72ddae4de0b890dece2feacc31 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 05630a72ddae4de0b890dece2feacc31 already in progress: creating tablet
I20250901 01:52:25.903164 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3618c5873be54618a17d195b865415fd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.896884 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4855448edb4b40c184838aa7f81c112b" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:25.904115 913 raft_consensus.cc:3058] T 4855448edb4b40c184838aa7f81c112b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:25.904306 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 41) took 1007 ms. Trace:
I20250901 01:52:25.904424 2113 rpcz_store.cc:276] 0901 01:52:24.896778 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:24.896821 (+ 43us) service_pool.cc:225] Handling call
0901 01:52:25.903458 (+1006637us) ts_tablet_manager.cc:1145] Deleting tablet c58edaa76ebf47dfb1d9d5fab6de7dfa
0901 01:52:25.903469 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:25.904278 (+ 809us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1006500}
I20250901 01:52:25.897805 969 heartbeater.cc:499] Master 127.0.104.60:44231 was elected leader, sending a full tablet report...
I20250901 01:52:25.906244 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_abf4fd5e031e4cf887fe8e186e6df782"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:25.906723 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet c58edaa76ebf47dfb1d9d5fab6de7dfa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c58edaa76ebf47dfb1d9d5fab6de7dfa already in progress: creating tablet
I20250901 01:52:25.907084 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 42) took 1009 ms. Trace:
I20250901 01:52:25.907088 913 raft_consensus.cc:2466] T 4855448edb4b40c184838aa7f81c112b P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:25.907196 2113 rpcz_store.cc:276] 0901 01:52:24.897375 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:24.897417 (+ 42us) service_pool.cc:225] Handling call
0901 01:52:25.906983 (+1009566us) ts_tablet_manager.cc:1145] Deleting tablet e249474dcbf642189560c91ec3a99fdf
0901 01:52:25.906993 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:25.907068 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1009420}
I20250901 01:52:25.907606 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 4902062977ea42149e3466e999117322 (table default.Table_fb5ee5a5a9514ddaac740c194a820fd2 [id=48b7b856dca841eba100ba8acf02b7e2]) successfully deleted
W20250901 01:52:25.908066 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet e249474dcbf642189560c91ec3a99fdf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e249474dcbf642189560c91ec3a99fdf already in progress: creating tablet
I20250901 01:52:25.908334 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c046041f08d447098558c53ac392ea02"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.907668 2118 tablet_replica.cc:331] T b81a12b38ac64efabd44aeb192d181a0 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:25.908792 2118 raft_consensus.cc:2241] T b81a12b38ac64efabd44aeb192d181a0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:25.908456 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 43) took 1009 ms. Trace:
I20250901 01:52:25.908917 2113 rpcz_store.cc:276] 0901 01:52:24.898563 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:24.898607 (+ 44us) service_pool.cc:225] Handling call
0901 01:52:25.908319 (+1009712us) ts_tablet_manager.cc:1145] Deleting tablet eaa521c52a2f4408b59770bd2a05907c
0901 01:52:25.908329 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:25.908440 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1009564}
I20250901 01:52:25.909181 2118 raft_consensus.cc:2270] T b81a12b38ac64efabd44aeb192d181a0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250901 01:52:25.909510 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet eaa521c52a2f4408b59770bd2a05907c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eaa521c52a2f4408b59770bd2a05907c already in progress: creating tablet
I20250901 01:52:25.909286 2271 tablet.cc:1722] T 5adf6b492640417ca071c94702b48cb9 P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250901 01:52:25.910068 2113 tablet_replica.cc:331] T 17e8340dbd1a406cb782dad3fe88cc7e P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:25.910367 1150 tablet_bootstrap.cc:492] T d1f2d98712ff4ea6aec665512de068bd P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:25.910609 2113 raft_consensus.cc:2241] T 17e8340dbd1a406cb782dad3fe88cc7e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:25.912081 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_6072b33579d0482eb9b35e546595f147"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.913072 2290 tablet.cc:1722] T e6c10426f60f43e8bde69d9b1b9c8eb6 P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:25.913582 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ea2d28fa112c4e0e87c6d5708f394e73"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:25.916314 1490 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_0e63d21acfcb4c028c9ea6406b6a04e4" } modify_external_catalogs: true
I20250901 01:52:25.916579 1490 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_0e63d21acfcb4c028c9ea6406b6a04e4" } modify_external_catalogs: true
I20250901 01:52:25.917409 2113 raft_consensus.cc:2270] T 17e8340dbd1a406cb782dad3fe88cc7e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:25.913451 2118 ts_tablet_manager.cc:1905] T b81a12b38ac64efabd44aeb192d181a0 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:25.911635 1151 tablet_bootstrap.cc:654] T e249474dcbf642189560c91ec3a99fdf P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:25.919859 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e4d972e8254cf899c3e0b71b6e2bd9" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:25.920060 913 raft_consensus.cc:2466] T 15e4d972e8254cf899c3e0b71b6e2bd9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:25.922042 1150 tablet_bootstrap.cc:654] T d1f2d98712ff4ea6aec665512de068bd P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:25.923848 2306 raft_consensus.cc:491] T f80f13db501c42c7950499df8fcb6fc8 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:25.923985 2306 raft_consensus.cc:513] T f80f13db501c42c7950499df8fcb6fc8 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:25.924692 2306 leader_election.cc:290] T f80f13db501c42c7950499df8fcb6fc8 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:25.927142 2271 tablet.cc:1722] T 3a8231590ea643b0a1f83eec4d9a8191 P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:25.928972 2113 ts_tablet_manager.cc:1905] T 17e8340dbd1a406cb782dad3fe88cc7e P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:25.933996 1485 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_a7fa05538bbd4766aa24d557677a692c" } modify_external_catalogs: true
I20250901 01:52:25.934782 2118 ts_tablet_manager.cc:1918] T b81a12b38ac64efabd44aeb192d181a0 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:25.940879 1489 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_f44c609420054cebad09692be9aa6ecd" } new_table_name: "default.Table_dd01ae355f73448cbca96f086b04a2fe" modify_external_catalogs: true
I20250901 01:52:25.941188 1151 tablet_bootstrap.cc:492] T e249474dcbf642189560c91ec3a99fdf P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:25.941375 1151 ts_tablet_manager.cc:1397] T e249474dcbf642189560c91ec3a99fdf P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.050s user 0.002s sys 0.004s
I20250901 01:52:25.942121 1151 raft_consensus.cc:357] T e249474dcbf642189560c91ec3a99fdf P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:25.942298 1151 raft_consensus.cc:383] T e249474dcbf642189560c91ec3a99fdf P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:25.942370 1151 raft_consensus.cc:738] T e249474dcbf642189560c91ec3a99fdf P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:25.942621 1151 consensus_queue.cc:260] T e249474dcbf642189560c91ec3a99fdf P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:25.943162 1151 ts_tablet_manager.cc:1428] T e249474dcbf642189560c91ec3a99fdf P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250901 01:52:25.943122 1492 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_4185b4f73a2743e08f24a26878cf73ca" } new_table_name: "default.Table_6d47266af0ed433eb1f2abbc81d18e69" modify_external_catalogs: true
I20250901 01:52:25.943671 1151 tablet_bootstrap.cc:492] T c754129977904f978f87fd89d7092e36 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:25.945451 1151 tablet_bootstrap.cc:654] T c754129977904f978f87fd89d7092e36 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:25.943029 917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "42c1156e8dd84c6fa1396b46de13eb6f" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:25.945621 917 raft_consensus.cc:3058] T 42c1156e8dd84c6fa1396b46de13eb6f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:25.946260 1488 master_service.cc:948] ReplaceTablet: received request to replace tablet 75a703c8e20a4a2fa0ab6ec58d41ef92 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:25.948105 917 raft_consensus.cc:2466] T 42c1156e8dd84c6fa1396b46de13eb6f P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:25.945258 2271 tablet.cc:1722] T f62d96c37cca4143a4b098b7c69979e6 P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:25.949563 1484 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_dc9c5d18c7ec478592a76dbd76424b88" } new_table_name: "default.Table_b989608c88204be2b6f706a9426c47bb" modify_external_catalogs: true
I20250901 01:52:25.951613 2118 log.cc:1199] T b81a12b38ac64efabd44aeb192d181a0 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/b81a12b38ac64efabd44aeb192d181a0
I20250901 01:52:25.951980 2118 ts_tablet_manager.cc:1939] T b81a12b38ac64efabd44aeb192d181a0 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:25.952841 2118 tablet_replica.cc:331] T 48a211e174054866b030cdfb0cf947f4 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:25.952925 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet b81a12b38ac64efabd44aeb192d181a0 (table default.Table_e21873578fc14cc3b67e08478ec42192 [id=a404c32b3e8146f288ce7ea3a017ab96]) successfully deleted
I20250901 01:52:25.953085 2118 raft_consensus.cc:2241] T 48a211e174054866b030cdfb0cf947f4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:25.953238 2118 raft_consensus.cc:2270] T 48a211e174054866b030cdfb0cf947f4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:25.954860 2301 tablet.cc:1722] T 629db80a8ab444e582628578155638fd P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:25.952203 2216 raft_consensus.cc:491] T f50f33de1b654663b9ef8a91e3436f11 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:25.952697 1993 raft_consensus.cc:491] T d0c3c363dda444519a17dca63ad9caa7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:25.955273 1055 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0c3c363dda444519a17dca63ad9caa7" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:25.955377 1993 raft_consensus.cc:513] T d0c3c363dda444519a17dca63ad9caa7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:25.953229 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abf6e3be60f74872b2308dc95bf9cd37" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:25.955713 1055 raft_consensus.cc:2466] T d0c3c363dda444519a17dca63ad9caa7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00c282f779c1477d920f24dc0dc16628 in term 0.
I20250901 01:52:25.955946 1993 leader_election.cc:290] T d0c3c363dda444519a17dca63ad9caa7 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:25.957964 2277 tablet.cc:1722] T b009cc95c819431ea9827a3e453b8da3 P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250901 01:52:25.958094 992 leader_election.cc:336] T 20f7bfab18134e079e91c52c89468cc3 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:25.958534 2216 raft_consensus.cc:513] T f50f33de1b654663b9ef8a91e3436f11 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
W20250901 01:52:25.959010 992 leader_election.cc:336] T 42c1156e8dd84c6fa1396b46de13eb6f P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:25.959370 992 consensus_peers.cc:489] T e1d18e999b5e42b1873631e0f1445081 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:25.959452 2118 ts_tablet_manager.cc:1905] T 48a211e174054866b030cdfb0cf947f4 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:25.964483 1492 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 858218c4427c41e980b2b5c969a5af40 consensus state has no leader
I20250901 01:52:25.964638 2216 leader_election.cc:290] T f50f33de1b654663b9ef8a91e3436f11 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:25.968420 2216 raft_consensus.cc:491] T c95798f0daf4437db18a291c4464522e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:25.968858 2216 raft_consensus.cc:513] T c95798f0daf4437db18a291c4464522e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:25.954147 1150 tablet_bootstrap.cc:492] T d1f2d98712ff4ea6aec665512de068bd P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:25.969213 1150 ts_tablet_manager.cc:1397] T d1f2d98712ff4ea6aec665512de068bd P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.059s user 0.002s sys 0.004s
I20250901 01:52:25.967787 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f80f13db501c42c7950499df8fcb6fc8" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:25.969705 913 raft_consensus.cc:2466] T f80f13db501c42c7950499df8fcb6fc8 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:25.970132 1150 raft_consensus.cc:357] T d1f2d98712ff4ea6aec665512de068bd P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:25.970310 1489 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7954393c8c1c4457ab8bb723c03a9c8b consensus state has no leader
I20250901 01:52:25.970353 1150 raft_consensus.cc:383] T d1f2d98712ff4ea6aec665512de068bd P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:25.970412 1150 raft_consensus.cc:738] T d1f2d98712ff4ea6aec665512de068bd P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:25.970629 1150 consensus_queue.cc:260] T d1f2d98712ff4ea6aec665512de068bd P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:25.971241 1150 ts_tablet_manager.cc:1428] T d1f2d98712ff4ea6aec665512de068bd P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20250901 01:52:25.971889 859 leader_election.cc:343] T abf6e3be60f74872b2308dc95bf9cd37 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:25.972020 859 leader_election.cc:304] T abf6e3be60f74872b2308dc95bf9cd37 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:25.972290 2123 raft_consensus.cc:2747] T abf6e3be60f74872b2308dc95bf9cd37 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:25.972388 859 leader_election.cc:304] T d0c3c363dda444519a17dca63ad9caa7 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:25.971894 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0c3c363dda444519a17dca63ad9caa7" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:25.972730 913 raft_consensus.cc:2466] T d0c3c363dda444519a17dca63ad9caa7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:25.972860 2123 raft_consensus.cc:2802] T d0c3c363dda444519a17dca63ad9caa7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:25.972940 2123 raft_consensus.cc:491] T d0c3c363dda444519a17dca63ad9caa7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:25.972941 2216 leader_election.cc:290] T c95798f0daf4437db18a291c4464522e P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:25.972997 2123 raft_consensus.cc:3058] T d0c3c363dda444519a17dca63ad9caa7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
W20250901 01:52:25.974009 1484 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 435b1cca5a7d4d49959a310cb250b2a0 consensus state has no leader
I20250901 01:52:25.975519 2123 raft_consensus.cc:513] T d0c3c363dda444519a17dca63ad9caa7 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:25.975832 1485 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_a7fa05538bbd4766aa24d557677a692c" } modify_external_catalogs: true
I20250901 01:52:25.977648 2123 leader_election.cc:290] T d0c3c363dda444519a17dca63ad9caa7 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:25.977910 858 leader_election.cc:336] T d0c3c363dda444519a17dca63ad9caa7 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:25.979125 1150 tablet_bootstrap.cc:492] T 06d5eeb70e664f73bf643922d445bc7a P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:25.979725 1490 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 2599f98abfdc4871b573edfc2ad97d22
W20250901 01:52:25.979979 1490 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 2599f98abfdc4871b573edfc2ad97d22: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:25.980281 1488 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 75a703c8e20a4a2fa0ab6ec58d41ef92
W20250901 01:52:25.981181 1488 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 75a703c8e20a4a2fa0ab6ec58d41ef92: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:25.981143 1993 consensus_queue.cc:1035] T 50e3da53433f4c14b57db8a733fa53bc P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:25.981560 1488 catalog_manager.cc:6652] ReplaceTablet: tablet 75a703c8e20a4a2fa0ab6ec58d41ef92 deleted and replaced by tablet 9745582cebfe47b58c21d0191ab4cb9b
I20250901 01:52:25.982896 900 tablet_service.cc:1515] Processing DeleteTablet for tablet 75a703c8e20a4a2fa0ab6ec58d41ef92 with delete_type TABLET_DATA_DELETED (replaced by tablet 9745582cebfe47b58c21d0191ab4cb9b) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.983091 897 tablet_service.cc:1515] Processing DeleteTablet for tablet e8e7d2a72f704f52a8f8d46911a8a9e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.983260 886 tablet_service.cc:1515] Processing DeleteTablet for tablet e2aa32450a584395ae07b49f76a7b26d with delete_type TABLET_DATA_DELETED (replaced by tablet 7304937048ca48a7bd153f3cdcfaeff0) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.983419 893 tablet_service.cc:1515] Processing DeleteTablet for tablet becbaae4fd7e44faa679e9479465a7c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:23 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.983575 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 31614db4c53a4f3c96fd190d326ce601 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.984671 901 tablet_service.cc:1515] Processing DeleteTablet for tablet 2599f98abfdc4871b573edfc2ad97d22 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:25 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.984882 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 6741c56499b44f648fb032c8cf50205b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.985026 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 879fa66e55544118aafe504587393c66 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:23 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.985173 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 4902062977ea42149e3466e999117322 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:18 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.985299 898 tablet_service.cc:1515] Processing DeleteTablet for tablet b81a12b38ac64efabd44aeb192d181a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:18 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.985499 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet 2599f98abfdc4871b573edfc2ad97d22 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:25 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:25.986383 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet 75a703c8e20a4a2fa0ab6ec58d41ef92 with delete_type TABLET_DATA_DELETED (replaced by tablet 9745582cebfe47b58c21d0191ab4cb9b) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:25.986868 992 leader_election.cc:304] T 4855448edb4b40c184838aa7f81c112b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:25.987532 1993 raft_consensus.cc:2802] T 4855448edb4b40c184838aa7f81c112b P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:25.987730 1993 raft_consensus.cc:695] T 4855448edb4b40c184838aa7f81c112b P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:25.987840 2306 raft_consensus.cc:491] T b73e362321284df1b18ef755dfe930a5 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:25.987949 2306 raft_consensus.cc:513] T b73e362321284df1b18ef755dfe930a5 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:25.988059 1993 consensus_queue.cc:237] T 4855448edb4b40c184838aa7f81c112b P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:25.988540 2306 leader_election.cc:290] T b73e362321284df1b18ef755dfe930a5 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:25.989493 898 tablet_service.cc:1515] Processing DeleteTablet for tablet 5e2472ee5d6e4d3bbf1f24027ba531ca with delete_type TABLET_DATA_DELETED (replaced by tablet 0813cc68d4b0488ba60a523e4479971b) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.990038 1150 tablet_bootstrap.cc:654] T 06d5eeb70e664f73bf643922d445bc7a P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:25.996105 898 tablet_service.cc:1515] Processing DeleteTablet for tablet 438ae684af234c868a419fbab43b0b98 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:24 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:25.998031 1151 tablet_bootstrap.cc:492] T c754129977904f978f87fd89d7092e36 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:25.998224 1151 ts_tablet_manager.cc:1397] T c754129977904f978f87fd89d7092e36 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.055s user 0.000s sys 0.004s
I20250901 01:52:25.999172 1151 raft_consensus.cc:357] T c754129977904f978f87fd89d7092e36 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:25.999531 1151 raft_consensus.cc:383] T c754129977904f978f87fd89d7092e36 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:25.999650 1151 raft_consensus.cc:738] T c754129977904f978f87fd89d7092e36 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:25.999771 1488 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_2ddf898959684cdc995c1351d84a9661" } modify_external_catalogs: true
I20250901 01:52:25.999884 1151 consensus_queue.cc:260] T c754129977904f978f87fd89d7092e36 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:26.000025 1488 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_2ddf898959684cdc995c1351d84a9661" } modify_external_catalogs: true
I20250901 01:52:26.000571 1151 ts_tablet_manager.cc:1428] T c754129977904f978f87fd89d7092e36 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20250901 01:52:26.000752 992 leader_election.cc:343] T abe27e749fa0485195e0064a49d5431e P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: abe27e749fa0485195e0064a49d5431e
I20250901 01:52:26.001394 992 leader_election.cc:304] T 15e4d972e8254cf899c3e0b71b6e2bd9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:26.001008 1151 tablet_bootstrap.cc:492] T 7a2f6e2c7ee2402ea310c058e046416c P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:26.002161 2216 raft_consensus.cc:2802] T 15e4d972e8254cf899c3e0b71b6e2bd9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:26.002357 2216 raft_consensus.cc:491] T 15e4d972e8254cf899c3e0b71b6e2bd9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:26.002511 2216 raft_consensus.cc:3058] T 15e4d972e8254cf899c3e0b71b6e2bd9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:26.003243 898 tablet_service.cc:1515] Processing DeleteTablet for tablet 88ceba7f48b04d7885cc0164668700a5 with delete_type TABLET_DATA_DELETED (replaced by tablet 9074543dea9949e7a085717b40913348) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:26.003567 2306 raft_consensus.cc:491] T 66a48a8ebe154b2cac89c234eaa378ff P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:26.003670 2306 raft_consensus.cc:513] T 66a48a8ebe154b2cac89c234eaa378ff P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:26.003679 898 tablet_service.cc:1515] Processing DeleteTablet for tablet 617cc33d0d23439cb5f1e0c31bf11281 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:24 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:26.004141 2306 leader_election.cc:290] T 66a48a8ebe154b2cac89c234eaa378ff P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:26.004611 992 leader_election.cc:343] T 4a87e68850c4459eb2c8eafac89c6aa9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 4a87e68850c4459eb2c8eafac89c6aa9
I20250901 01:52:26.005125 2216 raft_consensus.cc:513] T 15e4d972e8254cf899c3e0b71b6e2bd9 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:26.005745 2216 leader_election.cc:290] T 15e4d972e8254cf899c3e0b71b6e2bd9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:26.007474 1151 tablet_bootstrap.cc:654] T 7a2f6e2c7ee2402ea310c058e046416c P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:26.008644 2118 ts_tablet_manager.cc:1918] T 48a211e174054866b030cdfb0cf947f4 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:26.009018 2118 log.cc:1199] T 48a211e174054866b030cdfb0cf947f4 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/48a211e174054866b030cdfb0cf947f4
I20250901 01:52:26.009328 2323 raft_consensus.cc:491] T 8ed6616ce54548ba83ba299c4ecf3c38 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:26.009564 2323 raft_consensus.cc:513] T 8ed6616ce54548ba83ba299c4ecf3c38 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:26.009778 2118 ts_tablet_manager.cc:1939] T 48a211e174054866b030cdfb0cf947f4 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:26.010226 2323 leader_election.cc:290] T 8ed6616ce54548ba83ba299c4ecf3c38 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:26.010524 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 48a211e174054866b030cdfb0cf947f4 (table default.Table_7c23fc28bbae4efe982c240b87f40956 [id=65d035b79f004c3d9ac3b0cd6a515d45]) successfully deleted
W20250901 01:52:26.005177 992 leader_election.cc:343] T 98cca86164db4252b1b49de88747822a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 98cca86164db4252b1b49de88747822a
W20250901 01:52:26.011677 858 leader_election.cc:336] T 8ed6616ce54548ba83ba299c4ecf3c38 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:26.012032 898 tablet_service.cc:1515] Processing DeleteTablet for tablet 246b33338de647689d7404c70676e76f with delete_type TABLET_DATA_DELETED (replaced by tablet ae4844183a3c4a04a775e6fd878391a3) from {username='slave'} at 127.0.0.1:34472
W20250901 01:52:26.012317 992 leader_election.cc:343] T ef58f477ebeb49639ed7d071e1a62c10 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: ef58f477ebeb49639ed7d071e1a62c10
I20250901 01:52:26.013263 898 tablet_service.cc:1515] Processing DeleteTablet for tablet b11e8f77243e4764b96548d1132b8e5a with delete_type TABLET_DATA_DELETED (replaced by tablet 96824e68bbf24b8287dbb683038ffaba) from {username='slave'} at 127.0.0.1:34472
W20250901 01:52:26.013856 992 leader_election.cc:343] T 6741c56499b44f648fb032c8cf50205b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 6741c56499b44f648fb032c8cf50205b
I20250901 01:52:26.015236 1488 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet f62d96c37cca4143a4b098b7c69979e6
W20250901 01:52:26.015558 1488 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet f62d96c37cca4143a4b098b7c69979e6: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:26.016156 898 tablet_service.cc:1515] Processing DeleteTablet for tablet 4410da9e95454ecf95faf80221b0a71c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:24 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:26.016386 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 3c12d6a70ac24dce82e851782e6fe5aa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:24 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:26.016803 888 tablet_service.cc:1515] Processing DeleteTablet for tablet f62d96c37cca4143a4b098b7c69979e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:25 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:26.014293 1151 tablet_bootstrap.cc:492] T 7a2f6e2c7ee2402ea310c058e046416c P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:26.016659 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet f62d96c37cca4143a4b098b7c69979e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:25 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:26.018023 1151 ts_tablet_manager.cc:1397] T 7a2f6e2c7ee2402ea310c058e046416c P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20250901 01:52:26.018079 1150 tablet_bootstrap.cc:492] T 06d5eeb70e664f73bf643922d445bc7a P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
W20250901 01:52:26.018045 1472 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet b85d64e02cc646e380581ac9f59dee9e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b85d64e02cc646e380581ac9f59dee9e already in progress: creating tablet
I20250901 01:52:26.018234 1150 ts_tablet_manager.cc:1397] T 06d5eeb70e664f73bf643922d445bc7a P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.039s user 0.001s sys 0.004s
I20250901 01:52:26.018414 1485 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet f9b293bf6a17453b9ea9892a6970d0b0
W20250901 01:52:26.018705 1485 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet f9b293bf6a17453b9ea9892a6970d0b0: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:26.019207 1151 raft_consensus.cc:357] T 7a2f6e2c7ee2402ea310c058e046416c P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:26.019526 2118 tablet_replica.cc:331] T 1a60c10e42104502ac2d3b80b8126ceb P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:26.020299 2118 raft_consensus.cc:2241] T 1a60c10e42104502ac2d3b80b8126ceb P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:26.020684 2118 raft_consensus.cc:2270] T 1a60c10e42104502ac2d3b80b8126ceb P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:26.020015 888 tablet_service.cc:1515] Processing DeleteTablet for tablet f9b293bf6a17453b9ea9892a6970d0b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:25 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:26.021334 2118 ts_tablet_manager.cc:1905] T 1a60c10e42104502ac2d3b80b8126ceb P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:26.019340 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet f9b293bf6a17453b9ea9892a6970d0b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:25 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:26.019713 992 leader_election.cc:304] T 42c1156e8dd84c6fa1396b46de13eb6f P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:26.022003 1761 raft_consensus.cc:2802] T 42c1156e8dd84c6fa1396b46de13eb6f P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:26.022117 1761 raft_consensus.cc:695] T 42c1156e8dd84c6fa1396b46de13eb6f P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:26.022329 1761 consensus_queue.cc:237] T 42c1156e8dd84c6fa1396b46de13eb6f P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
W20250901 01:52:26.026662 1486 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:54603 (request call id 180) took 1133 ms (client timeout 1000 ms). Trace:
I20250901 01:52:26.044826 1151 raft_consensus.cc:383] T 7a2f6e2c7ee2402ea310c058e046416c P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:26.048978 1488 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_604f2c5c58d74154bacf2a065a467f32" } modify_external_catalogs: true
I20250901 01:52:26.049437 1488 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_604f2c5c58d74154bacf2a065a467f32" } modify_external_catalogs: true
W20250901 01:52:26.049386 1486 rpcz_store.cc:269] 0901 01:52:24.893170 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:24.893237 (+ 67us) service_pool.cc:225] Handling call
0901 01:52:24.965105 (+ 71868us) spinlock_profiling.cc:243] Waited 65.7 ms on lock 0x616000040c54. stack: 00007f0d64ec57d6 00007f0d64ec5590 00007f0d63eb83ad 0000000000972d94 0000000000972d5d 00007f0d73c9014d 00007f0d73c90243 00007f0d7094f941 00007f0d784daa5d 00007f0d7865e6d8 00007f0d6f498c86 00007f0d6f4989b5 00007f0d6bc732ce 00007f0d6bc721ae 00007f0d6bc783e5 00007f0d6bc78d0f
0901 01:52:26.026620 (+1061515us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0901 01:52:25.707915 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:25.708006 (+ 91us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:25.708012 (+ 6us) write_op.cc:435] Acquired schema lock
0901 01:52:25.708015 (+ 3us) tablet.cc:660] Decoding operations
0901 01:52:25.708126 (+ 111us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:25.708139 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:25.708144 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0901 01:52:25.708203 (+ 59us) tablet.cc:699] Row locks acquired
0901 01:52:25.708205 (+ 2us) write_op.cc:260] PREPARE: finished
0901 01:52:25.708240 (+ 35us) write_op.cc:270] Start()
0901 01:52:25.708276 (+ 36us) write_op.cc:276] Timestamp: P: 1756691545708203 usec, L: 0
0901 01:52:25.708279 (+ 3us) op_driver.cc:348] REPLICATION: starting
0901 01:52:25.708480 (+ 201us) log.cc:844] Serialized 356 byte log entry
0901 01:52:25.712160 (+ 3680us) op_driver.cc:464] REPLICATION: finished
0901 01:52:25.712273 (+ 113us) write_op.cc:301] APPLY: starting
0901 01:52:25.712288 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:25.712336 (+ 48us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:25.712366 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:25.712440 (+ 74us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:25.712485 (+ 45us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0901 01:52:25.712493 (+ 8us) write_op.cc:312] APPLY: finished
0901 01:52:25.712569 (+ 76us) log.cc:844] Serialized 23 byte log entry
0901 01:52:25.712639 (+ 70us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:25.712658 (+ 19us) write_op.cc:454] Released schema lock
0901 01:52:25.712664 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0901 01:52:25.712724 (+ 60us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:25.712727 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":26973,"reported_tablets":212,"spinlock_wait_cycles":151184640,"child_traces":[["op",{"apply.queue_time_us":66,"apply.run_cpu_time_us":444,"apply.run_wall_time_us":443,"num_ops":1,"prepare.queue_time_us":3484,"prepare.run_cpu_time_us":659,"prepare.run_wall_time_us":659,"raft.queue_time_us":401,"raft.run_cpu_time_us":191,"raft.run_wall_time_us":192,"replication_time_us":3850}]]}
I20250901 01:52:26.062626 1488 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 72ba689f9230411995dc96e2b45601df
W20250901 01:52:26.063010 1488 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 72ba689f9230411995dc96e2b45601df: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:26.076393 2221 log.cc:826] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Log is configured to *not* fsync() on all Append() calls
I20250901 01:52:26.078161 1151 raft_consensus.cc:738] T 7a2f6e2c7ee2402ea310c058e046416c P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:26.078653 1151 consensus_queue.cc:260] T 7a2f6e2c7ee2402ea310c058e046416c P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:26.019577 1150 raft_consensus.cc:357] T 06d5eeb70e664f73bf643922d445bc7a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:26.025266 888 tablet_service.cc:1515] Processing DeleteTablet for tablet c754129977904f978f87fd89d7092e36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:24 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:26.079564 1150 raft_consensus.cc:383] T 06d5eeb70e664f73bf643922d445bc7a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:26.079641 1150 raft_consensus.cc:738] T 06d5eeb70e664f73bf643922d445bc7a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:26.079866 1150 consensus_queue.cc:260] T 06d5eeb70e664f73bf643922d445bc7a P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:26.080515 1150 ts_tablet_manager.cc:1428] T 06d5eeb70e664f73bf643922d445bc7a P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.062s user 0.001s sys 0.001s
I20250901 01:52:26.080582 1151 ts_tablet_manager.cc:1428] T 7a2f6e2c7ee2402ea310c058e046416c P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.062s user 0.001s sys 0.001s
I20250901 01:52:26.080811 1150 tablet_bootstrap.cc:492] T 74b709cb5ec64006b2f12bead8217d80 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:26.080828 1151 tablet_bootstrap.cc:492] T 8ca5cf4e99014c0d84a3acad4ef713af P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:26.045998 2210 raft_consensus.cc:491] T 82d163681fda47f38d79863bd28f576d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:26.081108 2210 raft_consensus.cc:513] T 82d163681fda47f38d79863bd28f576d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:26.081157 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_44216a3412b24714b5f314d00f976ba8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.081645 2210 leader_election.cc:290] T 82d163681fda47f38d79863bd28f576d P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:26.081617 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_519a01e6e5894c48b67c7056e326ae50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.063728 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 72ba689f9230411995dc96e2b45601df with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:26 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:26.081935 1486 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_52733c9a363b4ed9b6ff11ad8571e421" } new_table_name: "default.Table_d1302673d68f424ebb54b220911ce385" modify_external_catalogs: true
I20250901 01:52:26.082087 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_098ca671d0914d6697c883af23dbac2f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.066757 1993 raft_consensus.cc:491] T 81c36693199945d98465579b509852e2 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:26.082633 1993 raft_consensus.cc:513] T 81c36693199945d98465579b509852e2 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:26.083160 1993 leader_election.cc:290] T 81c36693199945d98465579b509852e2 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:26.083266 1490 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_9f48256fcac7429ba7d7fdb86979eda6" } new_table_name: "default.Table_af5338b09bbc4c879c7f4f573c32fd8d" modify_external_catalogs: true
I20250901 01:52:26.084070 1483 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_37dedd79eacd4c56ac546fe8b1902259" } modify_external_catalogs: true
I20250901 01:52:26.084259 1483 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_37dedd79eacd4c56ac546fe8b1902259" } modify_external_catalogs: true
I20250901 01:52:26.068369 2306 raft_consensus.cc:491] T 46bfbdff3f904715a8c24907b20fcab0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:26.084667 2306 raft_consensus.cc:513] T 46bfbdff3f904715a8c24907b20fcab0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:26.084673 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_eb8f3054932445148c9a25036ce5a71b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.085204 2306 leader_election.cc:290] T 46bfbdff3f904715a8c24907b20fcab0 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:26.087622 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d051d1fec4ee47dab4ea647713eae323"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.087695 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a60cb4bcc16b4d7f9a9f51140453946b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.073971 1103 heartbeater.cc:499] Master 127.0.104.60:44231 was elected leader, sending a full tablet report...
I20250901 01:52:26.088320 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_315f2984b9f04498959ec00f4e2ad312"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.045637 992 leader_election.cc:304] T f80f13db501c42c7950499df8fcb6fc8 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:26.088349 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9cb8513792db435c8241521b999f8ca0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.088793 2306 raft_consensus.cc:2802] T f80f13db501c42c7950499df8fcb6fc8 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:26.088864 2306 raft_consensus.cc:491] T f80f13db501c42c7950499df8fcb6fc8 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:26.088927 2306 raft_consensus.cc:3058] T f80f13db501c42c7950499df8fcb6fc8 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:26.088995 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9736817eb19f4136a0ba995054e9e487"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.089098 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_86f2fa7f663e4f55a7ce49459df2c97c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.089357 1489 master_service.cc:948] ReplaceTablet: received request to replace tablet 606fac5c6efe48d1877104b01c23ba5b from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:26.089521 992 leader_election.cc:304] T d0c3c363dda444519a17dca63ad9caa7 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:26.090473 1872 raft_consensus.cc:2802] T d0c3c363dda444519a17dca63ad9caa7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:26.090606 1872 raft_consensus.cc:491] T d0c3c363dda444519a17dca63ad9caa7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:26.090755 1872 raft_consensus.cc:3058] T d0c3c363dda444519a17dca63ad9caa7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:26.091055 2271 tablet.cc:1722] T b009cc95c819431ea9827a3e453b8da3 P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250901 01:52:26.091979 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 72ba689f9230411995dc96e2b45601df with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:26 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:26.093562 1993 raft_consensus.cc:491] T ad45c90ba0da43c9860745e534664a98 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:26.109357 1150 tablet_bootstrap.cc:654] T 74b709cb5ec64006b2f12bead8217d80 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:26.110677 2118 ts_tablet_manager.cc:1918] T 1a60c10e42104502ac2d3b80b8126ceb P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:26.110795 2118 log.cc:1199] T 1a60c10e42104502ac2d3b80b8126ceb P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/1a60c10e42104502ac2d3b80b8126ceb
I20250901 01:52:26.111093 2118 ts_tablet_manager.cc:1939] T 1a60c10e42104502ac2d3b80b8126ceb P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:26.112725 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 1a60c10e42104502ac2d3b80b8126ceb (table default.Table_53e30205c5fd488eb3b48c685a159d04 [id=60837b865ee7415ea42a85a953d837a5]) successfully deleted
I20250901 01:52:26.093668 1993 raft_consensus.cc:513] T ad45c90ba0da43c9860745e534664a98 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:26.125787 1993 leader_election.cc:290] T ad45c90ba0da43c9860745e534664a98 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:26.126533 992 leader_election.cc:336] T ef58f477ebeb49639ed7d071e1a62c10 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:26.126714 992 leader_election.cc:304] T ef58f477ebeb49639ed7d071e1a62c10 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:26.126974 1993 raft_consensus.cc:2747] T ef58f477ebeb49639ed7d071e1a62c10 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:26.127245 2306 raft_consensus.cc:513] T f80f13db501c42c7950499df8fcb6fc8 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:26.127337 992 leader_election.cc:336] T 98cca86164db4252b1b49de88747822a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:26.127419 992 leader_election.cc:304] T 98cca86164db4252b1b49de88747822a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:26.127585 1993 raft_consensus.cc:2747] T 98cca86164db4252b1b49de88747822a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:26.127775 2306 leader_election.cc:290] T f80f13db501c42c7950499df8fcb6fc8 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:26.127844 992 leader_election.cc:336] T 4a87e68850c4459eb2c8eafac89c6aa9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:26.127918 992 leader_election.cc:304] T 4a87e68850c4459eb2c8eafac89c6aa9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
W20250901 01:52:26.128201 992 leader_election.cc:336] T 4855448edb4b40c184838aa7f81c112b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:26.128364 2306 raft_consensus.cc:2747] T 4a87e68850c4459eb2c8eafac89c6aa9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:26.131590 1872 raft_consensus.cc:513] T d0c3c363dda444519a17dca63ad9caa7 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:26.132156 1872 leader_election.cc:290] T d0c3c363dda444519a17dca63ad9caa7 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:26.133679 1151 tablet_bootstrap.cc:654] T 8ca5cf4e99014c0d84a3acad4ef713af P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:26.134297 992 leader_election.cc:336] T 42c1156e8dd84c6fa1396b46de13eb6f P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:26.134521 2118 tablet_replica.cc:331] T 56475ba491a24301a534304811f30140 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
W20250901 01:52:26.134538 992 leader_election.cc:336] T 6741c56499b44f648fb032c8cf50205b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:26.134622 992 leader_election.cc:304] T 6741c56499b44f648fb032c8cf50205b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:26.134960 2306 raft_consensus.cc:2747] T 6741c56499b44f648fb032c8cf50205b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:26.134953 1872 raft_consensus.cc:491] T e249474dcbf642189560c91ec3a99fdf P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:26.135193 2118 raft_consensus.cc:2241] T 56475ba491a24301a534304811f30140 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:26.135263 1872 raft_consensus.cc:513] T e249474dcbf642189560c91ec3a99fdf P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:26.136277 1150 tablet_bootstrap.cc:492] T 74b709cb5ec64006b2f12bead8217d80 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:26.136633 1150 ts_tablet_manager.cc:1397] T 74b709cb5ec64006b2f12bead8217d80 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.056s user 0.005s sys 0.002s
I20250901 01:52:26.136672 1872 leader_election.cc:290] T e249474dcbf642189560c91ec3a99fdf P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:26.137045 992 leader_election.cc:336] T 15e4d972e8254cf899c3e0b71b6e2bd9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:26.137270 992 leader_election.cc:336] T abe27e749fa0485195e0064a49d5431e P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:26.137387 992 leader_election.cc:304] T abe27e749fa0485195e0064a49d5431e P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:26.137367 1150 raft_consensus.cc:357] T 74b709cb5ec64006b2f12bead8217d80 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:26.137581 1150 raft_consensus.cc:383] T 74b709cb5ec64006b2f12bead8217d80 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:26.137679 1150 raft_consensus.cc:738] T 74b709cb5ec64006b2f12bead8217d80 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:26.138176 1150 consensus_queue.cc:260] T 74b709cb5ec64006b2f12bead8217d80 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:26.138371 1872 raft_consensus.cc:2747] T abe27e749fa0485195e0064a49d5431e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:26.138834 1150 ts_tablet_manager.cc:1428] T 74b709cb5ec64006b2f12bead8217d80 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250901 01:52:26.139035 2118 raft_consensus.cc:2270] T 56475ba491a24301a534304811f30140 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:26.139757 2118 ts_tablet_manager.cc:1905] T 56475ba491a24301a534304811f30140 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:26.143787 1872 raft_consensus.cc:491] T 3618a3eb4f174c43805ce7dc0e5315cb P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:26.144136 1872 raft_consensus.cc:513] T 3618a3eb4f174c43805ce7dc0e5315cb P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:26.145715 1872 leader_election.cc:290] T 3618a3eb4f174c43805ce7dc0e5315cb P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:26.139205 1150 tablet_bootstrap.cc:492] T 6c0abbfc8d904dce989ce51961e4237d P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:26.150494 1150 tablet_bootstrap.cc:654] T 6c0abbfc8d904dce989ce51961e4237d P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
E20250901 01:52:26.162274 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae4844183a3c4a04a775e6fd878391a3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_aee74198b51d4f089538c4f2bf3ce4dc'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:26.162771 1696 catalog_manager.cc:6205] Error selecting replicas for tablet be8dfbc466564842a2fdb4953eee939b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_75ba3ba70ca24a94920e7bce96190ff2'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:26.163053 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 96824e68bbf24b8287dbb683038ffaba, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_3c7c628f35fb4d02bb79a34995f12938'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:26.163223 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 7317eb7848ee44698038a4b404d4bf21, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b0e916c76a7a4766aa0887cc8691488f'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:26.163746 2118 ts_tablet_manager.cc:1918] T 56475ba491a24301a534304811f30140 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
E20250901 01:52:26.163414 1696 catalog_manager.cc:6205] Error selecting replicas for tablet e2ed32b3292141278af28026528e96c2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_50ae3908a7944f70949c54be8fbf1564'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:26.164045 1696 catalog_manager.cc:6205] Error selecting replicas for tablet a2ca69fabadd4ad69e1670cae7a0c503, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_32c4716e3aa24a87a6b1472984b255c4'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:26.164222 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f8f923ddcd2740bcb7e521f4f57c4034, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_5e65c5f068b84995a2d7a76fa65dda0f'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:26.164376 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 0813cc68d4b0488ba60a523e4479971b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_52733c9a363b4ed9b6ff11ad8571e421'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:26.164530 1696 catalog_manager.cc:6205] Error selecting replicas for tablet cc3b8736b9094d7e892c80da4d5b20f6, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_93afd37289414655980b22934695bd55'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:26.164902 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae9a4ad4fdfb4771ba23a91f7007447e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_5e84ae9a9c2a4084a1f9c5ffb4594de2'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:26.165132 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9074543dea9949e7a085717b40913348, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_53e30205c5fd488eb3b48c685a159d04'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:26.165393 2253 raft_consensus.cc:491] T fa7fb88ded21432a8757f37ec3db5e71 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:26.165643 2253 raft_consensus.cc:513] T fa7fb88ded21432a8757f37ec3db5e71 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:26.165840 2113 ts_tablet_manager.cc:1918] T 17e8340dbd1a406cb782dad3fe88cc7e P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:26.165944 2113 log.cc:1199] T 17e8340dbd1a406cb782dad3fe88cc7e P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/17e8340dbd1a406cb782dad3fe88cc7e
I20250901 01:52:26.166167 2253 leader_election.cc:290] T fa7fb88ded21432a8757f37ec3db5e71 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:26.166224 2113 ts_tablet_manager.cc:1939] T 17e8340dbd1a406cb782dad3fe88cc7e P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
W20250901 01:52:26.168277 858 leader_election.cc:336] T fa7fb88ded21432a8757f37ec3db5e71 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:26.168573 2118 log.cc:1199] T 56475ba491a24301a534304811f30140 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/56475ba491a24301a534304811f30140
I20250901 01:52:26.168992 2118 ts_tablet_manager.cc:1939] T 56475ba491a24301a534304811f30140 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:26.169359 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 38) took 1001 ms. Trace:
I20250901 01:52:26.169502 2118 rpcz_store.cc:276] 0901 01:52:25.167514 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.167871 (+ 357us) service_pool.cc:225] Handling call
0901 01:52:26.124050 (+956179us) ts_tablet_manager.cc:1145] Deleting tablet 56475ba491a24301a534304811f30140
0901 01:52:26.129759 (+ 5709us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:26.158465 (+ 28706us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:26.163642 (+ 5177us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:26.169342 (+ 5700us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":955450}
I20250901 01:52:26.170030 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15e4d972e8254cf899c3e0b71b6e2bd9" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:26.170209 913 raft_consensus.cc:3058] T 15e4d972e8254cf899c3e0b71b6e2bd9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:26.170209 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 56475ba491a24301a534304811f30140 (table default.Table_b0e916c76a7a4766aa0887cc8691488f [id=cf61137a852c48748a29f833f1860da7]) successfully deleted
I20250901 01:52:26.170637 2118 tablet_replica.cc:331] T afc002ae2486476b9208e1cf8215157c P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:26.170863 2118 raft_consensus.cc:2241] T afc002ae2486476b9208e1cf8215157c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:26.171065 2118 raft_consensus.cc:2270] T afc002ae2486476b9208e1cf8215157c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:26.171731 2118 ts_tablet_manager.cc:1905] T afc002ae2486476b9208e1cf8215157c P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:26.172984 1150 tablet_bootstrap.cc:492] T 6c0abbfc8d904dce989ce51961e4237d P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:26.173105 1150 ts_tablet_manager.cc:1397] T 6c0abbfc8d904dce989ce51961e4237d P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.034s user 0.000s sys 0.007s
I20250901 01:52:26.173200 913 raft_consensus.cc:2466] T 15e4d972e8254cf899c3e0b71b6e2bd9 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:26.173810 1150 raft_consensus.cc:357] T 6c0abbfc8d904dce989ce51961e4237d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:26.173974 1150 raft_consensus.cc:383] T 6c0abbfc8d904dce989ce51961e4237d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:26.174024 1150 raft_consensus.cc:738] T 6c0abbfc8d904dce989ce51961e4237d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:26.174193 1150 consensus_queue.cc:260] T 6c0abbfc8d904dce989ce51961e4237d P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:26.174640 1150 ts_tablet_manager.cc:1428] T 6c0abbfc8d904dce989ce51961e4237d P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250901 01:52:26.174906 1150 tablet_bootstrap.cc:492] T 879fa66e55544118aafe504587393c66 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:26.177712 2118 ts_tablet_manager.cc:1918] T afc002ae2486476b9208e1cf8215157c P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:26.177850 2118 log.cc:1199] T afc002ae2486476b9208e1cf8215157c P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/afc002ae2486476b9208e1cf8215157c
I20250901 01:52:26.178179 2118 ts_tablet_manager.cc:1939] T afc002ae2486476b9208e1cf8215157c P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:26.178476 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 39) took 1009 ms. Trace:
I20250901 01:52:26.178623 2118 rpcz_store.cc:276] 0901 01:52:25.168841 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.169362 (+ 521us) service_pool.cc:225] Handling call
0901 01:52:26.170589 (+1001227us) ts_tablet_manager.cc:1145] Deleting tablet afc002ae2486476b9208e1cf8215157c
0901 01:52:26.170599 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:26.174907 (+ 4308us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:26.177633 (+ 2726us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:26.178463 (+ 830us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1001052}
I20250901 01:52:26.179095 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet afc002ae2486476b9208e1cf8215157c (table default.Table_ad2bde750a6f4c9aad14a28a9a0848c9 [id=da00d16114194d41826ef0da1f9423a9]) successfully deleted
I20250901 01:52:26.179400 2118 tablet_replica.cc:331] T 75e7a90815e449c39f386901a504143c P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:26.179636 2118 raft_consensus.cc:2241] T 75e7a90815e449c39f386901a504143c P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:26.179824 2118 raft_consensus.cc:2270] T 75e7a90815e449c39f386901a504143c P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:26.180335 2118 ts_tablet_manager.cc:1905] T 75e7a90815e449c39f386901a504143c P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:26.181689 1150 tablet_bootstrap.cc:654] T 879fa66e55544118aafe504587393c66 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:26.186761 1151 tablet_bootstrap.cc:492] T 8ca5cf4e99014c0d84a3acad4ef713af P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:26.186987 1151 ts_tablet_manager.cc:1397] T 8ca5cf4e99014c0d84a3acad4ef713af P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.106s user 0.003s sys 0.001s
I20250901 01:52:26.187817 1151 raft_consensus.cc:357] T 8ca5cf4e99014c0d84a3acad4ef713af P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:26.188064 1151 raft_consensus.cc:383] T 8ca5cf4e99014c0d84a3acad4ef713af P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:26.188163 1151 raft_consensus.cc:738] T 8ca5cf4e99014c0d84a3acad4ef713af P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:26.188455 1151 consensus_queue.cc:260] T 8ca5cf4e99014c0d84a3acad4ef713af P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:26.189079 1151 ts_tablet_manager.cc:1428] T 8ca5cf4e99014c0d84a3acad4ef713af P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250901 01:52:26.189474 1151 tablet_bootstrap.cc:492] T 6cf09e8f299548dba2cce16d7b565bad P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:26.194784 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 44) took 1295 ms. Trace:
I20250901 01:52:26.196076 2118 ts_tablet_manager.cc:1918] T 75e7a90815e449c39f386901a504143c P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:26.196275 2118 log.cc:1199] T 75e7a90815e449c39f386901a504143c P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/75e7a90815e449c39f386901a504143c
I20250901 01:52:26.196691 2118 ts_tablet_manager.cc:1939] T 75e7a90815e449c39f386901a504143c P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:26.196983 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 40) took 1027 ms. Trace:
I20250901 01:52:26.197085 2118 rpcz_store.cc:276] 0901 01:52:25.170764 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.170857 (+ 93us) service_pool.cc:225] Handling call
0901 01:52:26.179361 (+1008504us) ts_tablet_manager.cc:1145] Deleting tablet 75e7a90815e449c39f386901a504143c
0901 01:52:26.179370 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:26.192878 (+ 13508us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:26.196002 (+ 3124us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:26.196969 (+ 967us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1008310}
I20250901 01:52:26.195814 2113 rpcz_store.cc:276] 0901 01:52:24.899399 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:24.899452 (+ 53us) service_pool.cc:225] Handling call
0901 01:52:25.910024 (+1010572us) ts_tablet_manager.cc:1145] Deleting tablet 17e8340dbd1a406cb782dad3fe88cc7e
0901 01:52:25.910033 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:26.035401 (+125368us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:26.165737 (+130336us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:26.194491 (+ 28754us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1010402}
I20250901 01:52:26.197614 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 75e7a90815e449c39f386901a504143c (table default.Table_ca246ea8f871412d949125b3714d8b67 [id=85d86bb68a4b43c4a8d08dd9ce12a4fc]) successfully deleted
I20250901 01:52:26.197966 2118 tablet_replica.cc:331] T 05630a72ddae4de0b890dece2feacc31 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:26.198515 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 17e8340dbd1a406cb782dad3fe88cc7e (table default.Table_50ae3908a7944f70949c54be8fbf1564 [id=58ff38d356d74951b663d8ef7530f1c8]) successfully deleted
I20250901 01:52:26.198791 2253 raft_consensus.cc:491] T 3239e58531d445fa80165cf678bc277c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:26.199852 2253 raft_consensus.cc:513] T 3239e58531d445fa80165cf678bc277c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:26.199990 2118 raft_consensus.cc:2241] T 05630a72ddae4de0b890dece2feacc31 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:26.200184 2118 raft_consensus.cc:2270] T 05630a72ddae4de0b890dece2feacc31 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:26.200101 2113 tablet_replica.cc:331] T e25923b2daf9429683b2d44a75a56189 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
E20250901 01:52:26.207515 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 18498c9cb09141d08c1c5c0854b63ebd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77f4810bc83c4d948531da2a45a3792e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:26.207749 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 81319e373c9a4475abceeeedb507aabc, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_22d8d38d0b624912a0d8968fb764420e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:26.207908 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 6ce9521b925b4f97bb4a810ce77e147e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_9e04f976a93f4d30baccf90fa3622dfe'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:26.212646 1484 master_service.cc:948] ReplaceTablet: received request to replace tablet eee34044c39d4013baf78d3705489626 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:26.212767 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_cf54de11da9f4af28886f8e577b86fe4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.214138 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_519b32f37d824c7dbfcc59ddd0d5b272"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.215301 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_26c8bfb6a64940f78444097dab9c994e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.215517 1489 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 606fac5c6efe48d1877104b01c23ba5b
W20250901 01:52:26.215718 1489 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 606fac5c6efe48d1877104b01c23ba5b: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:26.216060 1489 catalog_manager.cc:6652] ReplaceTablet: tablet 606fac5c6efe48d1877104b01c23ba5b deleted and replaced by tablet 787be682c0e44248a80d7037fb61dc6c
I20250901 01:52:26.216629 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 606fac5c6efe48d1877104b01c23ba5b with delete_type TABLET_DATA_DELETED (replaced by tablet 787be682c0e44248a80d7037fb61dc6c) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:26.216645 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_167ffe99b38e42c2bc8fda01107f4891"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.217895 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f622e9010d1c448ea6dd23723243c1f3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.218679 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_0258c4faac414bdcbf264a60a3004dec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.221037 2253 leader_election.cc:290] T 3239e58531d445fa80165cf678bc277c P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:26.205382 858 leader_election.cc:336] T 3239e58531d445fa80165cf678bc277c P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:26.223534 2253 raft_consensus.cc:491] T 7a2f6e2c7ee2402ea310c058e046416c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:26.223620 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8cdcc7a0ae1a44f5b0e44668ae226717"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.223652 2253 raft_consensus.cc:513] T 7a2f6e2c7ee2402ea310c058e046416c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:26.224076 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ff72ba0da87a4eaa9005e12bcc22c651"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.224231 2253 leader_election.cc:290] T 7a2f6e2c7ee2402ea310c058e046416c P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:26.224520 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1fceff0616844f11b3250703578c0455"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.225033 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8d8bd818b50d4876b8ba08f059566376"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.225946 2113 raft_consensus.cc:2241] T e25923b2daf9429683b2d44a75a56189 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:26.226280 2113 raft_consensus.cc:2270] T e25923b2daf9429683b2d44a75a56189 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:26.228873 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "81c36693199945d98465579b509852e2" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:26.229044 913 raft_consensus.cc:2466] T 81c36693199945d98465579b509852e2 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:26.231348 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 606fac5c6efe48d1877104b01c23ba5b with delete_type TABLET_DATA_DELETED (replaced by tablet 787be682c0e44248a80d7037fb61dc6c) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:26.231818 2113 ts_tablet_manager.cc:1905] T e25923b2daf9429683b2d44a75a56189 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:26.234474 858 leader_election.cc:336] T 7a2f6e2c7ee2402ea310c058e046416c P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:26.237082 2118 ts_tablet_manager.cc:1905] T 05630a72ddae4de0b890dece2feacc31 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:26.234829 1055 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0c3c363dda444519a17dca63ad9caa7" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58"
I20250901 01:52:26.238104 1055 raft_consensus.cc:2391] T d0c3c363dda444519a17dca63ad9caa7 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 00c282f779c1477d920f24dc0dc16628 in current term 1: Already voted for candidate fd27ab6665304aaea53adbd102dd4f58 in this term.
I20250901 01:52:26.235002 1054 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fa7fb88ded21432a8757f37ec3db5e71" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:26.235071 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8ed6616ce54548ba83ba299c4ecf3c38" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:26.235340 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3239e58531d445fa80165cf678bc277c" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:26.235585 2301 tablet.cc:1722] T b009cc95c819431ea9827a3e453b8da3 P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250901 01:52:26.235965 1058 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a2f6e2c7ee2402ea310c058e046416c" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:26.238667 1058 raft_consensus.cc:2466] T 7a2f6e2c7ee2402ea310c058e046416c P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00c282f779c1477d920f24dc0dc16628 in term 0.
I20250901 01:52:26.233858 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46bfbdff3f904715a8c24907b20fcab0" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:26.242834 913 raft_consensus.cc:2466] T 46bfbdff3f904715a8c24907b20fcab0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:26.246783 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8bda89f2cb0341dea4b540ed59803bcd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.249033 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_cbc686b22be14fb8851345bc5b31b87d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.250008 2306 raft_consensus.cc:491] T a289a66b2dfa44dc8c8547ff83c1716e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:26.250082 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b915b54dcf794919bb91b6e319de2bbb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.250128 2306 raft_consensus.cc:513] T a289a66b2dfa44dc8c8547ff83c1716e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:26.250715 2306 leader_election.cc:290] T a289a66b2dfa44dc8c8547ff83c1716e P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:26.250962 1491 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_8dd9c9817a5c4e3dbb0bbf1d5cf1b5db" } new_table_name: "default.Table_e53182d58e5244a9b7ba33cb48c939ba" modify_external_catalogs: true
I20250901 01:52:26.251607 1484 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_2ece91972d494680a8ad00273c10dee4" } new_table_name: "default.Table_3d5afc22c4e14d6990aea31564dfde20" modify_external_catalogs: true
I20250901 01:52:26.251971 1492 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_11f4f2d9a7404f919fad5a1c628cf5cb" } modify_external_catalogs: true
I20250901 01:52:26.252070 1489 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_d61fc9783e7b45e49afb36280adcbd86" } new_table_name: "default.Table_4e2ea69d21394b9db0447f7ec8766795" modify_external_catalogs: true
I20250901 01:52:26.252379 1483 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 87e97a9544a2485897b1053c8929a39a
I20250901 01:52:26.252457 1492 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_11f4f2d9a7404f919fad5a1c628cf5cb" } modify_external_catalogs: true
W20250901 01:52:26.252609 1483 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 87e97a9544a2485897b1053c8929a39a: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:26.253217 1483 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_ecb9e8f638a043cc87c38afbebce250f" } modify_external_catalogs: true
I20250901 01:52:26.253382 1483 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_ecb9e8f638a043cc87c38afbebce250f" } modify_external_catalogs: true
I20250901 01:52:26.254197 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 87e97a9544a2485897b1053c8929a39a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:26 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:26.255134 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 87e97a9544a2485897b1053c8929a39a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:26 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:26.268231 1492 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 0f37a24e693847f98b574950379081f3
W20250901 01:52:26.268468 1492 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 0f37a24e693847f98b574950379081f3: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:26.270076 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 0f37a24e693847f98b574950379081f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:26 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:26.270716 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 0f37a24e693847f98b574950379081f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:26 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:26.271924 1150 tablet_bootstrap.cc:492] T 879fa66e55544118aafe504587393c66 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:26.272236 1150 ts_tablet_manager.cc:1397] T 879fa66e55544118aafe504587393c66 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.097s user 0.003s sys 0.004s
I20250901 01:52:26.273134 2253 raft_consensus.cc:491] T 6c0abbfc8d904dce989ce51961e4237d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:26.273288 2253 raft_consensus.cc:513] T 6c0abbfc8d904dce989ce51961e4237d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:26.273938 2253 leader_election.cc:290] T 6c0abbfc8d904dce989ce51961e4237d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:26.274190 1150 raft_consensus.cc:357] T 879fa66e55544118aafe504587393c66 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:26.274590 1150 raft_consensus.cc:383] T 879fa66e55544118aafe504587393c66 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:26.274787 1150 raft_consensus.cc:738] T 879fa66e55544118aafe504587393c66 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:26.275254 1150 consensus_queue.cc:260] T 879fa66e55544118aafe504587393c66 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:26.275810 858 leader_election.cc:336] T 6c0abbfc8d904dce989ce51961e4237d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:26.278832 1150 ts_tablet_manager.cc:1428] T 879fa66e55544118aafe504587393c66 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
I20250901 01:52:26.287165 1150 tablet_bootstrap.cc:492] T b13afacd37474781877b9f2cd20b313a P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:26.289669 2253 raft_consensus.cc:491] T d1f2d98712ff4ea6aec665512de068bd P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:26.289806 2253 raft_consensus.cc:513] T d1f2d98712ff4ea6aec665512de068bd P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:26.290388 2253 leader_election.cc:290] T d1f2d98712ff4ea6aec665512de068bd P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:26.291079 1151 tablet_bootstrap.cc:654] T 6cf09e8f299548dba2cce16d7b565bad P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:26.292534 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad45c90ba0da43c9860745e534664a98" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:26.292762 913 raft_consensus.cc:2466] T ad45c90ba0da43c9860745e534664a98 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:26.293612 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f80f13db501c42c7950499df8fcb6fc8" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:26.293793 913 raft_consensus.cc:3058] T f80f13db501c42c7950499df8fcb6fc8 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
W20250901 01:52:26.295154 992 leader_election.cc:336] T f50f33de1b654663b9ef8a91e3436f11 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:26.295465 992 leader_election.cc:336] T f80f13db501c42c7950499df8fcb6fc8 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:26.295828 992 leader_election.cc:336] T d0c3c363dda444519a17dca63ad9caa7 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:26.295840 1492 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_942175dadf4b496195b2fa53f39f6ddc" } modify_external_catalogs: true
I20250901 01:52:26.296043 1492 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_942175dadf4b496195b2fa53f39f6ddc" } modify_external_catalogs: true
W20250901 01:52:26.296475 858 leader_election.cc:336] T d1f2d98712ff4ea6aec665512de068bd P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:26.298260 917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e249474dcbf642189560c91ec3a99fdf" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:26.298533 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0c3c363dda444519a17dca63ad9caa7" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:26.298815 917 raft_consensus.cc:2466] T e249474dcbf642189560c91ec3a99fdf P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:26.299194 914 raft_consensus.cc:2391] T d0c3c363dda444519a17dca63ad9caa7 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate fd27ab6665304aaea53adbd102dd4f58 in current term 1: Already voted for candidate 00c282f779c1477d920f24dc0dc16628 in this term.
I20250901 01:52:26.300467 913 raft_consensus.cc:2466] T f80f13db501c42c7950499df8fcb6fc8 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:26.302199 970 heartbeater.cc:344] Connected to a master server at 127.0.104.62:43871
I20250901 01:52:26.304594 1150 tablet_bootstrap.cc:654] T b13afacd37474781877b9f2cd20b313a P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:26.306731 2118 ts_tablet_manager.cc:1918] T 05630a72ddae4de0b890dece2feacc31 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:26.306972 2118 log.cc:1199] T 05630a72ddae4de0b890dece2feacc31 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/05630a72ddae4de0b890dece2feacc31
I20250901 01:52:26.307401 2118 ts_tablet_manager.cc:1939] T 05630a72ddae4de0b890dece2feacc31 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:26.307796 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 41) took 1136 ms. Trace:
I20250901 01:52:26.307978 2118 rpcz_store.cc:276] 0901 01:52:25.171256 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.172644 (+ 1388us) service_pool.cc:225] Handling call
0901 01:52:26.197922 (+1025278us) ts_tablet_manager.cc:1145] Deleting tablet 05630a72ddae4de0b890dece2feacc31
0901 01:52:26.197931 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:26.267334 (+ 69403us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:26.306567 (+ 39233us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:26.307765 (+ 1198us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1025127}
I20250901 01:52:26.308549 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 05630a72ddae4de0b890dece2feacc31 (table default.Table_ee242c34410e4c6887405d04a4b26124 [id=c9a2f96383664a4e8524c286cd9e4149]) successfully deleted
I20250901 01:52:26.308970 2118 tablet_replica.cc:331] T 7a607fac57994001b2304e5d81c9decc P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:26.309245 2118 raft_consensus.cc:2241] T 7a607fac57994001b2304e5d81c9decc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:26.329367 2113 ts_tablet_manager.cc:1918] T e25923b2daf9429683b2d44a75a56189 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:26.333513 2113 log.cc:1199] T e25923b2daf9429683b2d44a75a56189 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/e25923b2daf9429683b2d44a75a56189
I20250901 01:52:26.333981 2113 ts_tablet_manager.cc:1939] T e25923b2daf9429683b2d44a75a56189 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:26.334913 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 45) took 1434 ms. Trace:
I20250901 01:52:26.335042 2113 rpcz_store.cc:276] 0901 01:52:24.900159 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:24.900212 (+ 53us) service_pool.cc:225] Handling call
0901 01:52:26.199556 (+1299344us) ts_tablet_manager.cc:1145] Deleting tablet e25923b2daf9429683b2d44a75a56189
0901 01:52:26.199568 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:26.294732 (+ 95164us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:26.329253 (+ 34521us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:26.334894 (+ 5641us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":25043,"tablet-delete.queue_time_us":1298915}
I20250901 01:52:26.335841 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet e25923b2daf9429683b2d44a75a56189 (table default.Table_202b4c5ccc56474a8b2e7b9d7c4a28d6 [id=746845bd84c6444b8a1e5683eaba4a68]) successfully deleted
I20250901 01:52:26.338073 2113 tablet_replica.cc:331] T 6741c56499b44f648fb032c8cf50205b P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:26.338323 2113 raft_consensus.cc:2241] T 6741c56499b44f648fb032c8cf50205b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:26.338521 2113 raft_consensus.cc:2270] T 6741c56499b44f648fb032c8cf50205b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:26.339046 2113 ts_tablet_manager.cc:1905] T 6741c56499b44f648fb032c8cf50205b P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:26.345201 2113 ts_tablet_manager.cc:1918] T 6741c56499b44f648fb032c8cf50205b P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:26.345294 2113 log.cc:1199] T 6741c56499b44f648fb032c8cf50205b P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/6741c56499b44f648fb032c8cf50205b
I20250901 01:52:26.345667 2113 ts_tablet_manager.cc:1939] T 6741c56499b44f648fb032c8cf50205b P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:26.346555 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 46) took 1445 ms. Trace:
I20250901 01:52:26.346650 2113 rpcz_store.cc:276] 0901 01:52:24.901113 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:24.901169 (+ 56us) service_pool.cc:225] Handling call
0901 01:52:26.338021 (+1436852us) ts_tablet_manager.cc:1145] Deleting tablet 6741c56499b44f648fb032c8cf50205b
0901 01:52:26.338032 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:26.342190 (+ 4158us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:26.345125 (+ 2935us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:26.346535 (+ 1410us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1436668}
I20250901 01:52:26.347119 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 6741c56499b44f648fb032c8cf50205b (table default.Table_eb9eaa97a468420bb672b720834c6ddd [id=c5667c5a65d045cea3cb52deb5801015]) successfully deleted
W20250901 01:52:26.348760 992 leader_election.cc:343] T f50f33de1b654663b9ef8a91e3436f11 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: f50f33de1b654663b9ef8a91e3436f11
I20250901 01:52:26.348872 992 leader_election.cc:304] T f50f33de1b654663b9ef8a91e3436f11 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:26.349478 1485 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 81ce448b148447ddbc41875d863e0518 (table default.Table_6e461d9cb0ae4dd095975bbc21533794 [id=49673fa73f8c436e9b8026efc4f3cbbc]). Expected version 1 got 0
I20250901 01:52:26.349723 1485 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet cb7d7e1796c0456f8de98fb812b9e374 (table default.Table_c4148158ed7b48a3946559ba978e72fd [id=327c5a63904540508142ab600c308757]). Expected version 1 got 0
I20250901 01:52:26.350183 1485 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet b164b4a67a9143ceaff69b0ce49d63fe (table default.Table_da2c70a8f2fe49a0a8184e88c0a9e0ea [id=c1b1e9e3876244f68d31c63ce7bdebf1]). Expected version 1 got 0
I20250901 01:52:26.350390 1151 tablet_bootstrap.cc:492] T 6cf09e8f299548dba2cce16d7b565bad P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:26.350657 1151 ts_tablet_manager.cc:1397] T 6cf09e8f299548dba2cce16d7b565bad P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.161s user 0.003s sys 0.002s
I20250901 01:52:26.350692 1485 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet d1f2d98712ff4ea6aec665512de068bd (table default.Table_82d91a1ba2eb4265896824028dec8d3a [id=3f7624d2901445e39155c12fa68e34e9]). Expected version 1 got 0
I20250901 01:52:26.351119 1485 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 1495de70f0844f82a0eea99c059f076b (table default.Table_9f48256fcac7429ba7d7fdb86979eda6 [id=70a5ce71d16b47d4bc6dae97be4548e7]). Expected version 2 got 0
I20250901 01:52:26.351436 1485 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet d10c193452954c198bba686169b8025b (table default.Table_03fb48eaab9b42858ab7cefb722e43a8 [id=6c0576410d7340d6aa493a747e95565b]). Expected version 1 got 0
I20250901 01:52:26.351620 1485 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 435b1cca5a7d4d49959a310cb250b2a0 (table default.Table_b989608c88204be2b6f706a9426c47bb [id=126b298fef6c4653952d672adff1db7e]). Expected version 2 got 0
I20250901 01:52:26.351804 1485 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet ed1b41340ce742a88c3e4fd85cd25ca6 (table default.Table_139f6d174d794a6e9a618e90587d0c0d [id=0baa4993b72e4a8d94ee3059c9d2db6c]). Expected version 1 got 0
I20250901 01:52:26.352061 1485 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet c6f7c525f59b49938b697de98d4b5835 (table default.Table_db61e055dbd7450898e82c4c0c4763be [id=aabca67f6f754820977ac9c4c20f3e4f]). Expected version 1 got 0
I20250901 01:52:26.352136 1872 raft_consensus.cc:2747] T f50f33de1b654663b9ef8a91e3436f11 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:26.352217 1485 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 7954393c8c1c4457ab8bb723c03a9c8b (table default.Table_dd01ae355f73448cbca96f086b04a2fe [id=543f2074598d4c29972e564be1123fca]). Expected version 3 got 0
I20250901 01:52:26.352401 1485 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 6c0abbfc8d904dce989ce51961e4237d (table default.Table_3f9422b08b0349b9a5466218f0fe1b06 [id=f0dfb3811e744bd4ba6d59195cebde6f]). Expected version 2 got 0
I20250901 01:52:26.352731 1485 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet d0c3c363dda444519a17dca63ad9caa7 (table default.Table_0b4ac0a1caa5472d909448e41318e679 [id=8c1b263824ec4c15a5ec424364c78895]). Expected version 2 got 0
W20250901 01:52:26.352766 992 leader_election.cc:343] T c95798f0daf4437db18a291c4464522e P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: c95798f0daf4437db18a291c4464522e
I20250901 01:52:26.310824 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3618a3eb4f174c43805ce7dc0e5315cb" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:26.322381 1150 tablet_bootstrap.cc:492] T b13afacd37474781877b9f2cd20b313a P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:26.353032 1485 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet e6c10426f60f43e8bde69d9b1b9c8eb6 (table default.Table_eecfab7239cc40c5a2f1fb6ac6d35dc9 [id=d3dbeb4258424133844e20b181ee44e4]). Expected version 1 got 0
I20250901 01:52:26.353080 913 raft_consensus.cc:2466] T 3618a3eb4f174c43805ce7dc0e5315cb P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:26.353143 1150 ts_tablet_manager.cc:1397] T b13afacd37474781877b9f2cd20b313a P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.066s user 0.006s sys 0.001s
I20250901 01:52:26.353179 1485 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 615b2306fca84003b264da9d5f95ddab (table default.Table_302f6e7cf97d46b9a621eb852e6aefe0 [id=cecdd8d6ca6d445c9fd82decd2a95366]). Expected version 2 got 0
I20250901 01:52:26.353443 1485 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet fecf7d538b4c4c619182d5eccb44dd8b (table default.Table_1b5729d9768546fe99eaafc4e2604f42 [id=02d0056abe2146ac9af81ebf4447b99d]). Expected version 1 got 0
I20250901 01:52:26.353636 2118 raft_consensus.cc:2270] T 7a607fac57994001b2304e5d81c9decc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:26.354022 1485 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 89d40f0331114490b51de3ba6d773571 (table default.Table_4dacb3b1d2e94e42ac8937eb6eb58375 [id=8ee0aa1ba24042dd8ded004644913b9c]). Expected version 1 got 0
W20250901 01:52:26.354032 992 leader_election.cc:343] T b73e362321284df1b18ef755dfe930a5 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: b73e362321284df1b18ef755dfe930a5
I20250901 01:52:26.353960 1150 raft_consensus.cc:357] T b13afacd37474781877b9f2cd20b313a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
W20250901 01:52:26.354154 992 leader_election.cc:343] T 66a48a8ebe154b2cac89c234eaa378ff P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 66a48a8ebe154b2cac89c234eaa378ff
I20250901 01:52:26.354189 1150 raft_consensus.cc:383] T b13afacd37474781877b9f2cd20b313a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:26.354267 1150 raft_consensus.cc:738] T b13afacd37474781877b9f2cd20b313a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:26.354306 2113 tablet_replica.cc:331] T 46bfbdff3f904715a8c24907b20fcab0 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:26.354537 2113 raft_consensus.cc:2241] T 46bfbdff3f904715a8c24907b20fcab0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:26.354516 1150 consensus_queue.cc:260] T b13afacd37474781877b9f2cd20b313a P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:26.354810 1485 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet ae21b9ae2e3b472eb67fb5f96388a50f (table default.Table_046289f393ff432aae0b033bdec1fb17 [id=db57b18a9ce24e068b362f5fbe940531]). Expected version 1 got 0
I20250901 01:52:26.354982 1485 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 858218c4427c41e980b2b5c969a5af40 (table default.Table_6d47266af0ed433eb1f2abbc81d18e69 [id=264cc469a73841e79538843479a165ab]). Expected version 1 got 0
I20250901 01:52:26.323886 2253 raft_consensus.cc:491] T 89d40f0331114490b51de3ba6d773571 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:26.355139 1485 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 3618a3eb4f174c43805ce7dc0e5315cb (table default.Table_83f2c0acedbc4c6982caf5313646e90b [id=31a2e50c32644321825da40176127015]). Expected version 1 got 0
I20250901 01:52:26.355584 1485 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet f80f13db501c42c7950499df8fcb6fc8 (table default.Table_ce86974fd8974de9a5a737954fac2ad7 [id=ff772d20b287476f94716379b86d2a71]). Expected version 1 got 0
I20250901 01:52:26.356285 1485 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 508406601c994e42a5c5b1f665e74957 (table default.Table_345b7f9fe8214aa6878f20a166125364 [id=4aabc328f6374ec5b1dea26f778208b2]). Expected version 1 got 0
I20250901 01:52:26.334728 2323 raft_consensus.cc:491] T 879fa66e55544118aafe504587393c66 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:26.356535 1485 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 36e10328c34242ebb4dbd8f294162696 (table default.Table_8445fcd774874cadaf4acc3581eee417 [id=958e51a6bbee47638e504e1a44094737]). Expected version 1 got 0
I20250901 01:52:26.336870 2123 raft_consensus.cc:491] T 06d5eeb70e664f73bf643922d445bc7a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:26.357105 1485 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet b009cc95c819431ea9827a3e453b8da3 (table default.Table_8dd9c9817a5c4e3dbb0bbf1d5cf1b5db [id=cc26c89db2484a6fad919231116285f2]). Expected version 2 got 1
I20250901 01:52:26.357522 1485 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 93bcac1fd95d44dc927abfca81692921 (table default.Table_b8f8ee74648c406eaf589d3ab80934db [id=b8f69313439543be9c11eea88dfa69c5]). Expected version 1 got 0
I20250901 01:52:26.357736 1485 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet e9f1f892658b4736859ff507536920a1 (table default.Table_8825dd28afbe464193eb1b668cc9326a [id=9aa62cb634314a0a849612c54c1220d9]). Expected version 1 got 0
I20250901 01:52:26.358067 1485 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 7f46e9ddadaf4ff986d8c991178daa0c (table default.Table_9978497447394e5694ae928412634ecb [id=ea928501fe954cd0b81d38add4ef7128]). Expected version 2 got 0
I20250901 01:52:26.358422 1485 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet aa424ac6ef1441e6b1cc2867b1c48186 (table default.Table_bbbde49530e34a76b7cb17988ac6baca [id=76928d8caa2f4134a014656cf2cc871a]). Expected version 1 got 0
I20250901 01:52:26.358642 1485 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 5f5396045979450193d461ba1334a5c9 (table default.Table_a89cc8e48563460baabc34c821ceae35 [id=9c3930e9fcfe4b49ba4bea67a2b0ad07]). Expected version 1 got 0
I20250901 01:52:26.358608 1151 raft_consensus.cc:357] T 6cf09e8f299548dba2cce16d7b565bad P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:26.358942 1485 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 3dae00aa38204ee3ac77325eb334be42 (table default.Table_faf7ef12324f454396e5ae64d64ebe51 [id=5812902df9ea406ba59ff0ff23f130b6]). Expected version 1 got 0
I20250901 01:52:26.358966 1151 raft_consensus.cc:383] T 6cf09e8f299548dba2cce16d7b565bad P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:26.359128 1151 raft_consensus.cc:738] T 6cf09e8f299548dba2cce16d7b565bad P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:26.359428 1485 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 33f637f6a8134215bb1995ec38440073 (table default.Table_2ece91972d494680a8ad00273c10dee4 [id=557afe29976d40f6bfb6ae4ca348f887]). Expected version 1 got 0
I20250901 01:52:26.359504 1151 consensus_queue.cc:260] T 6cf09e8f299548dba2cce16d7b565bad P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:26.359731 1485 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 5adf6b492640417ca071c94702b48cb9 (table default.Table_589c302e02474fc3bc5d3a290a8bd54a [id=5d14706774d345c48ca3575feabb17b0]). Expected version 2 got 0
I20250901 01:52:26.360020 1485 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 61ddcf86c9f34726939cc116434f0822 (table default.Table_63d0f2ead9d946d39ce40162ebd769d6 [id=f9dcf4937b1143248bfa9d88053fce17]). Expected version 3 got 0
I20250901 01:52:26.360244 1485 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 3a8231590ea643b0a1f83eec4d9a8191 (table default.Table_a4ebd8ab87c842f082da5cd04d1bddac [id=fd6d8e92d6e64a89a7d38aa761376955]). Expected version 1 got 0
I20250901 01:52:26.360558 1485 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 8ca5cf4e99014c0d84a3acad4ef713af (table default.Table_ed6fd284be6f4ae3a008e21e7ce17930 [id=303e76c918bc4c39aa47a4d24849a358]). Expected version 1 got 0
I20250901 01:52:26.361008 1485 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet ad45c90ba0da43c9860745e534664a98 (table default.Table_1cd3f3fb6b244b3f8796bfca9f7622c8 [id=6fd739b0e725424ca844dbb652ea8028]). Expected version 2 got 0
I20250901 01:52:26.361865 1485 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 49aaaec611784e6bac131efd58dbca51 (table default.Table_95e1338a49a04f22b679e363e253262e [id=e24b91b4ac904ef0a8c4725f70bced2f]). Expected version 1 got 0
I20250901 01:52:26.362206 1485 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet e1d18e999b5e42b1873631e0f1445081 (table default.Table_fd17a2ad225546aa821489ce94a788a2 [id=033c76e0598342c495ce2be42a0df5dd]). Expected version 1 got 0
I20250901 01:52:26.362380 1485 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 616ff34e8d314e0aa0e11ce60be11789 (table default.Table_0451d0afa26f4d5f8d4214a946eea04f [id=5ee13d46e06a4fa59b1070547573e523]). Expected version 1 got 0
I20250901 01:52:26.362643 1485 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 42c1156e8dd84c6fa1396b46de13eb6f (table default.Table_b5d681f51fc84395bbabe1e5b4cfb358 [id=40d05d90b0354334840ad17764f5b6bb]). Expected version 1 got 0
I20250901 01:52:26.363029 1485 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 081e2630e284498aa819f85bf8326315 (table default.Table_9e7ca61fcc324a8984bd1e470162c61f [id=8199118617cb4707bc3ae9b1363750f7]). Expected version 2 got 0
I20250901 01:52:26.363188 1485 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 8ed6616ce54548ba83ba299c4ecf3c38 (table default.Table_1bdd9041d74045cab6f23de11bf4dd09 [id=946ec3e53dcd4de4b8501b3e87e99072]). Expected version 2 got 0
I20250901 01:52:26.363561 1485 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet e87d1281f35648b0b115b67c95e244e7 (table default.Table_f840789a58984799bd8057128f6ea439 [id=4453959e273c473fadd737f56f7db3dd]). Expected version 1 got 0
I20250901 01:52:26.364331 2113 raft_consensus.cc:2270] T 46bfbdff3f904715a8c24907b20fcab0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:26.364894 2113 ts_tablet_manager.cc:1905] T 46bfbdff3f904715a8c24907b20fcab0 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:26.370503 992 leader_election.cc:304] T 15e4d972e8254cf899c3e0b71b6e2bd9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:26.354697 2118 ts_tablet_manager.cc:1905] T 7a607fac57994001b2304e5d81c9decc P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:26.355243 2253 raft_consensus.cc:513] T 89d40f0331114490b51de3ba6d773571 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:26.371616 2253 leader_election.cc:290] T 89d40f0331114490b51de3ba6d773571 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:26.356567 2323 raft_consensus.cc:513] T 879fa66e55544118aafe504587393c66 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:26.372335 2323 leader_election.cc:290] T 879fa66e55544118aafe504587393c66 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:26.371249 2113 ts_tablet_manager.cc:1918] T 46bfbdff3f904715a8c24907b20fcab0 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:26.356941 2123 raft_consensus.cc:513] T 06d5eeb70e664f73bf643922d445bc7a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:26.373284 2113 log.cc:1199] T 46bfbdff3f904715a8c24907b20fcab0 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/46bfbdff3f904715a8c24907b20fcab0
I20250901 01:52:26.373565 1761 raft_consensus.cc:2802] T 15e4d972e8254cf899c3e0b71b6e2bd9 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:26.373698 1761 raft_consensus.cc:695] T 15e4d972e8254cf899c3e0b71b6e2bd9 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:26.373755 2123 leader_election.cc:290] T 06d5eeb70e664f73bf643922d445bc7a P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:26.373732 992 leader_election.cc:343] T 82d163681fda47f38d79863bd28f576d P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 82d163681fda47f38d79863bd28f576d
I20250901 01:52:26.374012 1761 consensus_queue.cc:237] T 15e4d972e8254cf899c3e0b71b6e2bd9 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:26.374922 2113 ts_tablet_manager.cc:1939] T 46bfbdff3f904715a8c24907b20fcab0 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:26.374866 2301 tablet.cc:1722] T f62d96c37cca4143a4b098b7c69979e6 P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250901 01:52:26.375365 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 47) took 1473 ms. Trace:
I20250901 01:52:26.375515 2113 rpcz_store.cc:276] 0901 01:52:24.902309 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:24.902348 (+ 39us) service_pool.cc:225] Handling call
0901 01:52:26.354252 (+1451904us) ts_tablet_manager.cc:1145] Deleting tablet 46bfbdff3f904715a8c24907b20fcab0
0901 01:52:26.354265 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:26.367967 (+ 13702us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:26.371150 (+ 3183us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:26.375317 (+ 4167us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9625,"tablet-delete.queue_time_us":1451766}
I20250901 01:52:26.375963 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 48) took 1473 ms. Trace:
I20250901 01:52:26.376075 2113 rpcz_store.cc:276] 0901 01:52:24.902945 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:24.903016 (+ 71us) service_pool.cc:225] Handling call
0901 01:52:26.375810 (+1472794us) ts_tablet_manager.cc:1145] Deleting tablet 0dbdcb76eeba4fefa56e7835c23c0316
0901 01:52:26.375820 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:26.375926 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1472624}
I20250901 01:52:26.376363 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 49) took 1472 ms. Trace:
I20250901 01:52:26.376513 2113 rpcz_store.cc:276] 0901 01:52:24.904067 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:24.904119 (+ 52us) service_pool.cc:225] Handling call
0901 01:52:26.376268 (+1472149us) ts_tablet_manager.cc:1145] Deleting tablet ded33e0950b544509af11739353c620d
0901 01:52:26.376276 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:26.376323 (+ 47us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1471991}
I20250901 01:52:26.376784 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 51) took 1467 ms. Trace:
I20250901 01:52:26.376880 2113 rpcz_store.cc:276] 0901 01:52:24.909564 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:24.909612 (+ 48us) service_pool.cc:225] Handling call
0901 01:52:26.376692 (+1467080us) ts_tablet_manager.cc:1145] Deleting tablet eac50cbae691416e903cb2cbc907f96d
0901 01:52:26.376700 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:26.376747 (+ 47us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1466895}
I20250901 01:52:26.377117 2113 tablet_replica.cc:331] T 7f64ebd080944190be2e64d02d70a8a7 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:26.377328 2113 raft_consensus.cc:2241] T 7f64ebd080944190be2e64d02d70a8a7 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:26.377627 2113 raft_consensus.cc:2270] T 7f64ebd080944190be2e64d02d70a8a7 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:26.378485 2113 ts_tablet_manager.cc:1905] T 7f64ebd080944190be2e64d02d70a8a7 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:26.379335 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 46bfbdff3f904715a8c24907b20fcab0 (table default.Table_a210833682434f928e7da36d2fb4e09c [id=e71440216cc74a5a87964a8e456f784c]) successfully deleted
W20250901 01:52:26.379652 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 0dbdcb76eeba4fefa56e7835c23c0316 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0dbdcb76eeba4fefa56e7835c23c0316 already in progress: creating tablet
W20250901 01:52:26.379882 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet ded33e0950b544509af11739353c620d because the tablet was not found. No further retry: Not found: Tablet not found: ded33e0950b544509af11739353c620d
W20250901 01:52:26.380113 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet eac50cbae691416e903cb2cbc907f96d because the tablet was not found. No further retry: Not found: Tablet not found: eac50cbae691416e903cb2cbc907f96d
I20250901 01:52:26.384631 992 leader_election.cc:304] T 81c36693199945d98465579b509852e2 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:26.385366 2306 raft_consensus.cc:2802] T 81c36693199945d98465579b509852e2 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:26.385511 992 leader_election.cc:304] T 46bfbdff3f904715a8c24907b20fcab0 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
W20250901 01:52:26.385751 992 raft_consensus.cc:2684] T 46bfbdff3f904715a8c24907b20fcab0 P fd27ab6665304aaea53adbd102dd4f58: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250901 01:52:26.385828 2113 ts_tablet_manager.cc:1918] T 7f64ebd080944190be2e64d02d70a8a7 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:26.385967 2113 log.cc:1199] T 7f64ebd080944190be2e64d02d70a8a7 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/7f64ebd080944190be2e64d02d70a8a7
I20250901 01:52:26.385941 992 leader_election.cc:304] T ad45c90ba0da43c9860745e534664a98 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:26.386245 992 leader_election.cc:304] T e249474dcbf642189560c91ec3a99fdf P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:26.386359 1993 raft_consensus.cc:2802] T ad45c90ba0da43c9860745e534664a98 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:26.386464 1993 raft_consensus.cc:491] T ad45c90ba0da43c9860745e534664a98 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:26.386581 1993 raft_consensus.cc:3058] T ad45c90ba0da43c9860745e534664a98 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:26.388533 2305 raft_consensus.cc:2802] T e249474dcbf642189560c91ec3a99fdf P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:26.388622 2305 raft_consensus.cc:491] T e249474dcbf642189560c91ec3a99fdf P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:26.388731 2305 raft_consensus.cc:3058] T e249474dcbf642189560c91ec3a99fdf P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:26.385514 2306 raft_consensus.cc:491] T 81c36693199945d98465579b509852e2 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:26.390801 2306 raft_consensus.cc:3058] T 81c36693199945d98465579b509852e2 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:26.386387 2113 ts_tablet_manager.cc:1939] T 7f64ebd080944190be2e64d02d70a8a7 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:26.394932 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 52) took 1484 ms. Trace:
I20250901 01:52:26.395023 2113 rpcz_store.cc:276] 0901 01:52:24.910312 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:24.910360 (+ 48us) service_pool.cc:225] Handling call
0901 01:52:26.377055 (+1466695us) ts_tablet_manager.cc:1145] Deleting tablet 7f64ebd080944190be2e64d02d70a8a7
0901 01:52:26.377063 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:26.382941 (+ 5878us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:26.385715 (+ 2774us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:26.394872 (+ 9157us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1466550}
I20250901 01:52:26.395494 2113 tablet_replica.cc:331] T 25ce0fe8c9544f828694184d19d13df9 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:26.395684 2113 raft_consensus.cc:2241] T 25ce0fe8c9544f828694184d19d13df9 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:26.395884 2113 pending_rounds.cc:70] T 25ce0fe8c9544f828694184d19d13df9 P fd27ab6665304aaea53adbd102dd4f58: Trying to abort 1 pending ops.
I20250901 01:52:26.395936 2118 ts_tablet_manager.cc:1918] T 7a607fac57994001b2304e5d81c9decc P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:26.395946 2113 pending_rounds.cc:77] T 25ce0fe8c9544f828694184d19d13df9 P fd27ab6665304aaea53adbd102dd4f58: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7195408568354795520 op_type: NO_OP noop_request { }
I20250901 01:52:26.395972 2306 raft_consensus.cc:513] T 81c36693199945d98465579b509852e2 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:26.396080 2113 raft_consensus.cc:2887] T 25ce0fe8c9544f828694184d19d13df9 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250901 01:52:26.396121 2118 log.cc:1199] T 7a607fac57994001b2304e5d81c9decc P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/7a607fac57994001b2304e5d81c9decc
I20250901 01:52:26.396167 2113 raft_consensus.cc:2270] T 25ce0fe8c9544f828694184d19d13df9 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:26.396749 2113 ts_tablet_manager.cc:1905] T 25ce0fe8c9544f828694184d19d13df9 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:26.397101 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 7f64ebd080944190be2e64d02d70a8a7 (table default.Table_f94fe76831a54c6e8c2b13c81c251b14 [id=8b4cf5984d174d71a736959f5db32144]) successfully deleted
I20250901 01:52:26.386587 992 leader_election.cc:304] T f80f13db501c42c7950499df8fcb6fc8 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:26.398281 1761 raft_consensus.cc:2802] T f80f13db501c42c7950499df8fcb6fc8 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:26.398504 1761 raft_consensus.cc:695] T f80f13db501c42c7950499df8fcb6fc8 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:26.398726 2305 raft_consensus.cc:513] T e249474dcbf642189560c91ec3a99fdf P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:26.398789 1761 consensus_queue.cc:237] T f80f13db501c42c7950499df8fcb6fc8 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:26.399320 2305 leader_election.cc:290] T e249474dcbf642189560c91ec3a99fdf P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:26.399610 2123 raft_consensus.cc:491] T 500e3c33598b45a4afa983da7cb0845a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:26.399751 2123 raft_consensus.cc:513] T 500e3c33598b45a4afa983da7cb0845a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:26.355090 1150 ts_tablet_manager.cc:1428] T b13afacd37474781877b9f2cd20b313a P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20250901 01:52:26.402029 858 leader_election.cc:336] T 879fa66e55544118aafe504587393c66 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:26.402189 858 leader_election.cc:336] T 06d5eeb70e664f73bf643922d445bc7a P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:26.404145 1993 raft_consensus.cc:513] T ad45c90ba0da43c9860745e534664a98 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:26.404624 2301 tablet.cc:1722] T e1d18e999b5e42b1873631e0f1445081 P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:26.405370 2306 leader_election.cc:290] T 81c36693199945d98465579b509852e2 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:26.405894 1993 leader_election.cc:290] T ad45c90ba0da43c9860745e534664a98 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:26.412724 992 leader_election.cc:304] T 3618a3eb4f174c43805ce7dc0e5315cb P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:26.413268 1993 raft_consensus.cc:2802] T 3618a3eb4f174c43805ce7dc0e5315cb P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:26.413481 1993 raft_consensus.cc:491] T 3618a3eb4f174c43805ce7dc0e5315cb P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:26.413668 1993 raft_consensus.cc:3058] T 3618a3eb4f174c43805ce7dc0e5315cb P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:26.414384 2113 ts_tablet_manager.cc:1918] T 25ce0fe8c9544f828694184d19d13df9 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:26.414597 2113 log.cc:1199] T 25ce0fe8c9544f828694184d19d13df9 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/25ce0fe8c9544f828694184d19d13df9
I20250901 01:52:26.417709 1993 raft_consensus.cc:513] T 3618a3eb4f174c43805ce7dc0e5315cb P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:26.414590 992 leader_election.cc:336] T f80f13db501c42c7950499df8fcb6fc8 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:26.418594 992 leader_election.cc:336] T ad45c90ba0da43c9860745e534664a98 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:26.419378 858 leader_election.cc:336] T 89d40f0331114490b51de3ba6d773571 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:26.420769 992 leader_election.cc:336] T 46bfbdff3f904715a8c24907b20fcab0 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:26.421469 992 leader_election.cc:336] T 81c36693199945d98465579b509852e2 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:26.421732 992 leader_election.cc:336] T 82d163681fda47f38d79863bd28f576d P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:26.421860 992 leader_election.cc:304] T 82d163681fda47f38d79863bd28f576d P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:26.422151 1761 raft_consensus.cc:2747] T 82d163681fda47f38d79863bd28f576d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:26.422224 992 leader_election.cc:336] T b73e362321284df1b18ef755dfe930a5 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:26.424125 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a289a66b2dfa44dc8c8547ff83c1716e" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:26.426590 1150 tablet_bootstrap.cc:492] T b85d64e02cc646e380581ac9f59dee9e P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:26.426564 914 raft_consensus.cc:2466] T a289a66b2dfa44dc8c8547ff83c1716e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
W20250901 01:52:26.426980 858 leader_election.cc:336] T 500e3c33598b45a4afa983da7cb0845a P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:26.428123 2123 leader_election.cc:290] T 500e3c33598b45a4afa983da7cb0845a P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:26.428262 2123 raft_consensus.cc:491] T 6cf09e8f299548dba2cce16d7b565bad P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:26.428426 2123 raft_consensus.cc:513] T 6cf09e8f299548dba2cce16d7b565bad P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:26.429011 2123 leader_election.cc:290] T 6cf09e8f299548dba2cce16d7b565bad P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:26.422341 992 leader_election.cc:304] T b73e362321284df1b18ef755dfe930a5 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:26.418869 1993 leader_election.cc:290] T 3618a3eb4f174c43805ce7dc0e5315cb P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:26.430850 858 leader_election.cc:336] T 6cf09e8f299548dba2cce16d7b565bad P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:26.419221 1151 ts_tablet_manager.cc:1428] T 6cf09e8f299548dba2cce16d7b565bad P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.068s user 0.003s sys 0.000s
W20250901 01:52:26.431159 992 leader_election.cc:336] T c95798f0daf4437db18a291c4464522e P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:26.431370 992 leader_election.cc:304] T c95798f0daf4437db18a291c4464522e P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
W20250901 01:52:26.431686 992 leader_election.cc:336] T 66a48a8ebe154b2cac89c234eaa378ff P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:26.431813 992 leader_election.cc:304] T 66a48a8ebe154b2cac89c234eaa378ff P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
W20250901 01:52:26.432042 992 leader_election.cc:336] T 15e4d972e8254cf899c3e0b71b6e2bd9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:26.432260 992 consensus_peers.cc:489] T 85322a864d154f299f2d670fdd2a7bfb P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250901 01:52:26.432281 2306 raft_consensus.cc:2747] T 66a48a8ebe154b2cac89c234eaa378ff P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:26.432427 992 leader_election.cc:336] T d0c3c363dda444519a17dca63ad9caa7 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:26.431159 1761 raft_consensus.cc:2747] T b73e362321284df1b18ef755dfe930a5 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:26.432554 992 leader_election.cc:304] T d0c3c363dda444519a17dca63ad9caa7 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:26.426728 2113 ts_tablet_manager.cc:1939] T 25ce0fe8c9544f828694184d19d13df9 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:26.431690 1993 raft_consensus.cc:2747] T c95798f0daf4437db18a291c4464522e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:26.432870 1761 raft_consensus.cc:2747] T d0c3c363dda444519a17dca63ad9caa7 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250901 01:52:26.433146 1151 tablet_bootstrap.cc:492] T a8483e7212d94daba7e5401256c33aef P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:26.433322 859 leader_election.cc:304] T d0c3c363dda444519a17dca63ad9caa7 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:26.433758 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 53) took 1520 ms. Trace:
I20250901 01:52:26.433856 2113 rpcz_store.cc:276] 0901 01:52:24.912907 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:24.912952 (+ 45us) service_pool.cc:225] Handling call
0901 01:52:26.395392 (+1482440us) ts_tablet_manager.cc:1145] Deleting tablet 25ce0fe8c9544f828694184d19d13df9
0901 01:52:26.395402 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:26.411395 (+ 15993us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:26.414278 (+ 2883us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:26.433694 (+ 19416us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1481990}
I20250901 01:52:26.434211 2113 tablet_replica.cc:331] T af2c4bcbb00741f789edee109252d8cb P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:26.434438 2113 raft_consensus.cc:2241] T af2c4bcbb00741f789edee109252d8cb P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:26.434651 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 25ce0fe8c9544f828694184d19d13df9 (table default.Table_dcacb44eed4241d3acbae153ebaad663 [id=0241aa3ab1984557a151461005c6b97c]) successfully deleted
I20250901 01:52:26.435204 2323 raft_consensus.cc:2747] T d0c3c363dda444519a17dca63ad9caa7 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250901 01:52:26.435377 2323 raft_consensus.cc:491] T 8ca5cf4e99014c0d84a3acad4ef713af P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:26.436162 2323 raft_consensus.cc:513] T 8ca5cf4e99014c0d84a3acad4ef713af P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:26.436580 859 leader_election.cc:343] T fa7fb88ded21432a8757f37ec3db5e71 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:26.436852 2113 raft_consensus.cc:2270] T af2c4bcbb00741f789edee109252d8cb P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:26.437005 859 leader_election.cc:304] T fa7fb88ded21432a8757f37ec3db5e71 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:26.437343 859 leader_election.cc:343] T 8ed6616ce54548ba83ba299c4ecf3c38 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:26.437404 1151 tablet_bootstrap.cc:654] T a8483e7212d94daba7e5401256c33aef P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:26.438794 859 leader_election.cc:304] T 8ed6616ce54548ba83ba299c4ecf3c38 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:26.439406 2113 ts_tablet_manager.cc:1905] T af2c4bcbb00741f789edee109252d8cb P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:26.440688 2123 raft_consensus.cc:2747] T fa7fb88ded21432a8757f37ec3db5e71 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:26.440886 2323 leader_election.cc:290] T 8ca5cf4e99014c0d84a3acad4ef713af P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:26.441299 859 leader_election.cc:343] T 3239e58531d445fa80165cf678bc277c P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:26.440590 1488 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 1495de70f0844f82a0eea99c059f076b (table default.Table_9f48256fcac7429ba7d7fdb86979eda6 [id=70a5ce71d16b47d4bc6dae97be4548e7]). Expected version 2 got 0
I20250901 01:52:26.442390 859 leader_election.cc:304] T 3239e58531d445fa80165cf678bc277c P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:26.442757 859 leader_election.cc:304] T 7a2f6e2c7ee2402ea310c058e046416c P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
W20250901 01:52:26.442996 992 leader_election.cc:336] T e249474dcbf642189560c91ec3a99fdf P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:26.444460 1150 tablet_bootstrap.cc:654] T b85d64e02cc646e380581ac9f59dee9e P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:26.444489 2253 raft_consensus.cc:2747] T 3239e58531d445fa80165cf678bc277c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:26.444984 1488 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet ed1b41340ce742a88c3e4fd85cd25ca6 (table default.Table_139f6d174d794a6e9a618e90587d0c0d [id=0baa4993b72e4a8d94ee3059c9d2db6c]). Expected version 1 got 0
I20250901 01:52:26.445190 1488 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet d10c193452954c198bba686169b8025b (table default.Table_03fb48eaab9b42858ab7cefb722e43a8 [id=6c0576410d7340d6aa493a747e95565b]). Expected version 1 got 0
I20250901 01:52:26.445415 1488 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 435b1cca5a7d4d49959a310cb250b2a0 (table default.Table_b989608c88204be2b6f706a9426c47bb [id=126b298fef6c4653952d672adff1db7e]). Expected version 2 got 0
I20250901 01:52:26.445842 1488 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet e87d1281f35648b0b115b67c95e244e7 (table default.Table_f840789a58984799bd8057128f6ea439 [id=4453959e273c473fadd737f56f7db3dd]). Expected version 1 got 0
I20250901 01:52:26.446110 1488 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 7954393c8c1c4457ab8bb723c03a9c8b (table default.Table_dd01ae355f73448cbca96f086b04a2fe [id=543f2074598d4c29972e564be1123fca]). Expected version 3 got 0
I20250901 01:52:26.446375 2323 raft_consensus.cc:2747] T 8ed6616ce54548ba83ba299c4ecf3c38 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:26.446280 1488 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 6c0abbfc8d904dce989ce51961e4237d (table default.Table_3f9422b08b0349b9a5466218f0fe1b06 [id=f0dfb3811e744bd4ba6d59195cebde6f]). Expected version 2 got 0
I20250901 01:52:26.446729 1488 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet c6f7c525f59b49938b697de98d4b5835 (table default.Table_db61e055dbd7450898e82c4c0c4763be [id=aabca67f6f754820977ac9c4c20f3e4f]). Expected version 1 got 0
I20250901 01:52:26.447489 2123 raft_consensus.cc:2802] T 7a2f6e2c7ee2402ea310c058e046416c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:26.448626 2123 raft_consensus.cc:491] T 7a2f6e2c7ee2402ea310c058e046416c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:26.448793 2123 raft_consensus.cc:3058] T 7a2f6e2c7ee2402ea310c058e046416c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:26.448491 2118 ts_tablet_manager.cc:1939] T 7a607fac57994001b2304e5d81c9decc P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
W20250901 01:52:26.449216 858 leader_election.cc:336] T 8ca5cf4e99014c0d84a3acad4ef713af P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:26.449824 1488 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 8ed6616ce54548ba83ba299c4ecf3c38 (table default.Table_1bdd9041d74045cab6f23de11bf4dd09 [id=946ec3e53dcd4de4b8501b3e87e99072]). Expected version 2 got 0
I20250901 01:52:26.450066 1488 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 081e2630e284498aa819f85bf8326315 (table default.Table_9e7ca61fcc324a8984bd1e470162c61f [id=8199118617cb4707bc3ae9b1363750f7]). Expected version 2 got 0
I20250901 01:52:26.450623 1488 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 81ce448b148447ddbc41875d863e0518 (table default.Table_6e461d9cb0ae4dd095975bbc21533794 [id=49673fa73f8c436e9b8026efc4f3cbbc]). Expected version 1 got 0
I20250901 01:52:26.459450 1488 catalog_manager.cc:5582] T 42c1156e8dd84c6fa1396b46de13eb6f P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } }
I20250901 01:52:26.460796 1488 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 42c1156e8dd84c6fa1396b46de13eb6f (table default.Table_b5d681f51fc84395bbabe1e5b4cfb358 [id=40d05d90b0354334840ad17764f5b6bb]). Expected version 1 got 0
I20250901 01:52:26.461056 2113 ts_tablet_manager.cc:1918] T af2c4bcbb00741f789edee109252d8cb P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:26.461282 2113 log.cc:1199] T af2c4bcbb00741f789edee109252d8cb P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/af2c4bcbb00741f789edee109252d8cb
I20250901 01:52:26.461200 1488 catalog_manager.cc:5582] T aa424ac6ef1441e6b1cc2867b1c48186 P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: UNKNOWN } } }
I20250901 01:52:26.462574 1488 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet aa424ac6ef1441e6b1cc2867b1c48186 (table default.Table_bbbde49530e34a76b7cb17988ac6baca [id=76928d8caa2f4134a014656cf2cc871a]). Expected version 1 got 0
I20250901 01:52:26.462610 1872 raft_consensus.cc:491] T c754129977904f978f87fd89d7092e36 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:26.467952 1488 catalog_manager.cc:5582] T f4fd117c395e41c2ab679c684908c5b7 P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: UNKNOWN } } }
I20250901 01:52:26.468297 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 42) took 1276 ms. Trace:
I20250901 01:52:26.468688 1488 catalog_manager.cc:5582] T 616ff34e8d314e0aa0e11ce60be11789 P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: UNKNOWN } } }
I20250901 01:52:26.469106 1488 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 616ff34e8d314e0aa0e11ce60be11789 (table default.Table_0451d0afa26f4d5f8d4214a946eea04f [id=5ee13d46e06a4fa59b1070547573e523]). Expected version 1 got 0
I20250901 01:52:26.469071 2123 raft_consensus.cc:513] T 7a2f6e2c7ee2402ea310c058e046416c P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:26.469424 1488 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet e1d18e999b5e42b1873631e0f1445081 (table default.Table_fd17a2ad225546aa821489ce94a788a2 [id=033c76e0598342c495ce2be42a0df5dd]). Expected version 1 got 0
I20250901 01:52:26.469743 1488 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 49aaaec611784e6bac131efd58dbca51 (table default.Table_95e1338a49a04f22b679e363e253262e [id=e24b91b4ac904ef0a8c4725f70bced2f]). Expected version 1 got 0
I20250901 01:52:26.470543 1488 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet d0c3c363dda444519a17dca63ad9caa7 (table default.Table_0b4ac0a1caa5472d909448e41318e679 [id=8c1b263824ec4c15a5ec424364c78895]). Expected version 2 got 0
I20250901 01:52:26.471740 2271 tablet.cc:1722] T f62d96c37cca4143a4b098b7c69979e6 P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250901 01:52:26.472673 1488 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet e9f1f892658b4736859ff507536920a1 (table default.Table_8825dd28afbe464193eb1b668cc9326a [id=9aa62cb634314a0a849612c54c1220d9]). Expected version 1 got 0
I20250901 01:52:26.475188 2118 rpcz_store.cc:276] 0901 01:52:25.172986 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.173257 (+ 271us) service_pool.cc:225] Handling call
0901 01:52:26.308910 (+1135653us) ts_tablet_manager.cc:1145] Deleting tablet 7a607fac57994001b2304e5d81c9decc
0901 01:52:26.308920 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:26.392132 (+ 83212us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:26.395791 (+ 3659us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:26.449295 (+ 53504us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1135416}
I20250901 01:52:26.476140 2113 ts_tablet_manager.cc:1939] T af2c4bcbb00741f789edee109252d8cb P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
W20250901 01:52:26.476213 858 leader_election.cc:336] T 7a2f6e2c7ee2402ea310c058e046416c P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:26.477041 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 7a607fac57994001b2304e5d81c9decc (table default.Table_048a628c8a35498f94012e430b3c29bc [id=5014fdf96714467f98fd130c3b66f37a]) successfully deleted
I20250901 01:52:26.477245 2123 leader_election.cc:290] T 7a2f6e2c7ee2402ea310c058e046416c P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:26.477252 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 54) took 1563 ms. Trace:
I20250901 01:52:26.477353 2113 rpcz_store.cc:276] 0901 01:52:24.914134 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:24.914181 (+ 47us) service_pool.cc:225] Handling call
0901 01:52:26.434130 (+1519949us) ts_tablet_manager.cc:1145] Deleting tablet af2c4bcbb00741f789edee109252d8cb
0901 01:52:26.434140 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:26.457032 (+ 22892us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:26.459944 (+ 2912us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:26.477216 (+ 17272us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1519803}
I20250901 01:52:26.478665 917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad45c90ba0da43c9860745e534664a98" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:26.479827 917 raft_consensus.cc:3058] T ad45c90ba0da43c9860745e534664a98 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:26.480690 2113 tablet_replica.cc:331] T 6cf09e8f299548dba2cce16d7b565bad P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:26.480886 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet af2c4bcbb00741f789edee109252d8cb (table default.Table_6f470109ee524f10a48d1080ec15a199 [id=dc57478deb384d52ae97315fcdce074a]) successfully deleted
I20250901 01:52:26.481268 922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "81c36693199945d98465579b509852e2" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:26.481498 922 raft_consensus.cc:3058] T 81c36693199945d98465579b509852e2 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:26.480819 1872 raft_consensus.cc:513] T c754129977904f978f87fd89d7092e36 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:26.482738 2113 raft_consensus.cc:2241] T 6cf09e8f299548dba2cce16d7b565bad P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:26.482918 2113 raft_consensus.cc:2270] T 6cf09e8f299548dba2cce16d7b565bad P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:26.483798 2113 ts_tablet_manager.cc:1905] T 6cf09e8f299548dba2cce16d7b565bad P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:26.484303 1872 leader_election.cc:290] T c754129977904f978f87fd89d7092e36 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:26.486586 1058 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c0abbfc8d904dce989ce51961e4237d" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:26.486686 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1f2d98712ff4ea6aec665512de068bd" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:26.487105 1058 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "89d40f0331114490b51de3ba6d773571" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:26.489606 1151 tablet_bootstrap.cc:492] T a8483e7212d94daba7e5401256c33aef P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:26.489825 1151 ts_tablet_manager.cc:1397] T a8483e7212d94daba7e5401256c33aef P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.058s user 0.013s sys 0.000s
I20250901 01:52:26.490754 1151 raft_consensus.cc:357] T a8483e7212d94daba7e5401256c33aef P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:26.491055 1151 raft_consensus.cc:383] T a8483e7212d94daba7e5401256c33aef P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:26.491189 1151 raft_consensus.cc:738] T a8483e7212d94daba7e5401256c33aef P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:26.491497 1151 consensus_queue.cc:260] T a8483e7212d94daba7e5401256c33aef P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:26.491602 1150 tablet_bootstrap.cc:492] T b85d64e02cc646e380581ac9f59dee9e P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:26.491571 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "879fa66e55544118aafe504587393c66" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:26.492297 1151 ts_tablet_manager.cc:1428] T a8483e7212d94daba7e5401256c33aef P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250901 01:52:26.492455 1150 ts_tablet_manager.cc:1397] T b85d64e02cc646e380581ac9f59dee9e P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.090s user 0.002s sys 0.010s
I20250901 01:52:26.492776 1151 tablet_bootstrap.cc:492] T eaa521c52a2f4408b59770bd2a05907c P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:26.494109 2118 tablet_replica.cc:331] T e9b179dba2cc49428ba8090c962d6136 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:26.494203 2113 ts_tablet_manager.cc:1918] T 6cf09e8f299548dba2cce16d7b565bad P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:26.494395 2113 log.cc:1199] T 6cf09e8f299548dba2cce16d7b565bad P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/6cf09e8f299548dba2cce16d7b565bad
I20250901 01:52:26.494532 1488 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 5f5396045979450193d461ba1334a5c9 (table default.Table_a89cc8e48563460baabc34c821ceae35 [id=9c3930e9fcfe4b49ba4bea67a2b0ad07]). Expected version 1 got 0
I20250901 01:52:26.492504 2354 tablet.cc:1722] T e1d18e999b5e42b1873631e0f1445081 P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:26.494850 2113 ts_tablet_manager.cc:1939] T 6cf09e8f299548dba2cce16d7b565bad P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:26.494135 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e249474dcbf642189560c91ec3a99fdf" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:26.495239 2118 raft_consensus.cc:2241] T e9b179dba2cc49428ba8090c962d6136 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:26.496091 1488 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet b164b4a67a9143ceaff69b0ce49d63fe (table default.Table_da2c70a8f2fe49a0a8184e88c0a9e0ea [id=c1b1e9e3876244f68d31c63ce7bdebf1]). Expected version 1 got 0
I20250901 01:52:26.496341 1488 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet ae21b9ae2e3b472eb67fb5f96388a50f (table default.Table_046289f393ff432aae0b033bdec1fb17 [id=db57b18a9ce24e068b362f5fbe940531]). Expected version 1 got 0
I20250901 01:52:26.496646 917 raft_consensus.cc:2466] T ad45c90ba0da43c9860745e534664a98 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:26.496647 1488 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 36e10328c34242ebb4dbd8f294162696 (table default.Table_8445fcd774874cadaf4acc3581eee417 [id=958e51a6bbee47638e504e1a44094737]). Expected version 1 got 0
I20250901 01:52:26.499251 913 raft_consensus.cc:3058] T e249474dcbf642189560c91ec3a99fdf P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:26.499217 1150 raft_consensus.cc:357] T b85d64e02cc646e380581ac9f59dee9e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:26.499393 1150 raft_consensus.cc:383] T b85d64e02cc646e380581ac9f59dee9e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:26.499480 1150 raft_consensus.cc:738] T b85d64e02cc646e380581ac9f59dee9e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:26.499701 1150 consensus_queue.cc:260] T b85d64e02cc646e380581ac9f59dee9e P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:26.500236 1150 ts_tablet_manager.cc:1428] T b85d64e02cc646e380581ac9f59dee9e P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.008s user 0.000s sys 0.001s
I20250901 01:52:26.500500 2118 pending_rounds.cc:70] T e9b179dba2cc49428ba8090c962d6136 P 00c282f779c1477d920f24dc0dc16628: Trying to abort 1 pending ops.
I20250901 01:52:26.500581 2118 pending_rounds.cc:77] T e9b179dba2cc49428ba8090c962d6136 P 00c282f779c1477d920f24dc0dc16628: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7195408566788640768 op_type: NO_OP noop_request { }
I20250901 01:52:26.500706 2118 raft_consensus.cc:2887] T e9b179dba2cc49428ba8090c962d6136 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250901 01:52:26.500826 2118 raft_consensus.cc:2270] T e9b179dba2cc49428ba8090c962d6136 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:26.501644 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 55) took 1586 ms. Trace:
I20250901 01:52:26.501765 2113 rpcz_store.cc:276] 0901 01:52:24.914765 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:24.914812 (+ 47us) service_pool.cc:225] Handling call
0901 01:52:26.480612 (+1565800us) ts_tablet_manager.cc:1145] Deleting tablet 6cf09e8f299548dba2cce16d7b565bad
0901 01:52:26.480623 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:26.486836 (+ 6213us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:26.493671 (+ 6835us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:26.501590 (+ 7919us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1565667}
I20250901 01:52:26.501933 913 raft_consensus.cc:2466] T e249474dcbf642189560c91ec3a99fdf P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:26.502182 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 56) took 1585 ms. Trace:
I20250901 01:52:26.502265 2113 rpcz_store.cc:276] 0901 01:52:24.916242 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:24.916284 (+ 42us) service_pool.cc:225] Handling call
0901 01:52:26.502052 (+1585768us) ts_tablet_manager.cc:1145] Deleting tablet 2240302d9178445d8700e35f5ea274f7
0901 01:52:26.502061 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:26.502132 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1585617}
I20250901 01:52:26.502542 2113 tablet_replica.cc:331] T b7b802f976f3471faaaecc964ec87e48 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:26.501945 2118 ts_tablet_manager.cc:1905] T e9b179dba2cc49428ba8090c962d6136 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:26.502823 2113 raft_consensus.cc:2241] T b7b802f976f3471faaaecc964ec87e48 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:26.502558 922 raft_consensus.cc:2466] T 81c36693199945d98465579b509852e2 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:26.503194 2113 raft_consensus.cc:2270] T b7b802f976f3471faaaecc964ec87e48 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:26.503463 1150 tablet_bootstrap.cc:492] T 31614db4c53a4f3c96fd190d326ce601 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:26.503512 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "06d5eeb70e664f73bf643922d445bc7a" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:26.504086 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "500e3c33598b45a4afa983da7cb0845a" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:26.505013 2113 ts_tablet_manager.cc:1905] T b7b802f976f3471faaaecc964ec87e48 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:26.505422 1150 tablet_bootstrap.cc:654] T 31614db4c53a4f3c96fd190d326ce601 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:26.509855 1488 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 93bcac1fd95d44dc927abfca81692921 (table default.Table_b8f8ee74648c406eaf589d3ab80934db [id=b8f69313439543be9c11eea88dfa69c5]). Expected version 1 got 0
I20250901 01:52:26.510139 1488 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 508406601c994e42a5c5b1f665e74957 (table default.Table_345b7f9fe8214aa6878f20a166125364 [id=4aabc328f6374ec5b1dea26f778208b2]). Expected version 1 got 0
I20250901 01:52:26.511386 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8ca5cf4e99014c0d84a3acad4ef713af" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:26.511608 1056 raft_consensus.cc:2466] T 8ca5cf4e99014c0d84a3acad4ef713af P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00c282f779c1477d920f24dc0dc16628 in term 0.
I20250901 01:52:26.511996 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a2f6e2c7ee2402ea310c058e046416c" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58"
I20250901 01:52:26.512159 1056 raft_consensus.cc:3058] T 7a2f6e2c7ee2402ea310c058e046416c P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:26.502287 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 6cf09e8f299548dba2cce16d7b565bad (table default.Table_749fcd4bf97d439c9fd49b138a52a3fd [id=24fffc7ebf5a47bdb37211910301acf7]) successfully deleted
W20250901 01:52:26.512661 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 2240302d9178445d8700e35f5ea274f7 because the tablet was not found. No further retry: Not found: Tablet not found: 2240302d9178445d8700e35f5ea274f7
I20250901 01:52:26.514727 1056 raft_consensus.cc:2466] T 7a2f6e2c7ee2402ea310c058e046416c P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c282f779c1477d920f24dc0dc16628 in term 1.
W20250901 01:52:26.515760 992 leader_election.cc:336] T a289a66b2dfa44dc8c8547ff83c1716e P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:26.516345 992 leader_election.cc:336] T 3618a3eb4f174c43805ce7dc0e5315cb P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:26.516842 922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3618a3eb4f174c43805ce7dc0e5315cb" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:26.494957 1151 tablet_bootstrap.cc:654] T eaa521c52a2f4408b59770bd2a05907c P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:26.517045 922 raft_consensus.cc:3058] T 3618a3eb4f174c43805ce7dc0e5315cb P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:26.518266 1488 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 7f46e9ddadaf4ff986d8c991178daa0c (table default.Table_9978497447394e5694ae928412634ecb [id=ea928501fe954cd0b81d38add4ef7128]). Expected version 2 got 0
I20250901 01:52:26.518837 1488 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 3dae00aa38204ee3ac77325eb334be42 (table default.Table_faf7ef12324f454396e5ae64d64ebe51 [id=5812902df9ea406ba59ff0ff23f130b6]). Expected version 1 got 0
I20250901 01:52:26.519075 1488 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet cb7d7e1796c0456f8de98fb812b9e374 (table default.Table_c4148158ed7b48a3946559ba978e72fd [id=327c5a63904540508142ab600c308757]). Expected version 1 got 0
I20250901 01:52:26.519304 1488 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet b009cc95c819431ea9827a3e453b8da3 (table default.Table_8dd9c9817a5c4e3dbb0bbf1d5cf1b5db [id=cc26c89db2484a6fad919231116285f2]). Expected version 2 got 1
W20250901 01:52:26.519469 859 leader_election.cc:343] T d1f2d98712ff4ea6aec665512de068bd P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:26.519516 1488 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 33f637f6a8134215bb1995ec38440073 (table default.Table_2ece91972d494680a8ad00273c10dee4 [id=557afe29976d40f6bfb6ae4ca348f887]). Expected version 1 got 0
I20250901 01:52:26.519577 859 leader_election.cc:304] T d1f2d98712ff4ea6aec665512de068bd P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:26.519971 859 leader_election.cc:343] T 6c0abbfc8d904dce989ce51961e4237d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:26.520066 859 leader_election.cc:304] T 6c0abbfc8d904dce989ce51961e4237d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:26.520440 859 leader_election.cc:343] T 89d40f0331114490b51de3ba6d773571 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:26.520529 859 leader_election.cc:304] T 89d40f0331114490b51de3ba6d773571 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:26.520756 992 leader_election.cc:304] T a289a66b2dfa44dc8c8547ff83c1716e P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
W20250901 01:52:26.520862 859 leader_election.cc:343] T 879fa66e55544118aafe504587393c66 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:26.520953 859 leader_election.cc:304] T 879fa66e55544118aafe504587393c66 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:26.521250 859 leader_election.cc:343] T 06d5eeb70e664f73bf643922d445bc7a P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:26.521334 859 leader_election.cc:304] T 06d5eeb70e664f73bf643922d445bc7a P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:26.521622 2253 raft_consensus.cc:2747] T d1f2d98712ff4ea6aec665512de068bd P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:26.521787 2253 raft_consensus.cc:2747] T 6c0abbfc8d904dce989ce51961e4237d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:26.521881 2253 raft_consensus.cc:2747] T 89d40f0331114490b51de3ba6d773571 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:26.521986 2253 raft_consensus.cc:2747] T 879fa66e55544118aafe504587393c66 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:26.522085 2253 raft_consensus.cc:2747] T 06d5eeb70e664f73bf643922d445bc7a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:26.525746 2113 ts_tablet_manager.cc:1918] T b7b802f976f3471faaaecc964ec87e48 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:26.525892 2113 log.cc:1199] T b7b802f976f3471faaaecc964ec87e48 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/b7b802f976f3471faaaecc964ec87e48
W20250901 01:52:26.526732 859 leader_election.cc:343] T 500e3c33598b45a4afa983da7cb0845a P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:26.526887 1488 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet f80f13db501c42c7950499df8fcb6fc8 (table default.Table_ce86974fd8974de9a5a737954fac2ad7 [id=ff772d20b287476f94716379b86d2a71]). Expected version 1 got 0
I20250901 01:52:26.526932 859 leader_election.cc:304] T 500e3c33598b45a4afa983da7cb0845a P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:26.527151 1488 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 3618a3eb4f174c43805ce7dc0e5315cb (table default.Table_83f2c0acedbc4c6982caf5313646e90b [id=31a2e50c32644321825da40176127015]). Expected version 1 got 0
I20250901 01:52:26.527328 1488 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet d1f2d98712ff4ea6aec665512de068bd (table default.Table_82d91a1ba2eb4265896824028dec8d3a [id=3f7624d2901445e39155c12fa68e34e9]). Expected version 1 got 0
W20250901 01:52:26.527514 859 leader_election.cc:343] T 6cf09e8f299548dba2cce16d7b565bad P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Not found: Tablet not found: 6cf09e8f299548dba2cce16d7b565bad
I20250901 01:52:26.527665 1488 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 615b2306fca84003b264da9d5f95ddab (table default.Table_302f6e7cf97d46b9a621eb852e6aefe0 [id=cecdd8d6ca6d445c9fd82decd2a95366]). Expected version 2 got 0
I20250901 01:52:26.527660 859 leader_election.cc:304] T 6cf09e8f299548dba2cce16d7b565bad P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:26.528048 2323 raft_consensus.cc:2747] T 500e3c33598b45a4afa983da7cb0845a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:26.528151 859 leader_election.cc:304] T 8ca5cf4e99014c0d84a3acad4ef713af P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:26.528262 2323 raft_consensus.cc:2747] T 6cf09e8f299548dba2cce16d7b565bad P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:26.528501 2123 raft_consensus.cc:2802] T 8ca5cf4e99014c0d84a3acad4ef713af P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:26.528613 2123 raft_consensus.cc:491] T 8ca5cf4e99014c0d84a3acad4ef713af P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:26.528614 1488 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet ad45c90ba0da43c9860745e534664a98 (table default.Table_1cd3f3fb6b244b3f8796bfca9f7622c8 [id=6fd739b0e725424ca844dbb652ea8028]). Expected version 2 got 0
I20250901 01:52:26.528738 2123 raft_consensus.cc:3058] T 8ca5cf4e99014c0d84a3acad4ef713af P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:26.528848 1488 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet fecf7d538b4c4c619182d5eccb44dd8b (table default.Table_1b5729d9768546fe99eaafc4e2604f42 [id=02d0056abe2146ac9af81ebf4447b99d]). Expected version 1 got 0
I20250901 01:52:26.529279 1488 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 8ca5cf4e99014c0d84a3acad4ef713af (table default.Table_ed6fd284be6f4ae3a008e21e7ce17930 [id=303e76c918bc4c39aa47a4d24849a358]). Expected version 1 got 0
I20250901 01:52:26.529531 1993 raft_consensus.cc:2802] T a289a66b2dfa44dc8c8547ff83c1716e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:26.531068 992 leader_election.cc:304] T ad45c90ba0da43c9860745e534664a98 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:26.531294 1761 raft_consensus.cc:2802] T ad45c90ba0da43c9860745e534664a98 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:26.531600 1993 raft_consensus.cc:491] T a289a66b2dfa44dc8c8547ff83c1716e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:26.531517 2123 raft_consensus.cc:513] T 8ca5cf4e99014c0d84a3acad4ef713af P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:26.531706 1993 raft_consensus.cc:3058] T a289a66b2dfa44dc8c8547ff83c1716e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:26.531754 1150 tablet_bootstrap.cc:492] T 31614db4c53a4f3c96fd190d326ce601 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:26.531915 1150 ts_tablet_manager.cc:1397] T 31614db4c53a4f3c96fd190d326ce601 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.029s user 0.008s sys 0.000s
I20250901 01:52:26.532169 2123 leader_election.cc:290] T 8ca5cf4e99014c0d84a3acad4ef713af P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:26.532963 859 leader_election.cc:304] T 7a2f6e2c7ee2402ea310c058e046416c P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:26.533262 2301 tablet.cc:1722] T 49aaaec611784e6bac131efd58dbca51 P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:26.533474 2123 raft_consensus.cc:2802] T 7a2f6e2c7ee2402ea310c058e046416c P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:26.533684 2123 raft_consensus.cc:695] T 7a2f6e2c7ee2402ea310c058e046416c P 00c282f779c1477d920f24dc0dc16628 [term 1 LEADER]: Becoming Leader. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Running, Role: LEADER
I20250901 01:52:26.533902 2123 consensus_queue.cc:237] T 7a2f6e2c7ee2402ea310c058e046416c P 00c282f779c1477d920f24dc0dc16628 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:26.534216 1993 raft_consensus.cc:513] T a289a66b2dfa44dc8c8547ff83c1716e P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:26.534719 1993 leader_election.cc:290] T a289a66b2dfa44dc8c8547ff83c1716e P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:26.535111 2113 ts_tablet_manager.cc:1939] T b7b802f976f3471faaaecc964ec87e48 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:26.535501 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 57) took 1618 ms. Trace:
I20250901 01:52:26.535626 922 raft_consensus.cc:2466] T 3618a3eb4f174c43805ce7dc0e5315cb P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:26.535602 2113 rpcz_store.cc:276] 0901 01:52:24.916849 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:24.916905 (+ 56us) service_pool.cc:225] Handling call
0901 01:52:26.502455 (+1585550us) ts_tablet_manager.cc:1145] Deleting tablet b7b802f976f3471faaaecc964ec87e48
0901 01:52:26.502464 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:26.508032 (+ 5568us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:26.510854 (+ 2822us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:26.535457 (+ 24603us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1585367}
I20250901 01:52:26.536034 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 58) took 1618 ms. Trace:
I20250901 01:52:26.536121 2113 rpcz_store.cc:276] 0901 01:52:24.917536 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:24.917589 (+ 53us) service_pool.cc:225] Handling call
0901 01:52:26.535891 (+1618302us) ts_tablet_manager.cc:1145] Deleting tablet 764bfeb2a7d14c40815b5ec1b07bdd62
0901 01:52:26.535900 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:26.536021 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1618032}
I20250901 01:52:26.536384 2113 tablet_replica.cc:331] T 4855448edb4b40c184838aa7f81c112b P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:26.536633 2113 raft_consensus.cc:2241] T 4855448edb4b40c184838aa7f81c112b P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:26.536857 2113 pending_rounds.cc:70] T 4855448edb4b40c184838aa7f81c112b P fd27ab6665304aaea53adbd102dd4f58: Trying to abort 1 pending ops.
I20250901 01:52:26.536827 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet b7b802f976f3471faaaecc964ec87e48 (table default.Table_3552b4561f10466b9b77f797ca003e53 [id=ec380a4600f34080bc1e343130537377]) successfully deleted
W20250901 01:52:26.537083 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 764bfeb2a7d14c40815b5ec1b07bdd62 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 764bfeb2a7d14c40815b5ec1b07bdd62 already in progress: creating tablet
W20250901 01:52:26.537950 858 leader_election.cc:336] T 8ca5cf4e99014c0d84a3acad4ef713af P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:26.532756 1150 raft_consensus.cc:357] T 31614db4c53a4f3c96fd190d326ce601 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:26.539225 1150 raft_consensus.cc:383] T 31614db4c53a4f3c96fd190d326ce601 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:26.539328 1150 raft_consensus.cc:738] T 31614db4c53a4f3c96fd190d326ce601 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:26.539604 1150 consensus_queue.cc:260] T 31614db4c53a4f3c96fd190d326ce601 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:26.540143 1150 ts_tablet_manager.cc:1428] T 31614db4c53a4f3c96fd190d326ce601 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250901 01:52:26.540445 1150 tablet_bootstrap.cc:492] T 9fbd44641d9e43c3b592e388e4adf350 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:26.537819 1761 raft_consensus.cc:695] T ad45c90ba0da43c9860745e534664a98 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:26.544593 1150 tablet_bootstrap.cc:654] T 9fbd44641d9e43c3b592e388e4adf350 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:26.544579 1761 consensus_queue.cc:237] T ad45c90ba0da43c9860745e534664a98 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:26.545672 1488 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 61ddcf86c9f34726939cc116434f0822 (table default.Table_63d0f2ead9d946d39ce40162ebd769d6 [id=f9dcf4937b1143248bfa9d88053fce17]). Expected version 3 got 0
I20250901 01:52:26.546216 1488 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 858218c4427c41e980b2b5c969a5af40 (table default.Table_6d47266af0ed433eb1f2abbc81d18e69 [id=264cc469a73841e79538843479a165ab]). Expected version 1 got 0
I20250901 01:52:26.546381 1488 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 89d40f0331114490b51de3ba6d773571 (table default.Table_4dacb3b1d2e94e42ac8937eb6eb58375 [id=8ee0aa1ba24042dd8ded004644913b9c]). Expected version 1 got 0
I20250901 01:52:26.548250 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_733616e391ff4d4ebc9cf3b2133f306e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.549407 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7ded73c0875d4f699c13d9d3a6275c22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.550292 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_813aecb6dc354adebd278ab932846de7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.551055 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_26a04ad2143d4a4fbab6cef57f32a864"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.551883 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a7a8c33df1db4944ae72d7b6e57369f0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.552623 1492 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet ddbc15685fca4c708329420109ad0c57
W20250901 01:52:26.552806 1492 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet ddbc15685fca4c708329420109ad0c57: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:26.553429 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1c044cbefafa45f79920eededecef25f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.553653 2118 ts_tablet_manager.cc:1918] T e9b179dba2cc49428ba8090c962d6136 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:26.553753 2118 log.cc:1199] T e9b179dba2cc49428ba8090c962d6136 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/e9b179dba2cc49428ba8090c962d6136
I20250901 01:52:26.554172 2118 ts_tablet_manager.cc:1939] T e9b179dba2cc49428ba8090c962d6136 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:26.554232 2113 pending_rounds.cc:77] T 4855448edb4b40c184838aa7f81c112b P fd27ab6665304aaea53adbd102dd4f58: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7195408572370026496 op_type: NO_OP noop_request { }
I20250901 01:52:26.554227 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8db587c822a04c498a52d09a275bdbd8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.554343 2113 raft_consensus.cc:2887] T 4855448edb4b40c184838aa7f81c112b P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250901 01:52:26.554425 2113 raft_consensus.cc:2270] T 4855448edb4b40c184838aa7f81c112b P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:26.554471 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 43) took 1380 ms. Trace:
I20250901 01:52:26.554543 2118 rpcz_store.cc:276] 0901 01:52:25.173735 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.174043 (+ 308us) service_pool.cc:225] Handling call
0901 01:52:26.475656 (+1301613us) ts_tablet_manager.cc:1145] Deleting tablet e9b179dba2cc49428ba8090c962d6136
0901 01:52:26.482518 (+ 6862us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:26.511291 (+ 28773us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:26.553532 (+ 42241us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:26.554454 (+ 922us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1301451}
I20250901 01:52:26.555048 2118 tablet_replica.cc:331] T 74b709cb5ec64006b2f12bead8217d80 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:26.554961 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_eaaf4e1083a54ee6b907de2c3953ad92"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.555229 2118 raft_consensus.cc:2241] T 74b709cb5ec64006b2f12bead8217d80 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:26.555364 2118 raft_consensus.cc:2270] T 74b709cb5ec64006b2f12bead8217d80 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:26.555686 992 leader_election.cc:304] T e249474dcbf642189560c91ec3a99fdf P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:26.555735 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1cd9a413d23e40cfb9d185955da7912e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.555951 1993 raft_consensus.cc:2802] T e249474dcbf642189560c91ec3a99fdf P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:26.556025 992 leader_election.cc:304] T 81c36693199945d98465579b509852e2 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:26.556238 2307 raft_consensus.cc:2802] T 81c36693199945d98465579b509852e2 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:26.556345 2307 raft_consensus.cc:695] T 81c36693199945d98465579b509852e2 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
W20250901 01:52:26.556293 992 leader_election.cc:343] T c754129977904f978f87fd89d7092e36 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: c754129977904f978f87fd89d7092e36
I20250901 01:52:26.556604 2307 consensus_queue.cc:237] T 81c36693199945d98465579b509852e2 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:26.557901 992 leader_election.cc:304] T 3618a3eb4f174c43805ce7dc0e5315cb P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:26.556147 1993 raft_consensus.cc:695] T e249474dcbf642189560c91ec3a99fdf P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:26.558183 2306 raft_consensus.cc:2802] T 3618a3eb4f174c43805ce7dc0e5315cb P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:26.558208 888 tablet_service.cc:1515] Processing DeleteTablet for tablet ddbc15685fca4c708329420109ad0c57 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:26 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:26.558283 2306 raft_consensus.cc:695] T 3618a3eb4f174c43805ce7dc0e5315cb P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:26.558223 1993 consensus_queue.cc:237] T e249474dcbf642189560c91ec3a99fdf P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:26.558461 2306 consensus_queue.cc:237] T 3618a3eb4f174c43805ce7dc0e5315cb P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:26.559396 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet ddbc15685fca4c708329420109ad0c57 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:26 UTC) from {username='slave'} at 127.0.0.1:35946
W20250901 01:52:26.560755 1484 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 33f637f6a8134215bb1995ec38440073 consensus state has no leader
W20250901 01:52:26.563982 1490 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1495de70f0844f82a0eea99c059f076b consensus state has no leader
W20250901 01:52:26.564079 992 leader_election.cc:336] T 3618a3eb4f174c43805ce7dc0e5315cb P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:26.565999 2118 ts_tablet_manager.cc:1905] T 74b709cb5ec64006b2f12bead8217d80 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:26.564365 992 leader_election.cc:336] T ad45c90ba0da43c9860745e534664a98 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:26.567564 992 consensus_peers.cc:489] T 6c05eff658924ca4a9a1e7a37f917d67 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250901 01:52:26.570547 1486 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0813cc68d4b0488ba60a523e4479971b has no consensus state
I20250901 01:52:26.570878 1486 master_service.cc:948] ReplaceTablet: received request to replace tablet 8ed6616ce54548ba83ba299c4ecf3c38 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:26.571192 1485 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: default.Table_2ceaff93c8f44f00aab53e48d8466dfd [id=4d9ea39016ca4bf4859f4f80ea4e9ed6] alter complete (version 4)
I20250901 01:52:26.574393 2323 raft_consensus.cc:491] T ff013ea889d44cba8797834e4611b489 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:26.574564 2323 raft_consensus.cc:513] T ff013ea889d44cba8797834e4611b489 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:26.575157 2323 leader_election.cc:290] T ff013ea889d44cba8797834e4611b489 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:26.576503 1483 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet da5f69c5b0ca4257ad2ff7918ee2af2e
W20250901 01:52:26.576689 1483 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet da5f69c5b0ca4257ad2ff7918ee2af2e: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:26.576745 1489 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_4f2be95a1b824f06ba16e56abcdebc8a" } modify_external_catalogs: true
I20250901 01:52:26.576891 1489 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_4f2be95a1b824f06ba16e56abcdebc8a" } modify_external_catalogs: true
W20250901 01:52:26.576923 858 leader_election.cc:336] T ff013ea889d44cba8797834e4611b489 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:26.577231 1483 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_8ac222c93e36436c9a3ecb2c489f6d74" } modify_external_catalogs: true
I20250901 01:52:26.577425 1483 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_8ac222c93e36436c9a3ecb2c489f6d74" } modify_external_catalogs: true
I20250901 01:52:26.577879 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet e9b179dba2cc49428ba8090c962d6136 (table default.Table_271fd01373bf49f2ab8b4faf82db7393 [id=d409bc5c1aa548c69feacab89bd3d3ef]) successfully deleted
I20250901 01:52:26.578487 1484 master_service.cc:948] ReplaceTablet: received request to replace tablet 36e10328c34242ebb4dbd8f294162696 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:26.578749 1490 master_service.cc:948] ReplaceTablet: received request to replace tablet c49d1143d54b4c319739488906b0e271 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:26.578927 1491 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_09ffc2a7b4b14b25adbc5471363c02af" } modify_external_catalogs: true
I20250901 01:52:26.579077 1491 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_09ffc2a7b4b14b25adbc5471363c02af" } modify_external_catalogs: true
I20250901 01:52:26.580319 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet da5f69c5b0ca4257ad2ff7918ee2af2e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:26 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:26.580533 898 tablet_service.cc:1515] Processing DeleteTablet for tablet da5f69c5b0ca4257ad2ff7918ee2af2e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:26 UTC) from {username='slave'} at 127.0.0.1:34472
W20250901 01:52:26.583484 992 leader_election.cc:336] T e249474dcbf642189560c91ec3a99fdf P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:26.584030 992 leader_election.cc:336] T 81c36693199945d98465579b509852e2 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:26.586081 992 consensus_peers.cc:489] T f4fd117c395e41c2ab679c684908c5b7 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:26.587397 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b48111aebc874bd2aaa54bd75d18f047"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.588198 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_81652036b9d042c48d8937d1c32b7d04"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.588999 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d2282a82b0ae4e35bdcf07b954f16cdc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.589777 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b95241149a9441c2b9223a02fd93818b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.590605 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b7a93a2fcc7949a29680f71ac3475644"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.591410 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_dd0c4228916c4c609b52dc545de958dd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.592206 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3532708ca2a34564937c8616d508fc75"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.592234 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a289a66b2dfa44dc8c8547ff83c1716e" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:26.592489 913 raft_consensus.cc:3058] T a289a66b2dfa44dc8c8547ff83c1716e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:26.595177 913 raft_consensus.cc:2466] T a289a66b2dfa44dc8c8547ff83c1716e P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:26.555850 2113 ts_tablet_manager.cc:1905] T 4855448edb4b40c184838aa7f81c112b P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:26.597942 1486 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 8ed6616ce54548ba83ba299c4ecf3c38
W20250901 01:52:26.598132 1486 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 8ed6616ce54548ba83ba299c4ecf3c38: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:26.598433 1486 catalog_manager.cc:6652] ReplaceTablet: tablet 8ed6616ce54548ba83ba299c4ecf3c38 deleted and replaced by tablet 8a13346dcb3d4b6ab35c261829d50ae5
I20250901 01:52:26.598716 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_816b41fdadfb41ae8b7c8a034b259659"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.599442 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c4511b46542542e4b52a6537f8a1bcf8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.600054 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8ca5cf4e99014c0d84a3acad4ef713af" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58"
I20250901 01:52:26.600134 1058 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff013ea889d44cba8797834e4611b489" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:26.600215 1056 raft_consensus.cc:3058] T 8ca5cf4e99014c0d84a3acad4ef713af P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:26.600903 898 tablet_service.cc:1515] Processing DeleteTablet for tablet 8ed6616ce54548ba83ba299c4ecf3c38 with delete_type TABLET_DATA_DELETED (replaced by tablet 8a13346dcb3d4b6ab35c261829d50ae5) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:26.601964 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet 8ed6616ce54548ba83ba299c4ecf3c38 with delete_type TABLET_DATA_DELETED (replaced by tablet 8a13346dcb3d4b6ab35c261829d50ae5) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:26.603016 1056 raft_consensus.cc:2466] T 8ca5cf4e99014c0d84a3acad4ef713af P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c282f779c1477d920f24dc0dc16628 in term 1.
I20250901 01:52:26.583771 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7a8743ea588f4c579113c439f93d89d3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.607466 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_643fcc4753764dfcb2aa262b4c1b2f8f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.608762 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_49094f9d79e148189221fa239d52ef71"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.609879 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5d0832c1e5f048ac9c37b54350d83076"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.610600 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4520886c24e24851b5a64d15e218c9e4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.610885 1151 tablet_bootstrap.cc:492] T eaa521c52a2f4408b59770bd2a05907c P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:26.611095 1151 ts_tablet_manager.cc:1397] T eaa521c52a2f4408b59770bd2a05907c P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.118s user 0.000s sys 0.010s
I20250901 01:52:26.611251 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d0d220bac7964e4fabbb77f4b5c2aaa8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.611948 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_da9cc2377b2c4288b8d6177b5cc70ad9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.612267 1151 raft_consensus.cc:357] T eaa521c52a2f4408b59770bd2a05907c P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:26.612534 1151 raft_consensus.cc:383] T eaa521c52a2f4408b59770bd2a05907c P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:26.612650 1151 raft_consensus.cc:738] T eaa521c52a2f4408b59770bd2a05907c P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:26.600028 1486 master_service.cc:948] ReplaceTablet: received request to replace tablet b6daeff0070a4db08e90020752d21c29 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:26.612918 1151 consensus_queue.cc:260] T eaa521c52a2f4408b59770bd2a05907c P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:26.613547 1151 ts_tablet_manager.cc:1428] T eaa521c52a2f4408b59770bd2a05907c P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250901 01:52:26.613973 1151 tablet_bootstrap.cc:492] T 081e2630e284498aa819f85bf8326315 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:26.613991 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_72d634bf55264d458ab159b8a245aee5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.614764 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ee44c76ad7804fd0a79fb67abebdb0a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.615891 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1d1059ef085446a3b775da6c5e4c40ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.616587 1151 tablet_bootstrap.cc:654] T 081e2630e284498aa819f85bf8326315 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:26.616972 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4e9493cfb4d240a2a008ffa8d73d4374"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.618069 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4f566a4e459d464eae252f93a00049b8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.619227 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_0e5fe4a2c3724469afb65a00c6f0c0b5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.619891 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2f8a46ad9c524a0ea2d7dd5fb1246734"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.620544 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_280ba6d1ce8a4f2db98ab72c5fbba490"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.625053 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet 31614db4c53a4f3c96fd190d326ce601 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:26.625801 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet 4902062977ea42149e3466e999117322 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:18 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:26.626329 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet 48a211e174054866b030cdfb0cf947f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:18 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:26.627018 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet 533d14f5903e49599a34ea353ffd8c00 with delete_type TABLET_DATA_DELETED (replaced by tablet a54af99bc10045e8b0b6868e5aed98b4) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:26.627653 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet 1a60c10e42104502ac2d3b80b8126ceb with delete_type TABLET_DATA_DELETED (replaced by tablet 88ceba7f48b04d7885cc0164668700a5) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:26.628232 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet 56475ba491a24301a534304811f30140 with delete_type TABLET_DATA_DELETED (replaced by tablet 7317eb7848ee44698038a4b404d4bf21) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:26.628712 2113 ts_tablet_manager.cc:1918] T 4855448edb4b40c184838aa7f81c112b P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:26.634960 2354 tablet.cc:1722] T 49aaaec611784e6bac131efd58dbca51 P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:26.635598 1150 tablet_bootstrap.cc:492] T 9fbd44641d9e43c3b592e388e4adf350 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:26.636653 1150 ts_tablet_manager.cc:1397] T 9fbd44641d9e43c3b592e388e4adf350 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.096s user 0.003s sys 0.012s
I20250901 01:52:26.629205 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet b81a12b38ac64efabd44aeb192d181a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:18 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:26.638031 2113 log.cc:1199] T 4855448edb4b40c184838aa7f81c112b P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/4855448edb4b40c184838aa7f81c112b
I20250901 01:52:26.638504 2113 ts_tablet_manager.cc:1939] T 4855448edb4b40c184838aa7f81c112b P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:26.623299 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_acb81e39ebda42dc8a97e89ea8d29f9f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:26.629568 1482 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 410d44e5f1974ed69e5dee65e98856a4 (table default.Table_bec114a616504798bede9a9066e57192 [id=ae79f2c40fd7483ea400a36cf3c48888], current schema version=4): Not found: failed to reset TS proxy: No leader found: Tablet 410d44e5f1974ed69e5dee65e98856a4 consensus state has no leader
E20250901 01:52:26.634009 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae4844183a3c4a04a775e6fd878391a3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_aee74198b51d4f089538c4f2bf3ce4dc'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:26.639216 1696 catalog_manager.cc:6205] Error selecting replicas for tablet be8dfbc466564842a2fdb4953eee939b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_75ba3ba70ca24a94920e7bce96190ff2'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:26.639336 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 96824e68bbf24b8287dbb683038ffaba, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_3c7c628f35fb4d02bb79a34995f12938'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:26.639472 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 7317eb7848ee44698038a4b404d4bf21, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b0e916c76a7a4766aa0887cc8691488f'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:26.639601 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9745582cebfe47b58c21d0191ab4cb9b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2f09538b6fbe401f909105decb56e1cb'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:26.639721 1696 catalog_manager.cc:6205] Error selecting replicas for tablet e2ed32b3292141278af28026528e96c2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_50ae3908a7944f70949c54be8fbf1564'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:26.639838 1696 catalog_manager.cc:6205] Error selecting replicas for tablet a2ca69fabadd4ad69e1670cae7a0c503, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_32c4716e3aa24a87a6b1472984b255c4'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:26.639988 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f8f923ddcd2740bcb7e521f4f57c4034, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_5e65c5f068b84995a2d7a76fa65dda0f'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:26.639998 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 59) took 1721 ms. Trace:
E20250901 01:52:26.640100 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 0813cc68d4b0488ba60a523e4479971b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d1302673d68f424ebb54b220911ce385'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:26.640213 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 787be682c0e44248a80d7037fb61dc6c, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7c12d66048a6470e8c227dc1d9f45fa4'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:26.640353 1696 catalog_manager.cc:6205] Error selecting replicas for tablet cc3b8736b9094d7e892c80da4d5b20f6, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_93afd37289414655980b22934695bd55'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:26.640514 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae9a4ad4fdfb4771ba23a91f7007447e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_5e84ae9a9c2a4084a1f9c5ffb4594de2'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:26.640734 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9074543dea9949e7a085717b40913348, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_53e30205c5fd488eb3b48c685a159d04'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:26.640782 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f810f4fe2e3b48b78e11036d07ab50db"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:26.640928 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 18498c9cb09141d08c1c5c0854b63ebd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77f4810bc83c4d948531da2a45a3792e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:26.641382 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 81319e373c9a4475abceeeedb507aabc, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_22d8d38d0b624912a0d8968fb764420e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:26.641594 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 6ce9521b925b4f97bb4a810ce77e147e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_9e04f976a93f4d30baccf90fa3622dfe'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:26.643158 1150 raft_consensus.cc:357] T 9fbd44641d9e43c3b592e388e4adf350 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:26.643352 1150 raft_consensus.cc:383] T 9fbd44641d9e43c3b592e388e4adf350 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:26.643417 1150 raft_consensus.cc:738] T 9fbd44641d9e43c3b592e388e4adf350 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:26.643590 1150 consensus_queue.cc:260] T 9fbd44641d9e43c3b592e388e4adf350 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:26.644057 1150 ts_tablet_manager.cc:1428] T 9fbd44641d9e43c3b592e388e4adf350 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.007s user 0.000s sys 0.000s
I20250901 01:52:26.644312 1150 tablet_bootstrap.cc:492] T eb19e3a424324a24824f6caaf5848952 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:26.634222 1489 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet a289a66b2dfa44dc8c8547ff83c1716e
W20250901 01:52:26.645291 1489 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet a289a66b2dfa44dc8c8547ff83c1716e: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:26.645776 1150 tablet_bootstrap.cc:654] T eb19e3a424324a24824f6caaf5848952 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:26.645995 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet a289a66b2dfa44dc8c8547ff83c1716e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:26 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:26.646751 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7e4bbb9f1aed42e794a922bb93f458fb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.646837 898 tablet_service.cc:1515] Processing DeleteTablet for tablet a289a66b2dfa44dc8c8547ff83c1716e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:26 UTC) from {username='slave'} at 127.0.0.1:34472
W20250901 01:52:26.646859 992 leader_election.cc:336] T c754129977904f978f87fd89d7092e36 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:26.646981 992 leader_election.cc:304] T c754129977904f978f87fd89d7092e36 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:26.647879 992 leader_election.cc:304] T a289a66b2dfa44dc8c8547ff83c1716e P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:26.647643 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_14d8720057254564a4b260ffceea7aff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.648118 1993 raft_consensus.cc:2747] T c754129977904f978f87fd89d7092e36 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:26.648310 1993 raft_consensus.cc:2802] T a289a66b2dfa44dc8c8547ff83c1716e P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:26.648448 1993 raft_consensus.cc:695] T a289a66b2dfa44dc8c8547ff83c1716e P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:26.648684 1993 consensus_queue.cc:237] T a289a66b2dfa44dc8c8547ff83c1716e P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:26.639850 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_226da0554dc54d7bb6dbfd120eaa47f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.653301 1761 consensus_queue.cc:1035] T f4fd117c395e41c2ab679c684908c5b7 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:26.634384 1485 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: default.Table_f0e56dba1e8a4ed49a760e204c1ff96e [id=c3865e058daa4cea9a89f5046fca712a] alter complete (version 1)
W20250901 01:52:26.656849 859 leader_election.cc:343] T ff013ea889d44cba8797834e4611b489 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:26.657064 859 leader_election.cc:304] T ff013ea889d44cba8797834e4611b489 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:26.655853 992 leader_election.cc:336] T a289a66b2dfa44dc8c8547ff83c1716e P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:26.657613 859 leader_election.cc:304] T 8ca5cf4e99014c0d84a3acad4ef713af P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:26.640211 2113 rpcz_store.cc:276] 0901 01:52:24.918480 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:24.918524 (+ 44us) service_pool.cc:225] Handling call
0901 01:52:26.536339 (+1617815us) ts_tablet_manager.cc:1145] Deleting tablet 4855448edb4b40c184838aa7f81c112b
0901 01:52:26.536347 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:26.599140 (+ 62793us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:26.628587 (+ 29447us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:26.639978 (+ 11391us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1617613}
I20250901 01:52:26.658207 2113 tablet_replica.cc:331] T c49d1143d54b4c319739488906b0e271 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:26.658373 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 4855448edb4b40c184838aa7f81c112b (table default.Table_d1302673d68f424ebb54b220911ce385 [id=453153553bea48319e132857f628d48a]) successfully deleted
I20250901 01:52:26.658423 2113 raft_consensus.cc:2241] T c49d1143d54b4c319739488906b0e271 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:26.658814 2357 raft_consensus.cc:2747] T ff013ea889d44cba8797834e4611b489 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:26.659016 2357 raft_consensus.cc:2802] T 8ca5cf4e99014c0d84a3acad4ef713af P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:26.659139 2307 raft_consensus.cc:491] T 23af154a6f1c4e2e9352fb17c9024f10 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:26.659204 2357 raft_consensus.cc:695] T 8ca5cf4e99014c0d84a3acad4ef713af P 00c282f779c1477d920f24dc0dc16628 [term 1 LEADER]: Becoming Leader. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Running, Role: LEADER
I20250901 01:52:26.659245 2307 raft_consensus.cc:513] T 23af154a6f1c4e2e9352fb17c9024f10 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:26.659474 2357 consensus_queue.cc:237] T 8ca5cf4e99014c0d84a3acad4ef713af P 00c282f779c1477d920f24dc0dc16628 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:26.659742 2307 leader_election.cc:290] T 23af154a6f1c4e2e9352fb17c9024f10 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:26.662225 2113 raft_consensus.cc:2270] T c49d1143d54b4c319739488906b0e271 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:26.662789 2113 ts_tablet_manager.cc:1905] T c49d1143d54b4c319739488906b0e271 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:26.669857 1484 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 36e10328c34242ebb4dbd8f294162696
W20250901 01:52:26.670070 1484 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 36e10328c34242ebb4dbd8f294162696: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:26.670378 1484 catalog_manager.cc:6652] ReplaceTablet: tablet 36e10328c34242ebb4dbd8f294162696 deleted and replaced by tablet 4abf9725be5d42f3893f07f0a9c7614c
I20250901 01:52:26.670779 898 tablet_service.cc:1515] Processing DeleteTablet for tablet 36e10328c34242ebb4dbd8f294162696 with delete_type TABLET_DATA_DELETED (replaced by tablet 4abf9725be5d42f3893f07f0a9c7614c) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:26.670800 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet 36e10328c34242ebb4dbd8f294162696 with delete_type TABLET_DATA_DELETED (replaced by tablet 4abf9725be5d42f3893f07f0a9c7614c) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:26.670832 2307 consensus_queue.cc:1035] T 64fb46b942ff4e4c8749986ecc3dfe65 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:26.670922 1491 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 3363fd67b1ad44849ecb7b4090b61a72
W20250901 01:52:26.671083 1491 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 3363fd67b1ad44849ecb7b4090b61a72: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:26.671682 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet 3363fd67b1ad44849ecb7b4090b61a72 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:26 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:26.672232 898 tablet_service.cc:1515] Processing DeleteTablet for tablet 3363fd67b1ad44849ecb7b4090b61a72 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:26 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:26.675958 2354 tablet.cc:1722] T b009cc95c819431ea9827a3e453b8da3 P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250901 01:52:26.683609 1483 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 500e3c33598b45a4afa983da7cb0845a
W20250901 01:52:26.683816 1483 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 500e3c33598b45a4afa983da7cb0845a: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:26.684638 898 tablet_service.cc:1515] Processing DeleteTablet for tablet 500e3c33598b45a4afa983da7cb0845a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:26 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:26.684643 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet 500e3c33598b45a4afa983da7cb0845a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:26 UTC) from {username='slave'} at 127.0.0.1:35946
W20250901 01:52:26.689132 992 leader_election.cc:336] T 23af154a6f1c4e2e9352fb17c9024f10 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:26.689396 1488 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: default.Table_4a412a59b6db421eb0187ffd02ecf541 [id=fd72e46c84634b6c9ea74970480ecc62] alter complete (version 4)
I20250901 01:52:26.690105 2253 raft_consensus.cc:491] T 5f52698fb56d472d9e91fb10cb10c69a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:26.690268 2253 raft_consensus.cc:513] T 5f52698fb56d472d9e91fb10cb10c69a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:26.690825 2253 leader_election.cc:290] T 5f52698fb56d472d9e91fb10cb10c69a P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:26.692298 858 leader_election.cc:336] T 5f52698fb56d472d9e91fb10cb10c69a P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:26.641486 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2c49fb8d72d2438a9c9416536edbb84c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:26.634052 1465 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 669d26c6551c4fc58283b67a501d9bf1 (table default.Table_e67df63f08fb415293f39cad748a0bd5 [id=4d54f43f814d48f79124714c187ea383], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 669d26c6551c4fc58283b67a501d9bf1 consensus state has no leader
W20250901 01:52:26.697049 992 consensus_peers.cc:489] T aa424ac6ef1441e6b1cc2867b1c48186 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250901 01:52:26.697340 992 leader_election.cc:343] T 23af154a6f1c4e2e9352fb17c9024f10 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 23af154a6f1c4e2e9352fb17c9024f10
I20250901 01:52:26.697309 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8440fbb2a0dd4d5a96bf41ed0918251e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.697424 992 leader_election.cc:304] T 23af154a6f1c4e2e9352fb17c9024f10 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:26.697610 2307 raft_consensus.cc:2747] T 23af154a6f1c4e2e9352fb17c9024f10 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:26.699846 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f52698fb56d472d9e91fb10cb10c69a" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:26.700169 2301 tablet.cc:1722] T b009cc95c819431ea9827a3e453b8da3 P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250901 01:52:26.701439 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c6002ad001a440e0a39ac7ea676ff62c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.701717 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3217bbc277ee4677a688b159ad13a0a7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.702256 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1ce32fe22891482098aadadc696381fc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.702556 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5f32d400c07549869ad950229c382a53"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.703981 1483 master_service.cc:948] ReplaceTablet: received request to replace tablet f9b293bf6a17453b9ea9892a6970d0b0 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:26.704285 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet afc002ae2486476b9208e1cf8215157c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:17 UTC) from {username='slave'} at 127.0.0.1:35946
W20250901 01:52:26.704334 859 leader_election.cc:343] T 5f52698fb56d472d9e91fb10cb10c69a P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:26.638244 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ad97adfb87364aca9f0360e45bdbcb4f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.704454 859 leader_election.cc:304] T 5f52698fb56d472d9e91fb10cb10c69a P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:26.704900 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet 75e7a90815e449c39f386901a504143c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:18 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:26.705066 2323 raft_consensus.cc:2747] T 5f52698fb56d472d9e91fb10cb10c69a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:26.705176 2307 consensus_queue.cc:1035] T aa424ac6ef1441e6b1cc2867b1c48186 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:26.707262 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet 4d27fb61943c45d4aca67689cc3a9ff3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:13 UTC) from {username='slave'} at 127.0.0.1:35946
W20250901 01:52:26.707623 1482 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet f9b293bf6a17453b9ea9892a6970d0b0 (table default.Table_a7fa05538bbd4766aa24d557677a692c [id=b67bfe518a094e0781ef931524b55990], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet f9b293bf6a17453b9ea9892a6970d0b0 consensus state has no leader
I20250901 01:52:26.708127 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet 7a607fac57994001b2304e5d81c9decc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:17 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:26.708801 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet e9b179dba2cc49428ba8090c962d6136 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:17 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:26.709343 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet 05630a72ddae4de0b890dece2feacc31 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:18 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:26.710340 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet 5e2472ee5d6e4d3bbf1f24027ba531ca with delete_type TABLET_DATA_DELETED (replaced by tablet 0813cc68d4b0488ba60a523e4479971b) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:26.711390 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet c58edaa76ebf47dfb1d9d5fab6de7dfa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:17 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:26.712463 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet e249474dcbf642189560c91ec3a99fdf with delete_type TABLET_DATA_DELETED (replaced by tablet 7f46e9ddadaf4ff986d8c991178daa0c) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:26.713871 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ae3063030e134ae7849971dc9a6c1fd6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.713940 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_89d2357f060e4e9ebc9c53d1939c5d99"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.714087 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4ce5af3f833f4d8986c4b87284eeddd8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.715140 1151 tablet_bootstrap.cc:492] T 081e2630e284498aa819f85bf8326315 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:26.715309 1151 ts_tablet_manager.cc:1397] T 081e2630e284498aa819f85bf8326315 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.101s user 0.005s sys 0.007s
I20250901 01:52:26.716032 1151 raft_consensus.cc:357] T 081e2630e284498aa819f85bf8326315 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:26.716256 1151 raft_consensus.cc:383] T 081e2630e284498aa819f85bf8326315 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:26.716378 1151 raft_consensus.cc:738] T 081e2630e284498aa819f85bf8326315 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:26.716656 1151 consensus_queue.cc:260] T 081e2630e284498aa819f85bf8326315 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:26.722119 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_319ea42367304a0aab311ab0152c9bc3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.722749 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_931f678f3d7f4b0aaaf61e94fd5bb0c4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.726433 2118 ts_tablet_manager.cc:1918] T 74b709cb5ec64006b2f12bead8217d80 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250901 01:52:26.729995 1482 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 2599f98abfdc4871b573edfc2ad97d22 (table default.Table_0e63d21acfcb4c028c9ea6406b6a04e4 [id=81323e6fbe424655b8e185f52657fadc], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 2599f98abfdc4871b573edfc2ad97d22 consensus state has no leader
I20250901 01:52:26.730057 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4a3726b4518141a39ac505749c373999"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.730250 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ae19ca1a737945fa8619b973b9ef39fd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.730808 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet eaa521c52a2f4408b59770bd2a05907c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:26.730912 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_89eb9658578a4feeaa4e500f35a3b044"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.731472 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet c754129977904f978f87fd89d7092e36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:24 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:26.732065 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet 17e8340dbd1a406cb782dad3fe88cc7e with delete_type TABLET_DATA_DELETED (replaced by tablet e2ed32b3292141278af28026528e96c2) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:26.692164 2113 ts_tablet_manager.cc:1918] T c49d1143d54b4c319739488906b0e271 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:26.732560 2113 log.cc:1199] T c49d1143d54b4c319739488906b0e271 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/c49d1143d54b4c319739488906b0e271
I20250901 01:52:26.732713 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet e25923b2daf9429683b2d44a75a56189 with delete_type TABLET_DATA_DELETED (replaced by tablet 2973b59bb1df41d18d9a5b6a29ae336f) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:26.733175 2113 ts_tablet_manager.cc:1939] T c49d1143d54b4c319739488906b0e271 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:26.733261 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet 6741c56499b44f648fb032c8cf50205b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:26.733606 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 61) took 1804 ms. Trace:
I20250901 01:52:26.733856 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet 46bfbdff3f904715a8c24907b20fcab0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:26.733741 2113 rpcz_store.cc:276] 0901 01:52:24.929236 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:24.929290 (+ 54us) service_pool.cc:225] Handling call
0901 01:52:26.658156 (+1728866us) ts_tablet_manager.cc:1145] Deleting tablet c49d1143d54b4c319739488906b0e271
0901 01:52:26.658168 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:26.689131 (+ 30963us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:26.692062 (+ 2931us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:26.733592 (+ 41530us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3515,"tablet-delete.queue_time_us":1728600}
I20250901 01:52:26.734395 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet c49d1143d54b4c319739488906b0e271 (table default.Table_583e53c2e86544e9be68eb27ab6068a3 [id=f376eaf65bb549429aa71b03fca81647]) successfully deleted
I20250901 01:52:26.734964 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 3c12d6a70ac24dce82e851782e6fe5aa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:24 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:26.734838 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet 0dbdcb76eeba4fefa56e7835c23c0316 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:17 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:26.735047 2113 tablet_replica.cc:331] T 0cb3262973934c8689b851adad60afe0 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:26.735231 2113 raft_consensus.cc:2241] T 0cb3262973934c8689b851adad60afe0 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:26.735608 2113 raft_consensus.cc:2270] T 0cb3262973934c8689b851adad60afe0 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:26.736011 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_0561e17698114cac85ec36332272be1f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.738616 2113 ts_tablet_manager.cc:1905] T 0cb3262973934c8689b851adad60afe0 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:26.741302 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_85d5c01faa794f69a961d34a606005f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.741427 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_91bd87cf72b640daaf4f9fb131ca2763"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.742476 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7419ab424c254c9bafda422447da4087"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.743196 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_865c8478b2c04a198ea560c48812e926"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.744679 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_53d5559aee764ba7b07ef1873c4a9885"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.744941 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f57a4631b1204c5cbce19bcca29c3c87"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.745935 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9f6dd885f17843b1965908c6b2013d81"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.747016 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7bd44fa3b3eb4f71972a2d46e2348400"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.748189 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet 7f64ebd080944190be2e64d02d70a8a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:15 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:26.748991 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet 25ce0fe8c9544f828694184d19d13df9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:16 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:26.750285 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_fa7b4aea5c91421181dcb092e324f59a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.751547 1151 ts_tablet_manager.cc:1428] T 081e2630e284498aa819f85bf8326315 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.036s user 0.001s sys 0.000s
I20250901 01:52:26.752745 2354 tablet.cc:1722] T 7442c6ff699141e1aff208ebdc9d192a P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:26.752063 2301 tablet.cc:1722] T 7442c6ff699141e1aff208ebdc9d192a P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:26.758299 1151 tablet_bootstrap.cc:492] T 2599f98abfdc4871b573edfc2ad97d22 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:26.758947 2118 log.cc:1199] T 74b709cb5ec64006b2f12bead8217d80 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/74b709cb5ec64006b2f12bead8217d80
I20250901 01:52:26.761314 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c094777871d145178a680f69e17f53ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.762698 2118 ts_tablet_manager.cc:1939] T 74b709cb5ec64006b2f12bead8217d80 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:26.764467 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 6cf09e8f299548dba2cce16d7b565bad with delete_type TABLET_DATA_DELETED (replaced by tablet 0e4009963cfe42d6ba164fc828897f09) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:26.764961 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet af2c4bcbb00741f789edee109252d8cb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:18 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:26.765175 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet b7b802f976f3471faaaecc964ec87e48 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:16 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:26.765636 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 764bfeb2a7d14c40815b5ec1b07bdd62 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:23 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:26.765977 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 4855448edb4b40c184838aa7f81c112b with delete_type TABLET_DATA_DELETED (replaced by tablet 5e2472ee5d6e4d3bbf1f24027ba531ca) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:26.766427 1151 tablet_bootstrap.cc:654] T 2599f98abfdc4871b573edfc2ad97d22 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:26.766731 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet c49d1143d54b4c319739488906b0e271 with delete_type TABLET_DATA_DELETED (replaced by tablet 69f61ec283484f9d8b3eebcea27f741b) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:26.767048 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 438ae684af234c868a419fbab43b0b98 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:24 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:26.767356 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a3e8abbcb0084e8c951c9659b94287e3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.767371 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 44) took 1593 ms. Trace:
I20250901 01:52:26.769045 1150 tablet_bootstrap.cc:492] T eb19e3a424324a24824f6caaf5848952 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:26.769197 1150 ts_tablet_manager.cc:1397] T eb19e3a424324a24824f6caaf5848952 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.125s user 0.006s sys 0.007s
I20250901 01:52:26.770557 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 246b33338de647689d7404c70676e76f with delete_type TABLET_DATA_DELETED (replaced by tablet ae4844183a3c4a04a775e6fd878391a3) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:26.771284 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 617cc33d0d23439cb5f1e0c31bf11281 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:24 UTC) from {username='slave'} at 127.0.0.1:35946
W20250901 01:52:26.771728 992 consensus_peers.cc:489] T 64fb46b942ff4e4c8749986ecc3dfe65 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:26.771760 1150 raft_consensus.cc:357] T eb19e3a424324a24824f6caaf5848952 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:26.772648 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 88ceba7f48b04d7885cc0164668700a5 with delete_type TABLET_DATA_DELETED (replaced by tablet 9074543dea9949e7a085717b40913348) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:26.772871 1150 raft_consensus.cc:383] T eb19e3a424324a24824f6caaf5848952 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:26.773089 1150 raft_consensus.cc:738] T eb19e3a424324a24824f6caaf5848952 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
W20250901 01:52:26.773403 1482 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet da5f69c5b0ca4257ad2ff7918ee2af2e (table default.Table_ecb9e8f638a043cc87c38afbebce250f [id=61f1e8a842844ed899e5913c76f362ff], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet da5f69c5b0ca4257ad2ff7918ee2af2e consensus state has no leader
I20250901 01:52:26.773677 1150 consensus_queue.cc:260] T eb19e3a424324a24824f6caaf5848952 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:26.774657 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_962cda1d4c1941319aea6af72250e9ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.775318 1150 ts_tablet_manager.cc:1428] T eb19e3a424324a24824f6caaf5848952 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250901 01:52:26.768215 2118 rpcz_store.cc:276] 0901 01:52:25.174338 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.174637 (+ 299us) service_pool.cc:225] Handling call
0901 01:52:26.554996 (+1380359us) ts_tablet_manager.cc:1145] Deleting tablet 74b709cb5ec64006b2f12bead8217d80
0901 01:52:26.555009 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:26.669682 (+114673us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:26.726323 (+ 56641us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:26.767350 (+ 41027us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1380146}
I20250901 01:52:26.776459 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 74b709cb5ec64006b2f12bead8217d80 (table default.Table_95884295301d476fb3e03054727f9cf1 [id=1aa18a2513db4e99a72572846527663c]) successfully deleted
I20250901 01:52:26.776989 2118 tablet_replica.cc:331] T 8dd587304e6342edbe96c9897b40e5c9 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:26.776115 1150 tablet_bootstrap.cc:492] T 6ba88af177874544b764cc32420e7ac9 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:26.778636 2118 raft_consensus.cc:2241] T 8dd587304e6342edbe96c9897b40e5c9 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:26.779279 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7c6fd07797534ab39f8cf01f3338dba4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.779847 2118 raft_consensus.cc:2270] T 8dd587304e6342edbe96c9897b40e5c9 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:26.783668 1486 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_0b4ac0a1caa5472d909448e41318e679" } new_table_name: "default.Table_42422c44ca254d8ca989134c2433391f" modify_external_catalogs: true
I20250901 01:52:26.784842 1993 consensus_queue.cc:1035] T 20f7bfab18134e079e91c52c89468cc3 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:26.785066 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_721882b969ae40db9a95673b0d4abdeb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.784934 2118 ts_tablet_manager.cc:1905] T 8dd587304e6342edbe96c9897b40e5c9 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:26.786312 1483 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_ac189f6a9fcf457e9fa0d77badc81594" } new_table_name: "default.Table_9896b980158948339a0b91f5882ed56b" modify_external_catalogs: true
I20250901 01:52:26.787982 1489 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_4e2ea69d21394b9db0447f7ec8766795" } modify_external_catalogs: true
I20250901 01:52:26.788250 1489 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_4e2ea69d21394b9db0447f7ec8766795" } modify_external_catalogs: true
I20250901 01:52:26.790256 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 34042ea5c1b645969c1de41fe587c343 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:23 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:26.792852 1492 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_ae42ba019ee14c69b1a16d1177f3642c" } new_table_name: "default.Table_927955c1048f4e9c84b5b3a113a473f7" modify_external_catalogs: true
I20250901 01:52:26.795598 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f56cbe52dafc4600bb56b05cc4ae3f56"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.796368 2123 raft_consensus.cc:491] T 2240302d9178445d8700e35f5ea274f7 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:26.796522 2123 raft_consensus.cc:513] T 2240302d9178445d8700e35f5ea274f7 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:26.797103 2123 leader_election.cc:290] T 2240302d9178445d8700e35f5ea274f7 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:26.798367 859 leader_election.cc:343] T 2240302d9178445d8700e35f5ea274f7 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Not found: Tablet not found: 2240302d9178445d8700e35f5ea274f7
W20250901 01:52:26.799109 858 leader_election.cc:336] T 2240302d9178445d8700e35f5ea274f7 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:26.799217 858 leader_election.cc:304] T 2240302d9178445d8700e35f5ea274f7 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:26.799755 2123 raft_consensus.cc:2747] T 2240302d9178445d8700e35f5ea274f7 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250901 01:52:26.801914 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d5dfed3a8c9740b58622d5fa49b3b7ae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.802795 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 756cfe84a517459ca015aecf5cd81571 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:24 UTC) from {username='slave'} at 127.0.0.1:35946
W20250901 01:52:26.803897 1492 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 14b466b2a5a249aa841f1b8b06301edc consensus state has no leader
I20250901 01:52:26.804669 1150 tablet_bootstrap.cc:654] T 6ba88af177874544b764cc32420e7ac9 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:26.805747 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_81498bcf47c7420cb1738aa04109621f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.806301 1484 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_820e1c5a64cd4128b7ae1b70bd951964" } new_table_name: "default.Table_e0902dfbdd984f85b3fdac8d283731f7" modify_external_catalogs: true
I20250901 01:52:26.811142 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a4301e0e6d4941a4b9e5c6aef9030b73"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:26.811489 1482 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.3:51295 (request call id 180) took 1741 ms (client timeout 1000 ms). Trace:
W20250901 01:52:26.811650 1482 rpcz_store.cc:269] 0901 01:52:25.070036 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.097791 (+ 27755us) service_pool.cc:225] Handling call
0901 01:52:26.811447 (+1713656us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0901 01:52:25.787418 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:25.787520 (+ 102us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:25.787526 (+ 6us) write_op.cc:435] Acquired schema lock
0901 01:52:25.787528 (+ 2us) tablet.cc:660] Decoding operations
0901 01:52:25.787627 (+ 99us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:25.787640 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:25.787644 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0901 01:52:25.787773 (+ 129us) tablet.cc:699] Row locks acquired
0901 01:52:25.787775 (+ 2us) write_op.cc:260] PREPARE: finished
0901 01:52:25.787814 (+ 39us) write_op.cc:270] Start()
0901 01:52:25.787847 (+ 33us) write_op.cc:276] Timestamp: P: 1756691545787767 usec, L: 0
0901 01:52:25.787852 (+ 5us) op_driver.cc:348] REPLICATION: starting
0901 01:52:25.788023 (+ 171us) log.cc:844] Serialized 901 byte log entry
0901 01:52:25.796486 (+ 8463us) op_driver.cc:464] REPLICATION: finished
0901 01:52:25.796613 (+ 127us) write_op.cc:301] APPLY: starting
0901 01:52:25.796629 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:25.796677 (+ 48us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:25.796712 (+ 35us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:25.796784 (+ 72us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:25.796818 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0901 01:52:25.796825 (+ 7us) write_op.cc:312] APPLY: finished
0901 01:52:25.796906 (+ 81us) log.cc:844] Serialized 29 byte log entry
0901 01:52:25.796968 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:25.796989 (+ 21us) write_op.cc:454] Released schema lock
0901 01:52:25.796996 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0901 01:52:25.797063 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:25.797067 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":18016,"reported_tablets":243,"child_traces":[["op",{"apply.queue_time_us":81,"apply.run_cpu_time_us":440,"apply.run_wall_time_us":439,"num_ops":2,"prepare.queue_time_us":72,"prepare.run_cpu_time_us":727,"prepare.run_wall_time_us":727,"raft.queue_time_us":62,"raft.run_cpu_time_us":205,"raft.run_wall_time_us":206,"replication_time_us":8604}]]}
I20250901 01:52:26.812582 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e4900d50122145e6b04a2a2aa24593c8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:26.814208 1484 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d000ba05e12843b4925e622ab55471c0 consensus state has no leader
W20250901 01:52:26.814687 1483 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 44af4764d4f64e44aec84ccb84be6f5f consensus state has no leader
I20250901 01:52:26.816869 2113 ts_tablet_manager.cc:1918] T 0cb3262973934c8689b851adad60afe0 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:26.817140 2113 log.cc:1199] T 0cb3262973934c8689b851adad60afe0 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/0cb3262973934c8689b851adad60afe0
I20250901 01:52:26.817595 1150 tablet_bootstrap.cc:492] T 6ba88af177874544b764cc32420e7ac9 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:26.817988 2113 ts_tablet_manager.cc:1939] T 0cb3262973934c8689b851adad60afe0 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:26.819649 1150 ts_tablet_manager.cc:1397] T 6ba88af177874544b764cc32420e7ac9 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.044s user 0.003s sys 0.001s
I20250901 01:52:26.823101 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 62) took 1892 ms. Trace:
I20250901 01:52:26.823536 1993 raft_consensus.cc:491] T a8483e7212d94daba7e5401256c33aef P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:26.823644 1993 raft_consensus.cc:513] T a8483e7212d94daba7e5401256c33aef P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:26.823755 1150 raft_consensus.cc:357] T 6ba88af177874544b764cc32420e7ac9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:26.823889 1150 raft_consensus.cc:383] T 6ba88af177874544b764cc32420e7ac9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:26.823933 1150 raft_consensus.cc:738] T 6ba88af177874544b764cc32420e7ac9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:26.824105 1150 consensus_queue.cc:260] T 6ba88af177874544b764cc32420e7ac9 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:26.824246 1993 leader_election.cc:290] T a8483e7212d94daba7e5401256c33aef P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:26.825852 911 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8483e7212d94daba7e5401256c33aef" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:26.826068 911 raft_consensus.cc:2466] T a8483e7212d94daba7e5401256c33aef P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:26.828819 992 leader_election.cc:304] T a8483e7212d94daba7e5401256c33aef P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:26.829123 1993 raft_consensus.cc:2802] T a8483e7212d94daba7e5401256c33aef P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:26.829218 1993 raft_consensus.cc:491] T a8483e7212d94daba7e5401256c33aef P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:26.829293 1993 raft_consensus.cc:3058] T a8483e7212d94daba7e5401256c33aef P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:26.829540 1150 ts_tablet_manager.cc:1428] T 6ba88af177874544b764cc32420e7ac9 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20250901 01:52:26.823315 2113 rpcz_store.cc:276] 0901 01:52:24.930042 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:24.930092 (+ 50us) service_pool.cc:225] Handling call
0901 01:52:26.735004 (+1804912us) ts_tablet_manager.cc:1145] Deleting tablet 0cb3262973934c8689b851adad60afe0
0901 01:52:26.735014 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:26.757889 (+ 22875us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:26.801310 (+ 43421us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:26.822885 (+ 21575us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1804515}
I20250901 01:52:26.829813 1150 tablet_bootstrap.cc:492] T becbaae4fd7e44faa679e9479465a7c4 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:26.830061 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 0cb3262973934c8689b851adad60afe0 (table default.Table_0fc5eac5c02943af9b6801ac9cee55a3 [id=28235bf690f747b398aea70f53caa578]) successfully deleted
I20250901 01:52:26.830472 2113 tablet_replica.cc:331] T b6daeff0070a4db08e90020752d21c29 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:26.830816 2113 raft_consensus.cc:2241] T b6daeff0070a4db08e90020752d21c29 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:26.831283 2113 raft_consensus.cc:2270] T b6daeff0070a4db08e90020752d21c29 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:26.831431 1150 tablet_bootstrap.cc:654] T becbaae4fd7e44faa679e9479465a7c4 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:26.831841 1993 raft_consensus.cc:513] T a8483e7212d94daba7e5401256c33aef P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:26.832382 2113 ts_tablet_manager.cc:1905] T b6daeff0070a4db08e90020752d21c29 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:26.832880 1993 leader_election.cc:290] T a8483e7212d94daba7e5401256c33aef P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:26.833262 992 leader_election.cc:336] T a8483e7212d94daba7e5401256c33aef P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:26.833549 992 consensus_peers.cc:489] T 616ff34e8d314e0aa0e11ce60be11789 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:26.834336 911 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8483e7212d94daba7e5401256c33aef" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:26.834492 911 raft_consensus.cc:3058] T a8483e7212d94daba7e5401256c33aef P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:26.835320 2118 ts_tablet_manager.cc:1918] T 8dd587304e6342edbe96c9897b40e5c9 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:26.835424 2118 log.cc:1199] T 8dd587304e6342edbe96c9897b40e5c9 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/8dd587304e6342edbe96c9897b40e5c9
I20250901 01:52:26.835786 2118 ts_tablet_manager.cc:1939] T 8dd587304e6342edbe96c9897b40e5c9 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:26.836055 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 45) took 1661 ms. Trace:
I20250901 01:52:26.836124 2118 rpcz_store.cc:276] 0901 01:52:25.175007 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.192676 (+ 17669us) service_pool.cc:225] Handling call
0901 01:52:26.776917 (+1584241us) ts_tablet_manager.cc:1145] Deleting tablet 8dd587304e6342edbe96c9897b40e5c9
0901 01:52:26.776925 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:26.820979 (+ 44054us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:26.835229 (+ 14250us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:26.836040 (+ 811us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1584011}
I20250901 01:52:26.836586 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 8dd587304e6342edbe96c9897b40e5c9 (table default.Table_ee242c34410e4c6887405d04a4b26124 [id=c9a2f96383664a4e8524c286cd9e4149]) successfully deleted
I20250901 01:52:26.836930 911 raft_consensus.cc:2466] T a8483e7212d94daba7e5401256c33aef P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:26.837707 992 leader_election.cc:304] T a8483e7212d94daba7e5401256c33aef P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
W20250901 01:52:26.838266 992 leader_election.cc:336] T a8483e7212d94daba7e5401256c33aef P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:26.838606 1993 raft_consensus.cc:2802] T a8483e7212d94daba7e5401256c33aef P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:26.838857 1993 raft_consensus.cc:695] T a8483e7212d94daba7e5401256c33aef P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:26.839109 1993 consensus_queue.cc:237] T a8483e7212d94daba7e5401256c33aef P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:26.840373 1482 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_b5d681f51fc84395bbabe1e5b4cfb358" } modify_external_catalogs: true
I20250901 01:52:26.840276 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_65389206e1534c38b27fd81fc6ce4ffc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.840373 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ca1bceb4123b454a928b43e5f9203da6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.840629 1482 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_b5d681f51fc84395bbabe1e5b4cfb358" } modify_external_catalogs: true
I20250901 01:52:26.841365 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3034892f99ff4a1ba379d69ff0e04875"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.841405 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c1647173fab34674be45953e6f99341c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.842089 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d54d1f7d89304ce291decdf1ed28c18c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.842796 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5c9684fd770a4ba196074ab90baa150b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.843423 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b17b4543a54044cc88dc1b9f05e7440a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.844074 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_52c10d5021a942c28a368b795376faa1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.844727 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8d26c19fce944e4eb8e498f26a17e120"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.845422 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_52da13573f354a2590bf3783a4190cd6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.848680 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_db1eeee778f94cc78585f00b8cab2765"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.849169 1482 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 42c1156e8dd84c6fa1396b46de13eb6f
W20250901 01:52:26.849367 1482 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 42c1156e8dd84c6fa1396b46de13eb6f: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:26.850049 898 tablet_service.cc:1515] Processing DeleteTablet for tablet 42c1156e8dd84c6fa1396b46de13eb6f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:26 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:26.850183 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 42c1156e8dd84c6fa1396b46de13eb6f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:26 UTC) from {username='slave'} at 127.0.0.1:35946
W20250901 01:52:26.851346 992 consensus_peers.cc:489] T 061b760d8e6647c6ab0b5d538840a230 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250901 01:52:26.852069 1484 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_dd01ae355f73448cbca96f086b04a2fe" } new_table_name: "default.Table_b2083370690443c6aa50c453bd29cb9a" modify_external_catalogs: true
I20250901 01:52:26.852602 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_eba6ac48ca244c84bd3803014ceb6f84"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.853859 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_357fed3e0690421d871f8b47ca2f0ff5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.856221 2118 tablet_replica.cc:331] T 40d74d7b5a3b4d1dae09eebd9e07a0c4 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:26.857796 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_aee6ea6a14204f919f91de03373ea78e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.858345 2118 raft_consensus.cc:2241] T 40d74d7b5a3b4d1dae09eebd9e07a0c4 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:26.859264 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_26e00904c2384bfdb6263994483887d0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:26.860056 1484 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7954393c8c1c4457ab8bb723c03a9c8b consensus state has no leader
I20250901 01:52:26.857071 1150 tablet_bootstrap.cc:492] T becbaae4fd7e44faa679e9479465a7c4 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:26.860303 1150 ts_tablet_manager.cc:1397] T becbaae4fd7e44faa679e9479465a7c4 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.031s user 0.001s sys 0.002s
I20250901 01:52:26.860993 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4f95917f952b464aadc0dcad53383617"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.860562 2118 raft_consensus.cc:2270] T 40d74d7b5a3b4d1dae09eebd9e07a0c4 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:26.862663 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_fa6d0c146b914087b0c49ce21a8b9827"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.863581 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2a75d984ea3f4160a6cfb4c3df670083"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.864349 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_188f06ec79524ac7aa43f7719c8ed93a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.866663 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_eb4c3e74ea03471d9f20271b504da66d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.862864 2118 ts_tablet_manager.cc:1905] T 40d74d7b5a3b4d1dae09eebd9e07a0c4 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:26.867342 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7713f580681c4348a888bf78a1d6be73"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.867915 1489 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 7442c6ff699141e1aff208ebdc9d192a
I20250901 01:52:26.862074 1150 raft_consensus.cc:357] T becbaae4fd7e44faa679e9479465a7c4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:26.868652 1489 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 7442c6ff699141e1aff208ebdc9d192a: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:26.868813 1150 raft_consensus.cc:383] T becbaae4fd7e44faa679e9479465a7c4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:26.869628 1150 raft_consensus.cc:738] T becbaae4fd7e44faa679e9479465a7c4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:26.869941 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 7442c6ff699141e1aff208ebdc9d192a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:26 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:26.870184 1150 consensus_queue.cc:260] T becbaae4fd7e44faa679e9479465a7c4 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:26.873565 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f840822e2e364a33bb5a35ca42e102da"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.885960 1150 ts_tablet_manager.cc:1428] T becbaae4fd7e44faa679e9479465a7c4 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.025s user 0.001s sys 0.001s
I20250901 01:52:26.886540 1150 tablet_bootstrap.cc:492] T b164b4a67a9143ceaff69b0ce49d63fe P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:26.891088 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9278d00bf81d4e8e85af44ba0c23dc03"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.872740 898 tablet_service.cc:1515] Processing DeleteTablet for tablet 7442c6ff699141e1aff208ebdc9d192a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:26 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:26.892112 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ee82b4859f4a44cba8f96a57d161d910"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.892277 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3238c46ba13545d79d7b39dd2798172b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.892115 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b554d7153d6445568e4cb7348ba35257"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.903822 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_09d61855ab8b4ffd93f19bd9d85c9196"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.905896 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a7e6417781584210916de3aa877370dd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.907034 2118 ts_tablet_manager.cc:1918] T 40d74d7b5a3b4d1dae09eebd9e07a0c4 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:26.908262 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_68c22d3b48fc41c4b94b761dd38f8823"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:26.913022 992 consensus_peers.cc:489] T 5adf6b492640417ca071c94702b48cb9 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250901 01:52:26.913635 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_51c92653c1df4677a6c53010e48e2bb1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.914042 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e20e0386ae9f467ebc836093119c9d5c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.916292 1489 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_fbba493de02249499d26467bf15c3d5f" } modify_external_catalogs: true
I20250901 01:52:26.916535 1489 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_fbba493de02249499d26467bf15c3d5f" } modify_external_catalogs: true
I20250901 01:52:26.917004 2118 log.cc:1199] T 40d74d7b5a3b4d1dae09eebd9e07a0c4 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/40d74d7b5a3b4d1dae09eebd9e07a0c4
W20250901 01:52:26.917032 1486 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d0c3c363dda444519a17dca63ad9caa7 consensus state has no leader
I20250901 01:52:26.918141 2118 ts_tablet_manager.cc:1939] T 40d74d7b5a3b4d1dae09eebd9e07a0c4 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:26.918527 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 46) took 1723 ms. Trace:
I20250901 01:52:26.918607 2118 rpcz_store.cc:276] 0901 01:52:25.195384 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.196505 (+ 1121us) service_pool.cc:225] Handling call
0901 01:52:26.856170 (+1659665us) ts_tablet_manager.cc:1145] Deleting tablet 40d74d7b5a3b4d1dae09eebd9e07a0c4
0901 01:52:26.856178 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:26.888652 (+ 32474us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:26.906921 (+ 18269us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:26.918501 (+ 11580us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1659442}
I20250901 01:52:26.918915 2118 tablet_replica.cc:331] T f3962b471bdb49aea6e9fa95f1f9dfe2 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:26.919092 2118 raft_consensus.cc:2241] T f3962b471bdb49aea6e9fa95f1f9dfe2 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:26.919234 2118 raft_consensus.cc:2270] T f3962b471bdb49aea6e9fa95f1f9dfe2 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:26.919746 2118 ts_tablet_manager.cc:1905] T f3962b471bdb49aea6e9fa95f1f9dfe2 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:26.921036 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 40d74d7b5a3b4d1dae09eebd9e07a0c4 (table default.Table_749fcd4bf97d439c9fd49b138a52a3fd [id=24fffc7ebf5a47bdb37211910301acf7]) successfully deleted
E20250901 01:52:26.924564 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae4844183a3c4a04a775e6fd878391a3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_aee74198b51d4f089538c4f2bf3ce4dc'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:26.926733 1696 catalog_manager.cc:6205] Error selecting replicas for tablet be8dfbc466564842a2fdb4953eee939b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_75ba3ba70ca24a94920e7bce96190ff2'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:26.926882 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 96824e68bbf24b8287dbb683038ffaba, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_3c7c628f35fb4d02bb79a34995f12938'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:26.927151 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 7317eb7848ee44698038a4b404d4bf21, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b0e916c76a7a4766aa0887cc8691488f'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:26.930867 2113 ts_tablet_manager.cc:1918] T b6daeff0070a4db08e90020752d21c29 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:26.937321 2113 log.cc:1199] T b6daeff0070a4db08e90020752d21c29 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/b6daeff0070a4db08e90020752d21c29
I20250901 01:52:26.944716 969 heartbeater.cc:499] Master 127.0.104.60:44231 was elected leader, sending a full tablet report...
I20250901 01:52:26.947872 2118 ts_tablet_manager.cc:1918] T f3962b471bdb49aea6e9fa95f1f9dfe2 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:26.950567 2113 ts_tablet_manager.cc:1939] T b6daeff0070a4db08e90020752d21c29 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:26.951838 2118 log.cc:1199] T f3962b471bdb49aea6e9fa95f1f9dfe2 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/f3962b471bdb49aea6e9fa95f1f9dfe2
I20250901 01:52:26.955269 1151 tablet_bootstrap.cc:492] T 2599f98abfdc4871b573edfc2ad97d22 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:26.956504 2118 ts_tablet_manager.cc:1939] T f3962b471bdb49aea6e9fa95f1f9dfe2 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:26.956912 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 47) took 1760 ms. Trace:
I20250901 01:52:26.957041 2118 rpcz_store.cc:276] 0901 01:52:25.196901 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.197216 (+ 315us) service_pool.cc:225] Handling call
0901 01:52:26.918868 (+1721652us) ts_tablet_manager.cc:1145] Deleting tablet f3962b471bdb49aea6e9fa95f1f9dfe2
0901 01:52:26.918877 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:26.928570 (+ 9693us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:26.943944 (+ 15374us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:26.956894 (+ 12950us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1721473}
I20250901 01:52:26.957909 1151 ts_tablet_manager.cc:1397] T 2599f98abfdc4871b573edfc2ad97d22 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.200s user 0.003s sys 0.000s
I20250901 01:52:26.960033 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 48) took 1759 ms. Trace:
I20250901 01:52:26.960161 2118 rpcz_store.cc:276] 0901 01:52:25.200754 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.200930 (+ 176us) service_pool.cc:225] Handling call
0901 01:52:26.957533 (+1756603us) ts_tablet_manager.cc:1145] Deleting tablet 75345ea2a3dc40fc8c9f8090d8839d2d
0901 01:52:26.957542 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:26.960014 (+ 2472us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1756379}
I20250901 01:52:26.959860 1151 raft_consensus.cc:357] T 2599f98abfdc4871b573edfc2ad97d22 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:26.962227 1151 raft_consensus.cc:383] T 2599f98abfdc4871b573edfc2ad97d22 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:26.962410 1151 raft_consensus.cc:738] T 2599f98abfdc4871b573edfc2ad97d22 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:26.963011 1151 consensus_queue.cc:260] T 2599f98abfdc4871b573edfc2ad97d22 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:26.931188 1486 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_0f91070385df441b96af4d5406a94724" } modify_external_catalogs: true
I20250901 01:52:26.966025 1150 tablet_bootstrap.cc:654] T b164b4a67a9143ceaff69b0ce49d63fe P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:26.966180 858 consensus_peers.cc:489] T 24ddc13d1bb04db4a7850acbff091577 P 00c282f779c1477d920f24dc0dc16628 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250901 01:52:26.964627 1486 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_0f91070385df441b96af4d5406a94724" } modify_external_catalogs: true
I20250901 01:52:26.938035 1482 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_b989608c88204be2b6f706a9426c47bb" } modify_external_catalogs: true
I20250901 01:52:26.966840 1482 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_b989608c88204be2b6f706a9426c47bb" } modify_external_catalogs: true
I20250901 01:52:26.937474 1489 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 061b760d8e6647c6ab0b5d538840a230
I20250901 01:52:26.957538 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet f3962b471bdb49aea6e9fa95f1f9dfe2 (table default.Table_e2ea7aa90231448888c6cfead7e3d9c1 [id=c264ed0bf1d84bc9bd5c2bb0d1e8a45b]) successfully deleted
W20250901 01:52:26.967439 1472 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 75345ea2a3dc40fc8c9f8090d8839d2d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 75345ea2a3dc40fc8c9f8090d8839d2d already in progress: creating tablet
W20250901 01:52:26.967762 1489 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 061b760d8e6647c6ab0b5d538840a230: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:26.968912 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 061b760d8e6647c6ab0b5d538840a230 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:26 UTC) from {username='slave'} at 127.0.0.1:35946
E20250901 01:52:26.969204 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9745582cebfe47b58c21d0191ab4cb9b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2f09538b6fbe401f909105decb56e1cb'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:26.969344 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 4abf9725be5d42f3893f07f0a9c7614c, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_8445fcd774874cadaf4acc3581eee417'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:26.969430 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 8a13346dcb3d4b6ab35c261829d50ae5, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_1bdd9041d74045cab6f23de11bf4dd09'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:26.969530 1696 catalog_manager.cc:6205] Error selecting replicas for tablet e2ed32b3292141278af28026528e96c2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_50ae3908a7944f70949c54be8fbf1564'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:26.969658 1696 catalog_manager.cc:6205] Error selecting replicas for tablet a2ca69fabadd4ad69e1670cae7a0c503, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_32c4716e3aa24a87a6b1472984b255c4'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:26.969775 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f8f923ddcd2740bcb7e521f4f57c4034, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_5e65c5f068b84995a2d7a76fa65dda0f'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:26.969875 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 0813cc68d4b0488ba60a523e4479971b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d1302673d68f424ebb54b220911ce385'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:26.969997 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 787be682c0e44248a80d7037fb61dc6c, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7c12d66048a6470e8c227dc1d9f45fa4'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:26.970115 1696 catalog_manager.cc:6205] Error selecting replicas for tablet cc3b8736b9094d7e892c80da4d5b20f6, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_93afd37289414655980b22934695bd55'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:26.970268 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae9a4ad4fdfb4771ba23a91f7007447e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_5e84ae9a9c2a4084a1f9c5ffb4594de2'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:26.970387 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9074543dea9949e7a085717b40913348, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_53e30205c5fd488eb3b48c685a159d04'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:26.968683 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 63) took 2037 ms. Trace:
E20250901 01:52:26.971031 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 18498c9cb09141d08c1c5c0854b63ebd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77f4810bc83c4d948531da2a45a3792e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:26.971201 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 81319e373c9a4475abceeeedb507aabc, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_22d8d38d0b624912a0d8968fb764420e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:26.971344 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 6ce9521b925b4f97bb4a810ce77e147e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_9e04f976a93f4d30baccf90fa3622dfe'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:26.971315 2123 raft_consensus.cc:491] T aec951f2cc84499b9162b9fa24d5440e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:26.970741 2113 rpcz_store.cc:276] 0901 01:52:24.931548 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:24.931592 (+ 44us) service_pool.cc:225] Handling call
0901 01:52:26.830364 (+1898772us) ts_tablet_manager.cc:1145] Deleting tablet b6daeff0070a4db08e90020752d21c29
0901 01:52:26.830401 (+ 37us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:26.838223 (+ 7822us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:26.930071 (+ 91848us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:26.968634 (+ 38563us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1898565}
I20250901 01:52:26.973774 2306 raft_consensus.cc:491] T eaa521c52a2f4408b59770bd2a05907c P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:26.973909 2306 raft_consensus.cc:513] T eaa521c52a2f4408b59770bd2a05907c P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:26.974431 2306 leader_election.cc:290] T eaa521c52a2f4408b59770bd2a05907c P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:26.975036 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet b6daeff0070a4db08e90020752d21c29 (table default.Table_8022c1172d8a4df8bc1295d193bb6542 [id=f278d7b5596944238ef56dfb5de097dc]) successfully deleted
I20250901 01:52:26.976660 2113 tablet_replica.cc:331] T 8dd587304e6342edbe96c9897b40e5c9 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:26.977671 2306 consensus_queue.cc:1035] T 616ff34e8d314e0aa0e11ce60be11789 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:26.979389 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_098262e5f24d43979e195620c9f15c7c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.979951 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b2880636d86e4ae1b350a8ce5769f878"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:26.979492 992 consensus_peers.cc:489] T 20f7bfab18134e079e91c52c89468cc3 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:26.983188 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_160f092d104746d5b77af2a061b6ab70"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:26.983912 1151 ts_tablet_manager.cc:1428] T 2599f98abfdc4871b573edfc2ad97d22 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.026s user 0.002s sys 0.000s
I20250901 01:52:26.984239 1151 tablet_bootstrap.cc:492] T 8ed6616ce54548ba83ba299c4ecf3c38 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:26.985302 2113 raft_consensus.cc:2241] T 8dd587304e6342edbe96c9897b40e5c9 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:26.971457 2123 raft_consensus.cc:513] T aec951f2cc84499b9162b9fa24d5440e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:26.990504 2123 leader_election.cc:290] T aec951f2cc84499b9162b9fa24d5440e P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:26.990677 898 tablet_service.cc:1515] Processing DeleteTablet for tablet 061b760d8e6647c6ab0b5d538840a230 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:26 UTC) from {username='slave'} at 127.0.0.1:34472
W20250901 01:52:26.997459 858 leader_election.cc:336] T aec951f2cc84499b9162b9fa24d5440e P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:26.997903 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 49) took 1796 ms. Trace:
I20250901 01:52:26.998049 2118 rpcz_store.cc:276] 0901 01:52:25.201273 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.201308 (+ 35us) service_pool.cc:225] Handling call
0901 01:52:26.960790 (+1759482us) ts_tablet_manager.cc:1145] Deleting tablet 088ef4861dc14f81ae470c3aaee5b4e1
0901 01:52:26.997739 (+ 36949us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:26.997882 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1759332}
W20250901 01:52:26.998634 1472 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 088ef4861dc14f81ae470c3aaee5b4e1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 088ef4861dc14f81ae470c3aaee5b4e1 already in progress: creating tablet
I20250901 01:52:26.999014 2118 tablet_replica.cc:331] T 831e42de95854541a49ae6b3e039d3e9 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:26.999241 2118 raft_consensus.cc:2241] T 831e42de95854541a49ae6b3e039d3e9 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:26.999451 2118 raft_consensus.cc:2270] T 831e42de95854541a49ae6b3e039d3e9 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:27.000160 2118 ts_tablet_manager.cc:1905] T 831e42de95854541a49ae6b3e039d3e9 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:27.006220 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_793396fdbce34c98a01f6dfd91facb9e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:27.006569 922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eaa521c52a2f4408b59770bd2a05907c" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:27.006762 922 raft_consensus.cc:2466] T eaa521c52a2f4408b59770bd2a05907c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:27.007205 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d6f44660aea74b4394fd08023ff85f57"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:27.009949 2118 ts_tablet_manager.cc:1918] T 831e42de95854541a49ae6b3e039d3e9 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:27.010339 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e17ee41cf6b04341ac7faad64d4d1dfc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:27.014755 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_41a8da045b1a424ca8b5d928ce0fd773"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:27.017901 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_79319d5f590a4137b712fa7cb3b2fa98"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:27.020375 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_27c2275794d34f14bc58b3d227c43085"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:27.021561 2113 raft_consensus.cc:2270] T 8dd587304e6342edbe96c9897b40e5c9 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:27.021761 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7a5d303a32ff45bab77dbad2ace902a2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:27.022958 1492 master_service.cc:948] ReplaceTablet: received request to replace tablet 3dae00aa38204ee3ac77325eb334be42 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:27.024354 1485 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: default.Table_bcd502e1360b48b1953fa97eadce7006 [id=b3914f33e856466e8479d0a2263ec56f] alter complete (version 1)
I20250901 01:52:26.986539 1151 tablet_bootstrap.cc:654] T 8ed6616ce54548ba83ba299c4ecf3c38 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:27.025628 1483 master_service.cc:948] ReplaceTablet: received request to replace tablet 008d23f236c348b6be9aee718f8698af from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:27.017170 992 leader_election.cc:304] T eaa521c52a2f4408b59770bd2a05907c P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:27.026232 1491 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_3552791ff8104f71a25c92ce0842e349" } new_table_name: "default.Table_985c3c59b6084cd596b5478e83ddb7b9" modify_external_catalogs: true
I20250901 01:52:27.026743 2306 raft_consensus.cc:2802] T eaa521c52a2f4408b59770bd2a05907c P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:27.026819 2306 raft_consensus.cc:491] T eaa521c52a2f4408b59770bd2a05907c P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:27.026876 2306 raft_consensus.cc:3058] T eaa521c52a2f4408b59770bd2a05907c P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:27.024839 2113 ts_tablet_manager.cc:1905] T 8dd587304e6342edbe96c9897b40e5c9 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:27.028241 1490 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_cbeebbc633854c008e07b7d39d2024ee" } new_table_name: "default.Table_2fbca8463d6546c9b8f5b018fc60251d" modify_external_catalogs: true
I20250901 01:52:27.029536 2306 raft_consensus.cc:513] T eaa521c52a2f4408b59770bd2a05907c P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:27.031651 1058 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aec951f2cc84499b9162b9fa24d5440e" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:27.031759 2306 leader_election.cc:290] T eaa521c52a2f4408b59770bd2a05907c P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:27.032330 1484 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_fc0336a00d084f6e98e1d8d32142e66f" } new_table_name: "default.Table_ef6832d0845044ed9f165ca3541b4c3f" modify_external_catalogs: true
I20250901 01:52:27.036989 1489 master_service.cc:948] ReplaceTablet: received request to replace tablet d10c193452954c198bba686169b8025b from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:27.039639 1483 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 008d23f236c348b6be9aee718f8698af
W20250901 01:52:27.039927 1490 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9fbd44641d9e43c3b592e388e4adf350 consensus state has no leader
W20250901 01:52:27.040181 1491 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 219fe3882d884ee49b08671b547cac85 consensus state has no leader
W20250901 01:52:27.039921 1483 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 008d23f236c348b6be9aee718f8698af: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:27.040684 1483 catalog_manager.cc:6652] ReplaceTablet: tablet 008d23f236c348b6be9aee718f8698af deleted and replaced by tablet b1ba0e19dc0b462e80c537bf2d183f5b
I20250901 01:52:27.041075 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 008d23f236c348b6be9aee718f8698af with delete_type TABLET_DATA_DELETED (replaced by tablet b1ba0e19dc0b462e80c537bf2d183f5b) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:27.043121 1492 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 3dae00aa38204ee3ac77325eb334be42
W20250901 01:52:27.043293 1492 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 3dae00aa38204ee3ac77325eb334be42: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:27.043654 1492 catalog_manager.cc:6652] ReplaceTablet: tablet 3dae00aa38204ee3ac77325eb334be42 deleted and replaced by tablet 76ba642c16734cdf969dbd3130639fda
I20250901 01:52:27.044286 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 3dae00aa38204ee3ac77325eb334be42 with delete_type TABLET_DATA_DELETED (replaced by tablet 76ba642c16734cdf969dbd3130639fda) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:27.050837 1151 tablet_bootstrap.cc:492] T 8ed6616ce54548ba83ba299c4ecf3c38 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:27.051033 1151 ts_tablet_manager.cc:1397] T 8ed6616ce54548ba83ba299c4ecf3c38 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.067s user 0.004s sys 0.000s
I20250901 01:52:27.051827 1151 raft_consensus.cc:357] T 8ed6616ce54548ba83ba299c4ecf3c38 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.052042 1151 raft_consensus.cc:383] T 8ed6616ce54548ba83ba299c4ecf3c38 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:27.052140 1151 raft_consensus.cc:738] T 8ed6616ce54548ba83ba299c4ecf3c38 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:27.052423 1151 consensus_queue.cc:260] T 8ed6616ce54548ba83ba299c4ecf3c38 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.053021 1151 ts_tablet_manager.cc:1428] T 8ed6616ce54548ba83ba299c4ecf3c38 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250901 01:52:27.053352 1151 tablet_bootstrap.cc:492] T 641a83a3e50d451189c33c7b7b4ca070 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
W20250901 01:52:27.053619 992 consensus_peers.cc:489] T 831e42de95854541a49ae6b3e039d3e9 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Couldn't send request to peer 00c282f779c1477d920f24dc0dc16628. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20250901 01:52:27.054627 992 leader_election.cc:336] T eaa521c52a2f4408b59770bd2a05907c P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:27.055135 1484 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 641a83a3e50d451189c33c7b7b4ca070 consensus state has no leader
I20250901 01:52:27.055444 1151 tablet_bootstrap.cc:654] T 641a83a3e50d451189c33c7b7b4ca070 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:27.056636 1492 master_service.cc:948] ReplaceTablet: received request to replace tablet 0cb3262973934c8689b851adad60afe0 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:27.056740 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7e3fb6770ea44eabb50a4ce9c3596fd6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:27.059248 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_0fdb75bf1d304cf8855165825780fe35"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:27.060659 992 consensus_peers.cc:489] T 42c1156e8dd84c6fa1396b46de13eb6f P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:27.068112 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ab6efbc3012a471e930f2a7ae1361355"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:27.069798 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_91ca89a1319841ea99c228392122f01f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:27.061513 2307 raft_consensus.cc:491] T 081e2630e284498aa819f85bf8326315 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250901 01:52:27.071180 992 leader_election.cc:336] T eaa521c52a2f4408b59770bd2a05907c P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:27.071143 2307 raft_consensus.cc:513] T 081e2630e284498aa819f85bf8326315 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.071552 1492 master_service.cc:948] ReplaceTablet: received request to replace tablet 641a83a3e50d451189c33c7b7b4ca070 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:27.071672 2307 leader_election.cc:290] T 081e2630e284498aa819f85bf8326315 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:27.066308 1151 tablet_bootstrap.cc:492] T 641a83a3e50d451189c33c7b7b4ca070 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:27.071998 1151 ts_tablet_manager.cc:1397] T 641a83a3e50d451189c33c7b7b4ca070 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.000s
I20250901 01:52:27.072060 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9d32cae853e54597a9e77e9a962fab75"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:27.072168 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_01692e24241541f0a6606ff2b186aa86"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:27.072825 1151 raft_consensus.cc:357] T 641a83a3e50d451189c33c7b7b4ca070 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.072851 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c5f32027e8054e769f6140f34c79db56"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:27.072978 1151 raft_consensus.cc:383] T 641a83a3e50d451189c33c7b7b4ca070 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:27.073029 1151 raft_consensus.cc:738] T 641a83a3e50d451189c33c7b7b4ca070 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:27.072954 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_71fec82e9a1840f18959b26444682ceb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:27.073222 1151 consensus_queue.cc:260] T 641a83a3e50d451189c33c7b7b4ca070 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.073447 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d2e73552afc740d2b7de4be756562f46"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:27.073711 1151 ts_tablet_manager.cc:1428] T 641a83a3e50d451189c33c7b7b4ca070 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250901 01:52:27.073884 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b0a90aea1d0b49d390544ff0af6b0f7e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:27.073990 1151 tablet_bootstrap.cc:492] T c58edaa76ebf47dfb1d9d5fab6de7dfa P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:27.071846 1482 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 435b1cca5a7d4d49959a310cb250b2a0
W20250901 01:52:27.074450 1482 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 435b1cca5a7d4d49959a310cb250b2a0: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:27.075652 1151 tablet_bootstrap.cc:654] T c58edaa76ebf47dfb1d9d5fab6de7dfa P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:27.078537 1490 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_7dd48000e9b24337b75ceb4c53b6be25" } new_table_name: "default.Table_1e59568bdcd14662ae9d5e98293d74bd" modify_external_catalogs: true
I20250901 01:52:27.078964 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 435b1cca5a7d4d49959a310cb250b2a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:26 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:27.079247 1482 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_59039012009944fdbe0179002ea44633" } modify_external_catalogs: true
I20250901 01:52:27.079434 1482 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_59039012009944fdbe0179002ea44633" } modify_external_catalogs: true
I20250901 01:52:27.089134 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b7a4839ef53c4c82a2eb175c5d3bbdff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:27.090108 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2c666422ec524843b96c1c8c342cec1e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:27.090338 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e6ce29a0703b45d88c8ad2272add62ee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:27.090677 1104 heartbeater.cc:344] Connected to a master server at 127.0.104.62:43871
I20250901 01:52:27.090824 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d76b1dfd829a4da4bf4c139922d64658"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:27.091094 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_bbedabdd989e49aeaa7ba3d585330866"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:27.091491 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e75c78a1090f4322bb65313cf8d2e7df"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:27.091922 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8411339d27f04e36823df82d1127d880"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:27.090692 992 leader_election.cc:336] T 081e2630e284498aa819f85bf8326315 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:27.092140 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2419d7f6c4ba4720b15f8a0189ddf501"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:27.093068 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e3e0b0ed4d87431692f5fe29e3e9ae1f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:27.093613 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4bb8b8ac0b144b7785d20ef96c630223"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:27.099841 992 consensus_peers.cc:489] T 40d74d7b5a3b4d1dae09eebd9e07a0c4 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Couldn't send request to peer 00c282f779c1477d920f24dc0dc16628. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 40d74d7b5a3b4d1dae09eebd9e07a0c4. This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:27.099509 2113 ts_tablet_manager.cc:1918] T 8dd587304e6342edbe96c9897b40e5c9 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:27.100495 2301 tablet.cc:1722] T aa424ac6ef1441e6b1cc2867b1c48186 P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:27.100529 2113 log.cc:1199] T 8dd587304e6342edbe96c9897b40e5c9 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/8dd587304e6342edbe96c9897b40e5c9
W20250901 01:52:27.101540 1490 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7304937048ca48a7bd153f3cdcfaeff0 consensus state has no leader
I20250901 01:52:27.102461 2113 ts_tablet_manager.cc:1939] T 8dd587304e6342edbe96c9897b40e5c9 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:27.105072 1482 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet f23df6f253a943fa8e146bb8c811ef51
W20250901 01:52:27.105444 1482 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet f23df6f253a943fa8e146bb8c811ef51: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:27.105525 1492 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 641a83a3e50d451189c33c7b7b4ca070
W20250901 01:52:27.105840 1492 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 641a83a3e50d451189c33c7b7b4ca070: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:27.106673 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet f23df6f253a943fa8e146bb8c811ef51 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:27 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:27.106774 1492 catalog_manager.cc:6652] ReplaceTablet: tablet 641a83a3e50d451189c33c7b7b4ca070 deleted and replaced by tablet 5f0e054f62de4cf6bc6f69ad6664b192
I20250901 01:52:27.106786 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 65) took 2154 ms. Trace:
I20250901 01:52:27.107184 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 641a83a3e50d451189c33c7b7b4ca070 with delete_type TABLET_DATA_DELETED (replaced by tablet 5f0e054f62de4cf6bc6f69ad6664b192) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:27.107187 2113 rpcz_store.cc:276] 0901 01:52:24.952658 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:24.952755 (+ 97us) service_pool.cc:225] Handling call
0901 01:52:26.976365 (+2023610us) ts_tablet_manager.cc:1145] Deleting tablet 8dd587304e6342edbe96c9897b40e5c9
0901 01:52:26.976378 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:27.046884 (+ 70506us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:27.098908 (+ 52024us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:27.106772 (+ 7864us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7622,"tablet-delete.queue_time_us":2022788}
I20250901 01:52:27.109104 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 8dd587304e6342edbe96c9897b40e5c9 (table default.Table_ee242c34410e4c6887405d04a4b26124 [id=c9a2f96383664a4e8524c286cd9e4149]) successfully deleted
I20250901 01:52:27.117389 1151 tablet_bootstrap.cc:492] T c58edaa76ebf47dfb1d9d5fab6de7dfa P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:27.117534 1151 ts_tablet_manager.cc:1397] T c58edaa76ebf47dfb1d9d5fab6de7dfa P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.044s user 0.006s sys 0.007s
I20250901 01:52:27.118338 1151 raft_consensus.cc:357] T c58edaa76ebf47dfb1d9d5fab6de7dfa P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.118497 1151 raft_consensus.cc:383] T c58edaa76ebf47dfb1d9d5fab6de7dfa P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:27.118546 1151 raft_consensus.cc:738] T c58edaa76ebf47dfb1d9d5fab6de7dfa P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:27.118726 1151 consensus_queue.cc:260] T c58edaa76ebf47dfb1d9d5fab6de7dfa P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.119261 1151 ts_tablet_manager.cc:1428] T c58edaa76ebf47dfb1d9d5fab6de7dfa P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250901 01:52:27.120254 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 67) took 2147 ms. Trace:
I20250901 01:52:27.120335 2113 rpcz_store.cc:276] 0901 01:52:24.972871 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:24.972923 (+ 52us) service_pool.cc:225] Handling call
0901 01:52:27.112808 (+2139885us) ts_tablet_manager.cc:1145] Deleting tablet 74b709cb5ec64006b2f12bead8217d80
0901 01:52:27.120092 (+ 7284us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:27.120236 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2139708}
I20250901 01:52:27.041623 898 tablet_service.cc:1515] Processing DeleteTablet for tablet 008d23f236c348b6be9aee718f8698af with delete_type TABLET_DATA_DELETED (replaced by tablet b1ba0e19dc0b462e80c537bf2d183f5b) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:27.120870 2118 log.cc:1199] T 831e42de95854541a49ae6b3e039d3e9 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/831e42de95854541a49ae6b3e039d3e9
I20250901 01:52:27.045117 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 3dae00aa38204ee3ac77325eb334be42 with delete_type TABLET_DATA_DELETED (replaced by tablet 76ba642c16734cdf969dbd3130639fda) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:27.047271 922 raft_consensus.cc:1273] T 42c1156e8dd84c6fa1396b46de13eb6f P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Refusing update from remote peer fd27ab6665304aaea53adbd102dd4f58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250901 01:52:27.050261 917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eaa521c52a2f4408b59770bd2a05907c" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:27.121222 917 raft_consensus.cc:3058] T eaa521c52a2f4408b59770bd2a05907c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:27.121287 2118 ts_tablet_manager.cc:1939] T 831e42de95854541a49ae6b3e039d3e9 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:27.122903 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 50) took 1921 ms. Trace:
I20250901 01:52:27.122985 2118 rpcz_store.cc:276] 0901 01:52:25.201576 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.201611 (+ 35us) service_pool.cc:225] Handling call
0901 01:52:26.998965 (+1797354us) ts_tablet_manager.cc:1145] Deleting tablet 831e42de95854541a49ae6b3e039d3e9
0901 01:52:26.998975 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:27.004564 (+ 5589us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:27.009849 (+ 5285us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:27.122886 (+113037us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1797164}
I20250901 01:52:27.123590 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 51) took 1921 ms. Trace:
I20250901 01:52:27.123662 2118 rpcz_store.cc:276] 0901 01:52:25.201948 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.201982 (+ 34us) service_pool.cc:225] Handling call
0901 01:52:27.123464 (+1921482us) ts_tablet_manager.cc:1145] Deleting tablet f06a77ecc841462396a8d5026c7327f4
0901 01:52:27.123473 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:27.123575 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1921335}
I20250901 01:52:27.123811 917 raft_consensus.cc:2466] T eaa521c52a2f4408b59770bd2a05907c P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:27.124002 2118 tablet_replica.cc:331] T 64fb46b942ff4e4c8749986ecc3dfe65 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
W20250901 01:52:27.051153 859 leader_election.cc:343] T aec951f2cc84499b9162b9fa24d5440e P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:27.124178 2118 raft_consensus.cc:2241] T 64fb46b942ff4e4c8749986ecc3dfe65 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:27.124147 859 leader_election.cc:304] T aec951f2cc84499b9162b9fa24d5440e P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:27.124322 2118 raft_consensus.cc:2270] T 64fb46b942ff4e4c8749986ecc3dfe65 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250901 01:52:27.124519 859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:39447, user_credentials={real_user=slave}} blocked reactor thread for 73376.8us
I20250901 01:52:27.075296 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 435b1cca5a7d4d49959a310cb250b2a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:26 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:27.124778 2323 raft_consensus.cc:2747] T aec951f2cc84499b9162b9fa24d5440e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:27.077033 1150 tablet_bootstrap.cc:492] T b164b4a67a9143ceaff69b0ce49d63fe P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:27.129562 1150 ts_tablet_manager.cc:1397] T b164b4a67a9143ceaff69b0ce49d63fe P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.243s user 0.006s sys 0.014s
I20250901 01:52:27.129621 992 leader_election.cc:304] T eaa521c52a2f4408b59770bd2a05907c P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:27.130354 1150 raft_consensus.cc:357] T b164b4a67a9143ceaff69b0ce49d63fe P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:27.130511 1150 raft_consensus.cc:383] T b164b4a67a9143ceaff69b0ce49d63fe P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:27.130566 1150 raft_consensus.cc:738] T b164b4a67a9143ceaff69b0ce49d63fe P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:27.130759 1150 consensus_queue.cc:260] T b164b4a67a9143ceaff69b0ce49d63fe P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:27.131244 1150 ts_tablet_manager.cc:1428] T b164b4a67a9143ceaff69b0ce49d63fe P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250901 01:52:27.131517 1150 tablet_bootstrap.cc:492] T cb7d7e1796c0456f8de98fb812b9e374 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:27.132277 1993 consensus_queue.cc:1035] T 42c1156e8dd84c6fa1396b46de13eb6f P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:27.132787 1993 raft_consensus.cc:2802] T eaa521c52a2f4408b59770bd2a05907c P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:27.132899 1993 raft_consensus.cc:695] T eaa521c52a2f4408b59770bd2a05907c P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:27.079015 2123 raft_consensus.cc:491] T eb19e3a424324a24824f6caaf5848952 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:27.133011 2123 raft_consensus.cc:513] T eb19e3a424324a24824f6caaf5848952 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.133106 1993 consensus_queue.cc:237] T eaa521c52a2f4408b59770bd2a05907c P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:27.133531 2123 leader_election.cc:290] T eb19e3a424324a24824f6caaf5848952 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:27.081298 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "081e2630e284498aa819f85bf8326315" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:27.133770 913 raft_consensus.cc:2466] T 081e2630e284498aa819f85bf8326315 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:27.106771 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 641a83a3e50d451189c33c7b7b4ca070 with delete_type TABLET_DATA_DELETED (replaced by tablet 5f0e054f62de4cf6bc6f69ad6664b192) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:27.107999 901 tablet_service.cc:1515] Processing DeleteTablet for tablet f23df6f253a943fa8e146bb8c811ef51 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:27 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:27.111764 2354 tablet.cc:1722] T aa424ac6ef1441e6b1cc2867b1c48186 P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:27.116123 2357 raft_consensus.cc:491] T 31614db4c53a4f3c96fd190d326ce601 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:27.135497 2357 raft_consensus.cc:513] T 31614db4c53a4f3c96fd190d326ce601 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.136068 2357 leader_election.cc:290] T 31614db4c53a4f3c96fd190d326ce601 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:27.140000 858 leader_election.cc:336] T eb19e3a424324a24824f6caaf5848952 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:27.140944 858 leader_election.cc:336] T 31614db4c53a4f3c96fd190d326ce601 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:27.142055 992 leader_election.cc:304] T 081e2630e284498aa819f85bf8326315 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:27.142344 2307 raft_consensus.cc:2802] T 081e2630e284498aa819f85bf8326315 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:27.142448 2307 raft_consensus.cc:491] T 081e2630e284498aa819f85bf8326315 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:27.142514 2307 raft_consensus.cc:3058] T 081e2630e284498aa819f85bf8326315 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:27.143033 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "31614db4c53a4f3c96fd190d326ce601" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:27.144131 859 leader_election.cc:343] T 31614db4c53a4f3c96fd190d326ce601 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:27.144232 859 leader_election.cc:304] T 31614db4c53a4f3c96fd190d326ce601 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:27.144608 2357 raft_consensus.cc:2747] T 31614db4c53a4f3c96fd190d326ce601 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:27.145558 2307 raft_consensus.cc:513] T 081e2630e284498aa819f85bf8326315 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.146301 2307 leader_election.cc:290] T 081e2630e284498aa819f85bf8326315 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:27.146543 1058 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eb19e3a424324a24824f6caaf5848952" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:27.147249 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "081e2630e284498aa819f85bf8326315" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:27.147420 913 raft_consensus.cc:3058] T 081e2630e284498aa819f85bf8326315 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:27.148417 2118 ts_tablet_manager.cc:1905] T 64fb46b942ff4e4c8749986ecc3dfe65 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:27.150686 913 raft_consensus.cc:2466] T 081e2630e284498aa819f85bf8326315 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
W20250901 01:52:27.151706 859 leader_election.cc:343] T eb19e3a424324a24824f6caaf5848952 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:27.151818 859 leader_election.cc:304] T eb19e3a424324a24824f6caaf5848952 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:27.152172 2357 raft_consensus.cc:2747] T eb19e3a424324a24824f6caaf5848952 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:27.152308 992 leader_election.cc:304] T 081e2630e284498aa819f85bf8326315 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
W20250901 01:52:27.152573 992 leader_election.cc:336] T 081e2630e284498aa819f85bf8326315 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:27.152575 2307 raft_consensus.cc:2802] T 081e2630e284498aa819f85bf8326315 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:27.152873 1150 tablet_bootstrap.cc:654] T cb7d7e1796c0456f8de98fb812b9e374 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:27.153098 2307 raft_consensus.cc:695] T 081e2630e284498aa819f85bf8326315 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:27.153558 2307 consensus_queue.cc:237] T 081e2630e284498aa819f85bf8326315 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.157784 1489 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet d10c193452954c198bba686169b8025b
I20250901 01:52:27.158114 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_6e97156b18054fce942ec71bda719c05"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:27.160234 1151 tablet_bootstrap.cc:492] T 89d40f0331114490b51de3ba6d773571 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:27.160287 1488 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: default.Table_eecfab7239cc40c5a2f1fb6ac6d35dc9 [id=d3dbeb4258424133844e20b181ee44e4] alter complete (version 1)
W20250901 01:52:27.160946 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 74b709cb5ec64006b2f12bead8217d80 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 74b709cb5ec64006b2f12bead8217d80 already in progress: creating tablet
I20250901 01:52:27.161581 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_97eb6a187cdc4e3483bba07dd252c747"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:27.161988 1151 tablet_bootstrap.cc:654] T 89d40f0331114490b51de3ba6d773571 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:27.162122 1489 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet d10c193452954c198bba686169b8025b: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:27.165498 2113 tablet_replica.cc:331] T f50f33de1b654663b9ef8a91e3436f11 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:27.165879 2113 raft_consensus.cc:2241] T f50f33de1b654663b9ef8a91e3436f11 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:27.166309 2113 raft_consensus.cc:2270] T f50f33de1b654663b9ef8a91e3436f11 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:27.167037 2113 ts_tablet_manager.cc:1905] T f50f33de1b654663b9ef8a91e3436f11 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:27.169829 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet d10c193452954c198bba686169b8025b with delete_type TABLET_DATA_DELETED (replaced by tablet a6b539e160194da3a849d9e2f1ec9859) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:27.171240 2357 raft_consensus.cc:491] T 9fbd44641d9e43c3b592e388e4adf350 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:27.171384 2357 raft_consensus.cc:513] T 9fbd44641d9e43c3b592e388e4adf350 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.172019 2357 leader_election.cc:290] T 9fbd44641d9e43c3b592e388e4adf350 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:27.176360 858 leader_election.cc:336] T 9fbd44641d9e43c3b592e388e4adf350 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:27.180452 1058 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9fbd44641d9e43c3b592e388e4adf350" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:27.160224 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 831e42de95854541a49ae6b3e039d3e9 (table default.Table_113664a018e84769a7ff61aa35a38618 [id=68214167814a4ecbab3ec100e83f4a64]) successfully deleted
W20250901 01:52:27.182758 859 leader_election.cc:343] T 9fbd44641d9e43c3b592e388e4adf350 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:27.182864 859 leader_election.cc:304] T 9fbd44641d9e43c3b592e388e4adf350 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:27.183171 2357 raft_consensus.cc:2747] T 9fbd44641d9e43c3b592e388e4adf350 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:27.183954 1472 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.2:40561, user_credentials={real_user=slave}} blocked reactor thread for 58574.7us
I20250901 01:52:27.184937 901 tablet_service.cc:1515] Processing DeleteTablet for tablet 48a211e174054866b030cdfb0cf947f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:18 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:27.185111 885 tablet_service.cc:1515] Processing DeleteTablet for tablet b85d64e02cc646e380581ac9f59dee9e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:27.185570 885 tablet_service.cc:1515] Processing DeleteTablet for tablet d10c193452954c198bba686169b8025b with delete_type TABLET_DATA_DELETED (replaced by tablet a6b539e160194da3a849d9e2f1ec9859) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:27.185983 1151 tablet_bootstrap.cc:492] T 89d40f0331114490b51de3ba6d773571 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:27.186162 1151 ts_tablet_manager.cc:1397] T 89d40f0331114490b51de3ba6d773571 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.026s user 0.009s sys 0.001s
I20250901 01:52:27.186321 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_16674b0eec084adbb4dace654f52362a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:27.186738 1486 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet ff013ea889d44cba8797834e4611b489
I20250901 01:52:27.187171 1151 raft_consensus.cc:357] T 89d40f0331114490b51de3ba6d773571 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.187383 1151 raft_consensus.cc:383] T 89d40f0331114490b51de3ba6d773571 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:27.187477 1151 raft_consensus.cc:738] T 89d40f0331114490b51de3ba6d773571 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:27.187592 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 1a60c10e42104502ac2d3b80b8126ceb with delete_type TABLET_DATA_DELETED (replaced by tablet 88ceba7f48b04d7885cc0164668700a5) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:27.187813 1151 consensus_queue.cc:260] T 89d40f0331114490b51de3ba6d773571 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.188097 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 56475ba491a24301a534304811f30140 with delete_type TABLET_DATA_DELETED (replaced by tablet 7317eb7848ee44698038a4b404d4bf21) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:27.188483 1151 ts_tablet_manager.cc:1428] T 89d40f0331114490b51de3ba6d773571 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250901 01:52:27.188833 1151 tablet_bootstrap.cc:492] T ea846f88b3534beebd0001b7f17b5ca9 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
W20250901 01:52:27.189862 1486 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet ff013ea889d44cba8797834e4611b489: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:27.190752 1489 catalog_manager.cc:6652] ReplaceTablet: tablet d10c193452954c198bba686169b8025b deleted and replaced by tablet a6b539e160194da3a849d9e2f1ec9859
I20250901 01:52:27.190979 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet ff013ea889d44cba8797834e4611b489 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:26 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:27.191262 2357 raft_consensus.cc:491] T f9b293bf6a17453b9ea9892a6970d0b0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:27.191412 2357 raft_consensus.cc:513] T f9b293bf6a17453b9ea9892a6970d0b0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.191546 1151 tablet_bootstrap.cc:654] T ea846f88b3534beebd0001b7f17b5ca9 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:27.191992 2357 leader_election.cc:290] T f9b293bf6a17453b9ea9892a6970d0b0 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:27.193500 2357 raft_consensus.cc:491] T 641a83a3e50d451189c33c7b7b4ca070 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:27.193607 2357 raft_consensus.cc:513] T 641a83a3e50d451189c33c7b7b4ca070 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.194007 2357 leader_election.cc:290] T 641a83a3e50d451189c33c7b7b4ca070 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:27.194082 2113 ts_tablet_manager.cc:1918] T f50f33de1b654663b9ef8a91e3436f11 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:27.194938 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_70baadb21512494ba5d5efa66a2ac58f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:27.195829 858 leader_election.cc:336] T f9b293bf6a17453b9ea9892a6970d0b0 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:27.197268 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_83beecdb93984588b113dd34a7d1fbc9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:27.197557 858 leader_election.cc:336] T 641a83a3e50d451189c33c7b7b4ca070 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:27.197863 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_0de653af362b42d0aebe014cd38973fb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:27.198734 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_6fa48b9ec1d54daa80d300f45070449b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:27.201233 1150 tablet_bootstrap.cc:492] T cb7d7e1796c0456f8de98fb812b9e374 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:27.201401 1150 ts_tablet_manager.cc:1397] T cb7d7e1796c0456f8de98fb812b9e374 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.070s user 0.003s sys 0.002s
I20250901 01:52:27.202176 1150 raft_consensus.cc:357] T cb7d7e1796c0456f8de98fb812b9e374 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.202426 1150 raft_consensus.cc:383] T cb7d7e1796c0456f8de98fb812b9e374 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:27.202549 1150 raft_consensus.cc:738] T cb7d7e1796c0456f8de98fb812b9e374 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:27.195856 1103 heartbeater.cc:499] Master 127.0.104.60:44231 was elected leader, sending a full tablet report...
I20250901 01:52:27.202893 1150 consensus_queue.cc:260] T cb7d7e1796c0456f8de98fb812b9e374 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.203505 1150 ts_tablet_manager.cc:1428] T cb7d7e1796c0456f8de98fb812b9e374 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250901 01:52:27.203825 1150 tablet_bootstrap.cc:492] T 3eba54e5ddd14c518f4cbd539268ae37 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:27.205258 1058 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9b293bf6a17453b9ea9892a6970d0b0" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:27.206548 1150 tablet_bootstrap.cc:654] T 3eba54e5ddd14c518f4cbd539268ae37 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:27.207055 1058 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "641a83a3e50d451189c33c7b7b4ca070" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:27.207221 1058 raft_consensus.cc:2466] T 641a83a3e50d451189c33c7b7b4ca070 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00c282f779c1477d920f24dc0dc16628 in term 0.
I20250901 01:52:27.208027 2118 ts_tablet_manager.cc:1918] T 64fb46b942ff4e4c8749986ecc3dfe65 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:27.208042 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_933c70ced5ec4f53a24860fd5c936b6a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:27.208185 2118 log.cc:1199] T 64fb46b942ff4e4c8749986ecc3dfe65 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/64fb46b942ff4e4c8749986ecc3dfe65
I20250901 01:52:27.208734 2118 ts_tablet_manager.cc:1939] T 64fb46b942ff4e4c8749986ecc3dfe65 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:27.209115 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 52) took 2006 ms. Trace:
I20250901 01:52:27.209084 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9b216d017def437e8384849b622ec49a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:27.209365 2118 rpcz_store.cc:276] 0901 01:52:25.202232 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.202263 (+ 31us) service_pool.cc:225] Handling call
0901 01:52:27.123961 (+1921698us) ts_tablet_manager.cc:1145] Deleting tablet 64fb46b942ff4e4c8749986ecc3dfe65
0901 01:52:27.123970 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:27.151764 (+ 27794us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:27.184766 (+ 33002us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:27.209082 (+ 24316us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1921509}
I20250901 01:52:27.209887 2118 tablet_replica.cc:331] T c763d252598745a088344aa0eb5fa3d3 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:27.210136 2118 raft_consensus.cc:2241] T c763d252598745a088344aa0eb5fa3d3 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
W20250901 01:52:27.209559 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet f06a77ecc841462396a8d5026c7327f4 because the tablet was not found. No further retry: Not found: Tablet not found: f06a77ecc841462396a8d5026c7327f4
I20250901 01:52:27.210371 2118 raft_consensus.cc:2270] T c763d252598745a088344aa0eb5fa3d3 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:27.210875 2118 ts_tablet_manager.cc:1905] T c763d252598745a088344aa0eb5fa3d3 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:27.211273 885 tablet_service.cc:1515] Processing DeleteTablet for tablet afc002ae2486476b9208e1cf8215157c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:17 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:27.211273 901 tablet_service.cc:1515] Processing DeleteTablet for tablet ff013ea889d44cba8797834e4611b489 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:26 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:27.212471 2113 log.cc:1199] T f50f33de1b654663b9ef8a91e3436f11 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/f50f33de1b654663b9ef8a91e3436f11
I20250901 01:52:27.212877 2113 ts_tablet_manager.cc:1939] T f50f33de1b654663b9ef8a91e3436f11 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
W20250901 01:52:27.212908 859 leader_election.cc:343] T f9b293bf6a17453b9ea9892a6970d0b0 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:27.213020 859 leader_election.cc:304] T f9b293bf6a17453b9ea9892a6970d0b0 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:27.213379 859 leader_election.cc:304] T 641a83a3e50d451189c33c7b7b4ca070 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:27.213722 2357 raft_consensus.cc:2747] T f9b293bf6a17453b9ea9892a6970d0b0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:27.213887 2357 raft_consensus.cc:2802] T 641a83a3e50d451189c33c7b7b4ca070 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:27.213935 2357 raft_consensus.cc:491] T 641a83a3e50d451189c33c7b7b4ca070 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:27.213986 2357 raft_consensus.cc:3058] T 641a83a3e50d451189c33c7b7b4ca070 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:27.217226 2357 raft_consensus.cc:513] T 641a83a3e50d451189c33c7b7b4ca070 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:27.217888 992 consensus_peers.cc:489] T c763d252598745a088344aa0eb5fa3d3 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Couldn't send request to peer 00c282f779c1477d920f24dc0dc16628. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:27.218802 2307 consensus_queue.cc:1035] T f80f13db501c42c7950499df8fcb6fc8 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250901 01:52:27.219941 858 leader_election.cc:336] T 641a83a3e50d451189c33c7b7b4ca070 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:27.221895 2357 leader_election.cc:290] T 641a83a3e50d451189c33c7b7b4ca070 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:27.232946 1150 tablet_bootstrap.cc:492] T 3eba54e5ddd14c518f4cbd539268ae37 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:27.236497 1058 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "641a83a3e50d451189c33c7b7b4ca070" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58"
I20250901 01:52:27.236718 1058 raft_consensus.cc:3058] T 641a83a3e50d451189c33c7b7b4ca070 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:27.239466 1058 raft_consensus.cc:2466] T 641a83a3e50d451189c33c7b7b4ca070 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c282f779c1477d920f24dc0dc16628 in term 1.
I20250901 01:52:27.239301 2357 raft_consensus.cc:491] T b13afacd37474781877b9f2cd20b313a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:27.242249 2357 raft_consensus.cc:513] T b13afacd37474781877b9f2cd20b313a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:27.242923 2357 leader_election.cc:290] T b13afacd37474781877b9f2cd20b313a P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:27.243379 859 leader_election.cc:304] T 641a83a3e50d451189c33c7b7b4ca070 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:27.242261 1150 ts_tablet_manager.cc:1397] T 3eba54e5ddd14c518f4cbd539268ae37 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.038s user 0.001s sys 0.002s
I20250901 01:52:27.243835 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_dfe9c127a1aa40b89e0b1fb35c644f07"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:27.244232 1150 raft_consensus.cc:357] T 3eba54e5ddd14c518f4cbd539268ae37 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.244864 1488 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: default.Table_589c302e02474fc3bc5d3a290a8bd54a [id=5d14706774d345c48ca3575feabb17b0] alter complete (version 2)
I20250901 01:52:27.245675 2357 raft_consensus.cc:2802] T 641a83a3e50d451189c33c7b7b4ca070 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:27.245783 2357 raft_consensus.cc:695] T 641a83a3e50d451189c33c7b7b4ca070 P 00c282f779c1477d920f24dc0dc16628 [term 1 LEADER]: Becoming Leader. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Running, Role: LEADER
I20250901 01:52:27.245993 2357 consensus_queue.cc:237] T 641a83a3e50d451189c33c7b7b4ca070 P 00c282f779c1477d920f24dc0dc16628 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.248667 1490 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_efc44c841f0b4ae2a168be904751b65c" } modify_external_catalogs: true
I20250901 01:52:27.248849 1490 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_efc44c841f0b4ae2a168be904751b65c" } modify_external_catalogs: true
I20250901 01:52:27.249497 1151 tablet_bootstrap.cc:492] T ea846f88b3534beebd0001b7f17b5ca9 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
W20250901 01:52:27.249485 858 leader_election.cc:336] T b13afacd37474781877b9f2cd20b313a P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:27.249715 1151 ts_tablet_manager.cc:1397] T ea846f88b3534beebd0001b7f17b5ca9 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.061s user 0.016s sys 0.005s
I20250901 01:52:27.250537 1151 raft_consensus.cc:357] T ea846f88b3534beebd0001b7f17b5ca9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.250757 1151 raft_consensus.cc:383] T ea846f88b3534beebd0001b7f17b5ca9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:27.250880 1151 raft_consensus.cc:738] T ea846f88b3534beebd0001b7f17b5ca9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:27.251123 1151 consensus_queue.cc:260] T ea846f88b3534beebd0001b7f17b5ca9 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.251669 1151 ts_tablet_manager.cc:1428] T ea846f88b3534beebd0001b7f17b5ca9 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250901 01:52:27.253896 1150 raft_consensus.cc:383] T 3eba54e5ddd14c518f4cbd539268ae37 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:27.253985 1150 raft_consensus.cc:738] T 3eba54e5ddd14c518f4cbd539268ae37 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:27.254384 1150 consensus_queue.cc:260] T 3eba54e5ddd14c518f4cbd539268ae37 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.254882 1150 ts_tablet_manager.cc:1428] T 3eba54e5ddd14c518f4cbd539268ae37 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20250901 01:52:27.255107 1150 tablet_bootstrap.cc:492] T 75345ea2a3dc40fc8c9f8090d8839d2d P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:27.256093 1489 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_5e8920128d214deb96ef83f369a8b3f1" } new_table_name: "default.Table_97622024147a400ea7db77c74af5f71e" modify_external_catalogs: true
I20250901 01:52:27.256311 1491 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_22d8d38d0b624912a0d8968fb764420e" } modify_external_catalogs: true
I20250901 01:52:27.257071 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b7abe35488004f749e8e3aefa0e74237"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:27.258215 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 64fb46b942ff4e4c8749986ecc3dfe65 (table default.Table_9bf84b7a07be4f57a58101480fe6d2c0 [id=92cdf22693354ddf93b83c67cf553a0e]) successfully deleted
W20250901 01:52:27.258365 1472 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.2:40561, user_credentials={real_user=slave}} blocked reactor thread for 47541.2us
I20250901 01:52:27.258862 901 tablet_service.cc:1515] Processing DeleteTablet for tablet 75e7a90815e449c39f386901a504143c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:18 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:27.259127 901 tablet_service.cc:1515] Processing DeleteTablet for tablet 74b709cb5ec64006b2f12bead8217d80 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:27.259271 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 05630a72ddae4de0b890dece2feacc31 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:18 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:27.259426 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 8dd587304e6342edbe96c9897b40e5c9 with delete_type TABLET_DATA_DELETED (replaced by tablet 05630a72ddae4de0b890dece2feacc31) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:27.265259 1058 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b13afacd37474781877b9f2cd20b313a" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:27.265727 1491 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_22d8d38d0b624912a0d8968fb764420e" } modify_external_catalogs: true
I20250901 01:52:27.266295 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 68) took 2286 ms. Trace:
I20250901 01:52:27.266381 2113 rpcz_store.cc:276] 0901 01:52:24.979671 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:24.979731 (+ 60us) service_pool.cc:225] Handling call
0901 01:52:27.153924 (+2174193us) ts_tablet_manager.cc:1145] Deleting tablet f50f33de1b654663b9ef8a91e3436f11
0901 01:52:27.165439 (+ 11515us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:27.178643 (+ 13204us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:27.193993 (+ 15350us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:27.266276 (+ 72283us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2173978}
I20250901 01:52:27.266726 2113 tablet_replica.cc:331] T 64fb46b942ff4e4c8749986ecc3dfe65 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:27.267055 1151 tablet_bootstrap.cc:492] T 71941b815d6840478c5acc1cbb187cc9 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:27.268368 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 40d74d7b5a3b4d1dae09eebd9e07a0c4 with delete_type TABLET_DATA_DELETED (replaced by tablet 6cf09e8f299548dba2cce16d7b565bad) from {username='slave'} at 127.0.0.1:34472
W20250901 01:52:27.269903 859 leader_election.cc:343] T b13afacd37474781877b9f2cd20b313a P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:27.270025 859 leader_election.cc:304] T b13afacd37474781877b9f2cd20b313a P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:27.270226 2113 raft_consensus.cc:2241] T 64fb46b942ff4e4c8749986ecc3dfe65 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
W20250901 01:52:27.270397 992 consensus_peers.cc:489] T 15e4d972e8254cf899c3e0b71b6e2bd9 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:27.270869 2357 raft_consensus.cc:2747] T b13afacd37474781877b9f2cd20b313a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:27.271138 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_053c176d34134f5096d6383d62e65734"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:27.271440 2113 raft_consensus.cc:2270] T 64fb46b942ff4e4c8749986ecc3dfe65 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:27.271901 1151 tablet_bootstrap.cc:654] T 71941b815d6840478c5acc1cbb187cc9 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:27.274127 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2257553b886e46d5894badfaf7a66224"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:27.274928 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8ef6e5d013a0455ca8b24d6c99805a56"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:27.275069 2323 raft_consensus.cc:491] T f6a8b425968e49deaecfa3adacfd3f1f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:27.272384 2113 ts_tablet_manager.cc:1905] T 64fb46b942ff4e4c8749986ecc3dfe65 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:27.275190 2323 raft_consensus.cc:513] T f6a8b425968e49deaecfa3adacfd3f1f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.275848 2323 leader_election.cc:290] T f6a8b425968e49deaecfa3adacfd3f1f P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:27.279127 858 leader_election.cc:336] T f6a8b425968e49deaecfa3adacfd3f1f P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:27.279168 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 75345ea2a3dc40fc8c9f8090d8839d2d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:27.279361 887 tablet_service.cc:1515] Processing DeleteTablet for tablet f3962b471bdb49aea6e9fa95f1f9dfe2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:17 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:27.281893 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d690463c209f4b529935dde679ff79c7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:27.285712 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 831e42de95854541a49ae6b3e039d3e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:16 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:27.285821 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 088ef4861dc14f81ae470c3aaee5b4e1 with delete_type TABLET_DATA_DELETED (replaced by tablet 6ce9521b925b4f97bb4a810ce77e147e) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:27.286598 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_6caac3632cbf4668853ec050d81f972b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:27.287307 1150 tablet_bootstrap.cc:654] T 75345ea2a3dc40fc8c9f8090d8839d2d P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:27.287380 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c89908e0cbb04758b34f2073aa314847"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:27.288053 1486 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_b2083370690443c6aa50c453bd29cb9a" } new_table_name: "default.Table_7050639ebd0147959ac4a251f534eb96" modify_external_catalogs: true
I20250901 01:52:27.288520 901 tablet_service.cc:1515] Processing DeleteTablet for tablet 64fb46b942ff4e4c8749986ecc3dfe65 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:27.289081 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet f50f33de1b654663b9ef8a91e3436f11 (table default.Table_4f1c4695ad9e4f24a31d3522281a41a5 [id=9af016a062a64218ad3b62e131d8b65d]) successfully deleted
W20250901 01:52:27.289523 1489 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e4fdbc8d4bff4173a20db2c182acfd9e consensus state has no leader
I20250901 01:52:27.289863 1489 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_589c302e02474fc3bc5d3a290a8bd54a" } new_table_name: "default.Table_99f0fb91a6994b2abf5af0c8f701159b" modify_external_catalogs: true
I20250901 01:52:27.291559 2118 ts_tablet_manager.cc:1918] T c763d252598745a088344aa0eb5fa3d3 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:27.292248 2113 ts_tablet_manager.cc:1918] T 64fb46b942ff4e4c8749986ecc3dfe65 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:27.294135 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_26c59b2866284fc580721307cfb57baa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:27.294994 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a0f1938d517f4a29b07e26892a140616"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:27.298885 2323 raft_consensus.cc:491] T 6ba88af177874544b764cc32420e7ac9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:27.302420 2323 raft_consensus.cc:513] T 6ba88af177874544b764cc32420e7ac9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.303115 2323 leader_election.cc:290] T 6ba88af177874544b764cc32420e7ac9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:27.303218 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1e3d097adf4f4ae2aefcbaa692605a7a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:27.304134 901 tablet_service.cc:1515] Processing DeleteTablet for tablet 7a607fac57994001b2304e5d81c9decc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:17 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:27.304543 2118 log.cc:1199] T c763d252598745a088344aa0eb5fa3d3 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/c763d252598745a088344aa0eb5fa3d3
W20250901 01:52:27.305403 858 leader_election.cc:336] T 6ba88af177874544b764cc32420e7ac9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:27.295668 1151 tablet_bootstrap.cc:492] T 71941b815d6840478c5acc1cbb187cc9 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:27.312636 1151 ts_tablet_manager.cc:1397] T 71941b815d6840478c5acc1cbb187cc9 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.046s user 0.003s sys 0.001s
I20250901 01:52:27.313478 1151 raft_consensus.cc:357] T 71941b815d6840478c5acc1cbb187cc9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.313711 1151 raft_consensus.cc:383] T 71941b815d6840478c5acc1cbb187cc9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:27.313807 1151 raft_consensus.cc:738] T 71941b815d6840478c5acc1cbb187cc9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:27.314024 1151 consensus_queue.cc:260] T 71941b815d6840478c5acc1cbb187cc9 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.314545 1151 ts_tablet_manager.cc:1428] T 71941b815d6840478c5acc1cbb187cc9 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250901 01:52:27.314894 1151 tablet_bootstrap.cc:492] T f6a8b425968e49deaecfa3adacfd3f1f P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:27.315320 2118 ts_tablet_manager.cc:1939] T c763d252598745a088344aa0eb5fa3d3 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:27.315683 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 53) took 2113 ms. Trace:
I20250901 01:52:27.315894 2118 rpcz_store.cc:276] 0901 01:52:25.202590 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.202625 (+ 35us) service_pool.cc:225] Handling call
0901 01:52:27.209842 (+2007217us) ts_tablet_manager.cc:1145] Deleting tablet c763d252598745a088344aa0eb5fa3d3
0901 01:52:27.209851 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:27.247106 (+ 37255us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:27.291427 (+ 44321us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:27.315666 (+ 24239us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2007075}
I20250901 01:52:27.316390 1151 tablet_bootstrap.cc:654] T f6a8b425968e49deaecfa3adacfd3f1f P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:27.316548 2118 tablet_replica.cc:331] T bb0cf5a6a27045a08df6845d7473184d P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:27.316776 2118 raft_consensus.cc:2241] T bb0cf5a6a27045a08df6845d7473184d P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:27.316995 2118 raft_consensus.cc:2270] T bb0cf5a6a27045a08df6845d7473184d P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:27.317232 2113 log.cc:1199] T 64fb46b942ff4e4c8749986ecc3dfe65 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/64fb46b942ff4e4c8749986ecc3dfe65
I20250901 01:52:27.317552 2118 ts_tablet_manager.cc:1905] T bb0cf5a6a27045a08df6845d7473184d P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:27.317685 2113 ts_tablet_manager.cc:1939] T 64fb46b942ff4e4c8749986ecc3dfe65 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:27.317889 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet c763d252598745a088344aa0eb5fa3d3 (table default.Table_142ccc8cc7414621bc0c31326d5aa0e9 [id=6a01b6c6977546be9c6128ca5f200a92]) successfully deleted
I20250901 01:52:27.318482 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 69) took 2334 ms. Trace:
I20250901 01:52:27.318601 2113 rpcz_store.cc:276] 0901 01:52:24.984346 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:24.984447 (+ 101us) service_pool.cc:225] Handling call
0901 01:52:27.266676 (+2282229us) ts_tablet_manager.cc:1145] Deleting tablet 64fb46b942ff4e4c8749986ecc3dfe65
0901 01:52:27.266686 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:27.289311 (+ 22625us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:27.292162 (+ 2851us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:27.318464 (+ 26302us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2282017}
I20250901 01:52:27.318971 2113 tablet_replica.cc:331] T 66a48a8ebe154b2cac89c234eaa378ff P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:27.319172 2113 raft_consensus.cc:2241] T 66a48a8ebe154b2cac89c234eaa378ff P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:27.319340 2113 raft_consensus.cc:2270] T 66a48a8ebe154b2cac89c234eaa378ff P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250901 01:52:27.319461 1485 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:54603 (request call id 181) took 1276 ms (client timeout 1000 ms). Trace:
I20250901 01:52:27.319893 2113 ts_tablet_manager.cc:1905] T 66a48a8ebe154b2cac89c234eaa378ff P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:27.320833 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 64fb46b942ff4e4c8749986ecc3dfe65 (table default.Table_9bf84b7a07be4f57a58101480fe6d2c0 [id=92cdf22693354ddf93b83c67cf553a0e]) successfully deleted
I20250901 01:52:27.322561 887 tablet_service.cc:1515] Processing DeleteTablet for tablet e9b179dba2cc49428ba8090c962d6136 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:17 UTC) from {username='slave'} at 127.0.0.1:34472
E20250901 01:52:27.330075 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae4844183a3c4a04a775e6fd878391a3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_aee74198b51d4f089538c4f2bf3ce4dc'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:27.330749 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_551986093161448ca9e6c1b9506c306c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:27.332005 1482 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_e533a2595cc9420f9ab7a36bda6dc4a3" } new_table_name: "default.Table_938421db3cf74cf8b878089bac931818" modify_external_catalogs: true
I20250901 01:52:27.332949 2323 raft_consensus.cc:491] T 2599f98abfdc4871b573edfc2ad97d22 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:27.333076 2323 raft_consensus.cc:513] T 2599f98abfdc4871b573edfc2ad97d22 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:27.333370 992 consensus_peers.cc:489] T f80f13db501c42c7950499df8fcb6fc8 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:27.333642 2323 leader_election.cc:290] T 2599f98abfdc4871b573edfc2ad97d22 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:27.319586 1485 rpcz_store.cc:269] 0901 01:52:26.043085 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.043159 (+ 74us) service_pool.cc:225] Handling call
0901 01:52:26.394021 (+350862us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:26.394027 (+ 6us) write_op.cc:454] Released schema lock
0901 01:52:27.024199 (+630172us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:27.024205 (+ 6us) write_op.cc:454] Released schema lock
0901 01:52:27.319426 (+295221us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0901 01:52:26.386535 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:26.386693 (+ 158us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:26.386699 (+ 6us) write_op.cc:435] Acquired schema lock
0901 01:52:26.386702 (+ 3us) tablet.cc:660] Decoding operations
0901 01:52:26.386802 (+ 100us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:26.386818 (+ 16us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:26.386825 (+ 7us) tablet.cc:683] Acquiring locks for 1 operations
0901 01:52:26.386880 (+ 55us) tablet.cc:699] Row locks acquired
0901 01:52:26.386882 (+ 2us) write_op.cc:260] PREPARE: finished
0901 01:52:26.386918 (+ 36us) write_op.cc:270] Start()
0901 01:52:26.386950 (+ 32us) write_op.cc:276] Timestamp: P: 1756691546386863 usec, L: 0
0901 01:52:26.386954 (+ 4us) op_driver.cc:348] REPLICATION: starting
0901 01:52:26.387171 (+ 217us) log.cc:844] Serialized 356 byte log entry
0901 01:52:26.391933 (+ 4762us) op_driver.cc:464] REPLICATION: finished
0901 01:52:26.392044 (+ 111us) write_op.cc:301] APPLY: starting
0901 01:52:26.392060 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:26.392100 (+ 40us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:26.392173 (+ 73us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:26.392224 (+ 51us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:26.392265 (+ 41us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0901 01:52:26.392273 (+ 8us) write_op.cc:312] APPLY: finished
0901 01:52:26.392354 (+ 81us) log.cc:844] Serialized 23 byte log entry
0901 01:52:26.392432 (+ 78us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:26.392450 (+ 18us) write_op.cc:454] Released schema lock
0901 01:52:26.392456 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0901 01:52:26.624484 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:26.624583 (+ 99us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:26.624589 (+ 6us) write_op.cc:435] Acquired schema lock
0901 01:52:26.624592 (+ 3us) tablet.cc:660] Decoding operations
0901 01:52:26.624681 (+ 89us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:26.624695 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:26.624700 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0901 01:52:26.624760 (+ 60us) tablet.cc:699] Row locks acquired
0901 01:52:26.624764 (+ 4us) write_op.cc:260] PREPARE: finished
0901 01:52:26.624803 (+ 39us) write_op.cc:270] Start()
0901 01:52:26.624838 (+ 35us) write_op.cc:276] Timestamp: P: 1756691546624691 usec, L: 0
0901 01:52:26.624842 (+ 4us) op_driver.cc:348] REPLICATION: starting
0901 01:52:26.625055 (+ 213us) log.cc:844] Serialized 356 byte log entry
0901 01:52:26.629225 (+ 4170us) op_driver.cc:464] REPLICATION: finished
0901 01:52:26.629344 (+ 119us) write_op.cc:301] APPLY: starting
0901 01:52:26.629362 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:26.629406 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:26.629429 (+ 23us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:26.629482 (+ 53us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:26.629534 (+ 52us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0901 01:52:26.629543 (+ 9us) write_op.cc:312] APPLY: finished
0901 01:52:26.629626 (+ 83us) log.cc:844] Serialized 23 byte log entry
0901 01:52:26.629685 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:26.629703 (+ 18us) write_op.cc:454] Released schema lock
0901 01:52:26.629710 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0901 01:52:26.629778 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:26.629782 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0901 01:52:27.017446 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:27.017547 (+ 101us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:27.017553 (+ 6us) write_op.cc:435] Acquired schema lock
0901 01:52:27.017556 (+ 3us) tablet.cc:660] Decoding operations
0901 01:52:27.017645 (+ 89us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:27.017657 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:27.017661 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0901 01:52:27.017712 (+ 51us) tablet.cc:699] Row locks acquired
0901 01:52:27.017714 (+ 2us) write_op.cc:260] PREPARE: finished
0901 01:52:27.017749 (+ 35us) write_op.cc:270] Start()
0901 01:52:27.017801 (+ 52us) write_op.cc:276] Timestamp: P: 1756691547017600 usec, L: 0
0901 01:52:27.017805 (+ 4us) op_driver.cc:348] REPLICATION: starting
0901 01:52:27.018019 (+ 214us) log.cc:844] Serialized 356 byte log entry
0901 01:52:27.021751 (+ 3732us) op_driver.cc:464] REPLICATION: finished
0901 01:52:27.021869 (+ 118us) write_op.cc:301] APPLY: starting
0901 01:52:27.021885 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:27.021927 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:27.021958 (+ 31us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:27.022006 (+ 48us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:27.022040 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0901 01:52:27.022048 (+ 8us) write_op.cc:312] APPLY: finished
0901 01:52:27.022133 (+ 85us) log.cc:844] Serialized 23 byte log entry
0901 01:52:27.022200 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:27.022218 (+ 18us) write_op.cc:454] Released schema lock
0901 01:52:27.022224 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":177568,"reported_tablets":205,"child_traces":[["op",{"apply.queue_time_us":47,"apply.run_cpu_time_us":465,"apply.run_wall_time_us":464,"num_ops":1,"prepare.queue_time_us":6355,"prepare.run_cpu_time_us":738,"prepare.run_wall_time_us":738,"raft.queue_time_us":387,"raft.run_cpu_time_us":254,"raft.run_wall_time_us":254,"replication_time_us":4950}],["op",{"apply.queue_time_us":70,"apply.run_cpu_time_us":427,"apply.run_wall_time_us":426,"num_ops":1,"prepare.queue_time_us":1719,"prepare.run_cpu_time_us":678,"prepare.run_wall_time_us":679,"raft.queue_time_us":460,"raft.run_cpu_time_us":191,"raft.run_wall_time_us":192,"replication_time_us":4355}],["op",{"apply.queue_time_us":71,"apply.run_cpu_time_us":410,"apply.run_wall_time_us":409,"num_ops":1,"prepare.queue_time_us":27912,"prepare.run_cpu_time_us":688,"prepare.run_wall_time_us":689,"raft.queue_time_us":403,"raft.run_cpu_time_us":216,"raft.run_wall_time_us":216,"replication_time_us":3913,"spinlock_wait_cycles":54144}]]}
W20250901 01:52:27.335810 858 leader_election.cc:336] T 2599f98abfdc4871b573edfc2ad97d22 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:27.336617 1150 tablet_bootstrap.cc:492] T 75345ea2a3dc40fc8c9f8090d8839d2d P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:27.336580 1483 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_5a72e7206f604795915f3a577d37f874" } modify_external_catalogs: true
I20250901 01:52:27.336932 1150 ts_tablet_manager.cc:1397] T 75345ea2a3dc40fc8c9f8090d8839d2d P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.082s user 0.003s sys 0.002s
I20250901 01:52:27.337750 1150 raft_consensus.cc:357] T 75345ea2a3dc40fc8c9f8090d8839d2d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.337960 1150 raft_consensus.cc:383] T 75345ea2a3dc40fc8c9f8090d8839d2d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:27.338061 1150 raft_consensus.cc:738] T 75345ea2a3dc40fc8c9f8090d8839d2d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:27.338307 1150 consensus_queue.cc:260] T 75345ea2a3dc40fc8c9f8090d8839d2d P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.338932 1150 ts_tablet_manager.cc:1428] T 75345ea2a3dc40fc8c9f8090d8839d2d P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250901 01:52:27.339215 1150 tablet_bootstrap.cc:492] T e2aa32450a584395ae07b49f76a7b26d P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:27.340916 1150 tablet_bootstrap.cc:654] T e2aa32450a584395ae07b49f76a7b26d P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:27.340530 1485 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_c4148158ed7b48a3946559ba978e72fd" } modify_external_catalogs: true
W20250901 01:52:27.342590 858 consensus_peers.cc:489] T b009cc95c819431ea9827a3e453b8da3 P 00c282f779c1477d920f24dc0dc16628 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250901 01:52:27.343906 2113 ts_tablet_manager.cc:1918] T 66a48a8ebe154b2cac89c234eaa378ff P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:27.344072 2113 log.cc:1199] T 66a48a8ebe154b2cac89c234eaa378ff P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/66a48a8ebe154b2cac89c234eaa378ff
E20250901 01:52:27.347168 1696 catalog_manager.cc:6205] Error selecting replicas for tablet be8dfbc466564842a2fdb4953eee939b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_75ba3ba70ca24a94920e7bce96190ff2'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:27.348851 1058 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f6a8b425968e49deaecfa3adacfd3f1f" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
E20250901 01:52:27.350610 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 96824e68bbf24b8287dbb683038ffaba, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_3c7c628f35fb4d02bb79a34995f12938'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:27.352085 2113 ts_tablet_manager.cc:1939] T 66a48a8ebe154b2cac89c234eaa378ff P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
E20250901 01:52:27.352936 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 7317eb7848ee44698038a4b404d4bf21, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b0e916c76a7a4766aa0887cc8691488f'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:27.352566 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ba88af177874544b764cc32420e7ac9" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:27.353044 1151 tablet_bootstrap.cc:492] T f6a8b425968e49deaecfa3adacfd3f1f P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:27.354188 1151 ts_tablet_manager.cc:1397] T f6a8b425968e49deaecfa3adacfd3f1f P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.039s user 0.008s sys 0.003s
I20250901 01:52:27.355037 1151 raft_consensus.cc:357] T f6a8b425968e49deaecfa3adacfd3f1f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.355190 1151 raft_consensus.cc:383] T f6a8b425968e49deaecfa3adacfd3f1f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:27.355233 1151 raft_consensus.cc:738] T f6a8b425968e49deaecfa3adacfd3f1f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:27.355295 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 70) took 2369 ms. Trace:
E20250901 01:52:27.354718 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9745582cebfe47b58c21d0191ab4cb9b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2f09538b6fbe401f909105decb56e1cb'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:27.355409 1151 consensus_queue.cc:260] T f6a8b425968e49deaecfa3adacfd3f1f P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.356202 1151 ts_tablet_manager.cc:1428] T f6a8b425968e49deaecfa3adacfd3f1f P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250901 01:52:27.356549 1151 tablet_bootstrap.cc:492] T f13b972e25214f168421439fe3943e8a P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:27.356564 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2599f98abfdc4871b573edfc2ad97d22" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:27.356760 1056 raft_consensus.cc:2466] T 2599f98abfdc4871b573edfc2ad97d22 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00c282f779c1477d920f24dc0dc16628 in term 0.
E20250901 01:52:27.356385 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 4abf9725be5d42f3893f07f0a9c7614c, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_8445fcd774874cadaf4acc3581eee417'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:27.358331 1151 tablet_bootstrap.cc:654] T f13b972e25214f168421439fe3943e8a P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
E20250901 01:52:27.357965 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 8a13346dcb3d4b6ab35c261829d50ae5, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_1bdd9041d74045cab6f23de11bf4dd09'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:27.355365 2113 rpcz_store.cc:276] 0901 01:52:24.985411 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:24.985469 (+ 58us) service_pool.cc:225] Handling call
0901 01:52:27.318931 (+2333462us) ts_tablet_manager.cc:1145] Deleting tablet 66a48a8ebe154b2cac89c234eaa378ff
0901 01:52:27.318939 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:27.323227 (+ 4288us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:27.343772 (+ 20545us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:27.355279 (+ 11507us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2333248}
I20250901 01:52:27.359874 1761 consensus_queue.cc:1035] T 15e4d972e8254cf899c3e0b71b6e2bd9 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
E20250901 01:52:27.359750 1696 catalog_manager.cc:6205] Error selecting replicas for tablet e2ed32b3292141278af28026528e96c2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_50ae3908a7944f70949c54be8fbf1564'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:27.360782 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 66a48a8ebe154b2cac89c234eaa378ff (table default.Table_ce16a1b94f714c2699ee6cd916bb0215 [id=551e2f9616d84026b96b6c22507f29de]) successfully deleted
I20250901 01:52:27.360862 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 71) took 2374 ms. Trace:
I20250901 01:52:27.360939 2113 rpcz_store.cc:276] 0901 01:52:24.986317 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:24.986367 (+ 50us) service_pool.cc:225] Handling call
0901 01:52:27.360460 (+2374093us) ts_tablet_manager.cc:1145] Deleting tablet f3962b471bdb49aea6e9fa95f1f9dfe2
0901 01:52:27.360694 (+ 234us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:27.360845 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2373888}
I20250901 01:52:27.361222 2113 tablet_replica.cc:331] T 3fda93767e494a4684978c8522b829dd P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:27.361382 2113 raft_consensus.cc:2241] T 3fda93767e494a4684978c8522b829dd P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:27.361524 2113 raft_consensus.cc:2270] T 3fda93767e494a4684978c8522b829dd P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:27.362017 2113 ts_tablet_manager.cc:1905] T 3fda93767e494a4684978c8522b829dd P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
E20250901 01:52:27.361418 1696 catalog_manager.cc:6205] Error selecting replicas for tablet a2ca69fabadd4ad69e1670cae7a0c503, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_32c4716e3aa24a87a6b1472984b255c4'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:27.362918 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 76ba642c16734cdf969dbd3130639fda, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_faf7ef12324f454396e5ae64d64ebe51'. Need at least 3 replicas, but only 2 tablet servers are available
W20250901 01:52:27.363512 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet f3962b471bdb49aea6e9fa95f1f9dfe2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f3962b471bdb49aea6e9fa95f1f9dfe2 already in progress: creating tablet
E20250901 01:52:27.364670 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f8f923ddcd2740bcb7e521f4f57c4034, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_5e65c5f068b84995a2d7a76fa65dda0f'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:27.366261 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 0813cc68d4b0488ba60a523e4479971b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d1302673d68f424ebb54b220911ce385'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:27.367950 1696 catalog_manager.cc:6205] Error selecting replicas for tablet b1ba0e19dc0b462e80c537bf2d183f5b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0e2cd82b00f1466eaef7f287402e644d'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:27.370252 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5f0e054f62de4cf6bc6f69ad6664b192, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ef6832d0845044ed9f165ca3541b4c3f'. Need at least 3 replicas, but only 2 tablet servers are available
W20250901 01:52:27.371251 859 leader_election.cc:343] T 6ba88af177874544b764cc32420e7ac9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:27.371403 859 leader_election.cc:304] T 6ba88af177874544b764cc32420e7ac9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:27.371835 2323 raft_consensus.cc:2747] T 6ba88af177874544b764cc32420e7ac9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:27.372388 2323 consensus_queue.cc:1035] T 7a2f6e2c7ee2402ea310c058e046416c P 00c282f779c1477d920f24dc0dc16628 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
E20250901 01:52:27.371791 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 787be682c0e44248a80d7037fb61dc6c, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7c12d66048a6470e8c227dc1d9f45fa4'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:27.373018 859 leader_election.cc:304] T 2599f98abfdc4871b573edfc2ad97d22 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:27.373404 2323 raft_consensus.cc:2802] T 2599f98abfdc4871b573edfc2ad97d22 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:27.373477 2323 raft_consensus.cc:491] T 2599f98abfdc4871b573edfc2ad97d22 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:27.373525 2323 raft_consensus.cc:3058] T 2599f98abfdc4871b573edfc2ad97d22 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:27.375398 1151 tablet_bootstrap.cc:492] T f13b972e25214f168421439fe3943e8a P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:27.375571 1151 ts_tablet_manager.cc:1397] T f13b972e25214f168421439fe3943e8a P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.001s
E20250901 01:52:27.375399 1696 catalog_manager.cc:6205] Error selecting replicas for tablet cc3b8736b9094d7e892c80da4d5b20f6, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_93afd37289414655980b22934695bd55'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:27.376355 1151 raft_consensus.cc:357] T f13b972e25214f168421439fe3943e8a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.376791 1151 raft_consensus.cc:383] T f13b972e25214f168421439fe3943e8a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:27.376904 1151 raft_consensus.cc:738] T f13b972e25214f168421439fe3943e8a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
E20250901 01:52:27.376672 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae9a4ad4fdfb4771ba23a91f7007447e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_5e84ae9a9c2a4084a1f9c5ffb4594de2'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:27.377200 1151 consensus_queue.cc:260] T f13b972e25214f168421439fe3943e8a P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.377914 1151 ts_tablet_manager.cc:1428] T f13b972e25214f168421439fe3943e8a P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250901 01:52:27.378238 1151 tablet_bootstrap.cc:492] T 500e3c33598b45a4afa983da7cb0845a P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:27.378486 2323 raft_consensus.cc:513] T 2599f98abfdc4871b573edfc2ad97d22 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
E20250901 01:52:27.378803 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9074543dea9949e7a085717b40913348, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_53e30205c5fd488eb3b48c685a159d04'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:27.380136 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 18498c9cb09141d08c1c5c0854b63ebd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77f4810bc83c4d948531da2a45a3792e'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:27.381515 1151 tablet_bootstrap.cc:654] T 500e3c33598b45a4afa983da7cb0845a P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
E20250901 01:52:27.381716 1696 catalog_manager.cc:6205] Error selecting replicas for tablet a6b539e160194da3a849d9e2f1ec9859, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_03fb48eaab9b42858ab7cefb722e43a8'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:27.383229 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 81319e373c9a4475abceeeedb507aabc, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_22d8d38d0b624912a0d8968fb764420e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:27.385146 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 6ce9521b925b4f97bb4a810ce77e147e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_9e04f976a93f4d30baccf90fa3622dfe'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:27.388136 2357 raft_consensus.cc:491] T 71941b815d6840478c5acc1cbb187cc9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:27.388556 2357 raft_consensus.cc:513] T 71941b815d6840478c5acc1cbb187cc9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.389071 2357 leader_election.cc:290] T 71941b815d6840478c5acc1cbb187cc9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:27.396505 858 leader_election.cc:336] T 71941b815d6840478c5acc1cbb187cc9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:27.385918 2113 ts_tablet_manager.cc:1918] T 3fda93767e494a4684978c8522b829dd P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:27.398332 2113 log.cc:1199] T 3fda93767e494a4684978c8522b829dd P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/3fda93767e494a4684978c8522b829dd
I20250901 01:52:27.398821 2113 ts_tablet_manager.cc:1939] T 3fda93767e494a4684978c8522b829dd P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:27.399744 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 72) took 2409 ms. Trace:
I20250901 01:52:27.400573 2113 rpcz_store.cc:276] 0901 01:52:24.989803 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:24.989844 (+ 41us) service_pool.cc:225] Handling call
0901 01:52:27.361181 (+2371337us) ts_tablet_manager.cc:1145] Deleting tablet 3fda93767e494a4684978c8522b829dd
0901 01:52:27.361191 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:27.382712 (+ 21521us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:27.385836 (+ 3124us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:27.399725 (+ 13889us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2371199}
I20250901 01:52:27.400486 1151 tablet_bootstrap.cc:492] T 500e3c33598b45a4afa983da7cb0845a P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:27.401129 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 3fda93767e494a4684978c8522b829dd (table default.Table_655d2fe6f5f044d2ac3c542497704343 [id=998e1151707c436bae86cdac62918ad1]) successfully deleted
I20250901 01:52:27.401175 1151 ts_tablet_manager.cc:1397] T 500e3c33598b45a4afa983da7cb0845a P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.002s
I20250901 01:52:27.401998 1151 raft_consensus.cc:357] T 500e3c33598b45a4afa983da7cb0845a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.402233 1151 raft_consensus.cc:383] T 500e3c33598b45a4afa983da7cb0845a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:27.402211 2301 tablet.cc:1722] T 42c1156e8dd84c6fa1396b46de13eb6f P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:27.402328 1151 raft_consensus.cc:738] T 500e3c33598b45a4afa983da7cb0845a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:27.402493 1151 consensus_queue.cc:260] T 500e3c33598b45a4afa983da7cb0845a P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.402853 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 73) took 2411 ms. Trace:
I20250901 01:52:27.403327 2113 rpcz_store.cc:276] 0901 01:52:24.991182 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:24.991223 (+ 41us) service_pool.cc:225] Handling call
0901 01:52:27.402719 (+2411496us) ts_tablet_manager.cc:1145] Deleting tablet b85d64e02cc646e380581ac9f59dee9e
0901 01:52:27.402731 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:27.402838 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2411286}
W20250901 01:52:27.403883 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet b85d64e02cc646e380581ac9f59dee9e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b85d64e02cc646e380581ac9f59dee9e already in progress: creating tablet
I20250901 01:52:27.403266 1151 ts_tablet_manager.cc:1428] T 500e3c33598b45a4afa983da7cb0845a P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250901 01:52:27.404347 1151 tablet_bootstrap.cc:492] T f3962b471bdb49aea6e9fa95f1f9dfe2 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:27.405853 1151 tablet_bootstrap.cc:654] T f3962b471bdb49aea6e9fa95f1f9dfe2 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:27.407785 2357 raft_consensus.cc:491] T 482a9a2cb81f448bb2823892e35920d4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:27.407927 2357 raft_consensus.cc:513] T 482a9a2cb81f448bb2823892e35920d4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.408744 2357 leader_election.cc:290] T 482a9a2cb81f448bb2823892e35920d4 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:27.411965 858 leader_election.cc:336] T 482a9a2cb81f448bb2823892e35920d4 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:27.410118 2301 tablet.cc:1722] T 616ff34e8d314e0aa0e11ce60be11789 P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:27.412649 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 74) took 2420 ms. Trace:
I20250901 01:52:27.417337 2113 rpcz_store.cc:276] 0901 01:52:24.992508 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:24.992551 (+ 43us) service_pool.cc:225] Handling call
0901 01:52:27.412514 (+2419963us) ts_tablet_manager.cc:1145] Deleting tablet 088ef4861dc14f81ae470c3aaee5b4e1
0901 01:52:27.412524 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:27.412631 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2419806}
I20250901 01:52:27.417753 2323 leader_election.cc:290] T 2599f98abfdc4871b573edfc2ad97d22 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:27.417925 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 75) took 2409 ms. Trace:
I20250901 01:52:27.418061 2113 rpcz_store.cc:276] 0901 01:52:25.008606 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.008664 (+ 58us) service_pool.cc:225] Handling call
0901 01:52:27.417732 (+2409068us) ts_tablet_manager.cc:1145] Deleting tablet e2aa32450a584395ae07b49f76a7b26d
0901 01:52:27.417741 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:27.417909 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2408879}
W20250901 01:52:27.418298 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 088ef4861dc14f81ae470c3aaee5b4e1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 088ef4861dc14f81ae470c3aaee5b4e1 already in progress: creating tablet
I20250901 01:52:27.418467 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 76) took 2408 ms. Trace:
I20250901 01:52:27.418555 2113 rpcz_store.cc:276] 0901 01:52:25.010124 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.010172 (+ 48us) service_pool.cc:225] Handling call
0901 01:52:27.418372 (+2408200us) ts_tablet_manager.cc:1145] Deleting tablet e8e7d2a72f704f52a8f8d46911a8a9e4
0901 01:52:27.418381 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:27.418454 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2408043}
W20250901 01:52:27.419041 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet e2aa32450a584395ae07b49f76a7b26d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e2aa32450a584395ae07b49f76a7b26d already in progress: creating tablet
I20250901 01:52:27.422040 2323 raft_consensus.cc:491] T cb7d7e1796c0456f8de98fb812b9e374 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:27.422223 2323 raft_consensus.cc:513] T cb7d7e1796c0456f8de98fb812b9e374 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.422987 2323 leader_election.cc:290] T cb7d7e1796c0456f8de98fb812b9e374 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:27.424584 858 leader_election.cc:336] T 2599f98abfdc4871b573edfc2ad97d22 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:27.424763 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet e8e7d2a72f704f52a8f8d46911a8a9e4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e8e7d2a72f704f52a8f8d46911a8a9e4 already in progress: creating tablet
I20250901 01:52:27.425241 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 77) took 2405 ms. Trace:
W20250901 01:52:27.425428 858 leader_election.cc:336] T cb7d7e1796c0456f8de98fb812b9e374 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:27.425387 2113 rpcz_store.cc:276] 0901 01:52:25.019460 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.019657 (+ 197us) service_pool.cc:225] Handling call
0901 01:52:27.425087 (+2405430us) ts_tablet_manager.cc:1145] Deleting tablet 75345ea2a3dc40fc8c9f8090d8839d2d
0901 01:52:27.425100 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:27.425216 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2405268}
W20250901 01:52:27.425895 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 75345ea2a3dc40fc8c9f8090d8839d2d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 75345ea2a3dc40fc8c9f8090d8839d2d already in progress: creating tablet
I20250901 01:52:27.426188 2113 tablet_replica.cc:331] T 5a8c21aee9f84216b6f90538a5bf4c17 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:27.426461 2113 raft_consensus.cc:2241] T 5a8c21aee9f84216b6f90538a5bf4c17 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:27.426645 2113 raft_consensus.cc:2270] T 5a8c21aee9f84216b6f90538a5bf4c17 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:27.427249 2113 ts_tablet_manager.cc:1905] T 5a8c21aee9f84216b6f90538a5bf4c17 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:27.436967 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "71941b815d6840478c5acc1cbb187cc9" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:27.437172 1057 raft_consensus.cc:2466] T 71941b815d6840478c5acc1cbb187cc9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00c282f779c1477d920f24dc0dc16628 in term 0.
I20250901 01:52:27.438562 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "482a9a2cb81f448bb2823892e35920d4" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:27.445763 2113 ts_tablet_manager.cc:1918] T 5a8c21aee9f84216b6f90538a5bf4c17 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:27.449062 2113 log.cc:1199] T 5a8c21aee9f84216b6f90538a5bf4c17 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/5a8c21aee9f84216b6f90538a5bf4c17
I20250901 01:52:27.449616 2113 ts_tablet_manager.cc:1939] T 5a8c21aee9f84216b6f90538a5bf4c17 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:27.450852 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 78) took 2431 ms. Trace:
I20250901 01:52:27.451861 2113 rpcz_store.cc:276] 0901 01:52:25.019606 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.019848 (+ 242us) service_pool.cc:225] Handling call
0901 01:52:27.426134 (+2406286us) ts_tablet_manager.cc:1145] Deleting tablet 5a8c21aee9f84216b6f90538a5bf4c17
0901 01:52:27.426146 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:27.432568 (+ 6422us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:27.445657 (+ 13089us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:27.450833 (+ 5176us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2406172}
I20250901 01:52:27.452474 1150 tablet_bootstrap.cc:492] T e2aa32450a584395ae07b49f76a7b26d P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:27.452683 1150 ts_tablet_manager.cc:1397] T e2aa32450a584395ae07b49f76a7b26d P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.114s user 0.001s sys 0.004s
I20250901 01:52:27.452651 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb7d7e1796c0456f8de98fb812b9e374" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:27.451949 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2599f98abfdc4871b573edfc2ad97d22" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58"
I20250901 01:52:27.452986 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 81) took 2413 ms. Trace:
I20250901 01:52:27.453042 1056 raft_consensus.cc:3058] T 2599f98abfdc4871b573edfc2ad97d22 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:27.454319 1150 raft_consensus.cc:357] T e2aa32450a584395ae07b49f76a7b26d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.454512 2113 rpcz_store.cc:276] 0901 01:52:25.039223 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.039283 (+ 60us) service_pool.cc:225] Handling call
0901 01:52:27.452856 (+2413573us) ts_tablet_manager.cc:1145] Deleting tablet 50472a7aa2f04e0fbd65ac6c86c34b3e
0901 01:52:27.452865 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:27.452961 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2413397}
I20250901 01:52:27.454706 1150 raft_consensus.cc:383] T e2aa32450a584395ae07b49f76a7b26d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:27.454831 1150 raft_consensus.cc:738] T e2aa32450a584395ae07b49f76a7b26d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:27.455224 1150 consensus_queue.cc:260] T e2aa32450a584395ae07b49f76a7b26d P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:27.455338 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 50472a7aa2f04e0fbd65ac6c86c34b3e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 50472a7aa2f04e0fbd65ac6c86c34b3e already in progress: creating tablet
I20250901 01:52:27.455660 2113 tablet_replica.cc:331] T b541a6e14f114ab29a864cca11806316 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:27.456037 1150 ts_tablet_manager.cc:1428] T e2aa32450a584395ae07b49f76a7b26d P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250901 01:52:27.456494 1150 tablet_bootstrap.cc:492] T 8b9d79a25452410891c09fd3f7c4be6d P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:27.456657 2113 raft_consensus.cc:2241] T b541a6e14f114ab29a864cca11806316 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:27.457087 2113 raft_consensus.cc:2270] T b541a6e14f114ab29a864cca11806316 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:27.457237 1056 raft_consensus.cc:2466] T 2599f98abfdc4871b573edfc2ad97d22 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c282f779c1477d920f24dc0dc16628 in term 1.
I20250901 01:52:27.457943 2113 ts_tablet_manager.cc:1905] T b541a6e14f114ab29a864cca11806316 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:27.464790 1150 tablet_bootstrap.cc:654] T 8b9d79a25452410891c09fd3f7c4be6d P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:27.463855 992 consensus_peers.cc:489] T e6c10426f60f43e8bde69d9b1b9c8eb6 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250901 01:52:27.465552 1761 raft_consensus.cc:491] T 89d40f0331114490b51de3ba6d773571 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:27.465786 1761 raft_consensus.cc:513] T 89d40f0331114490b51de3ba6d773571 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.466323 1761 leader_election.cc:290] T 89d40f0331114490b51de3ba6d773571 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:27.466885 1483 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_5a72e7206f604795915f3a577d37f874" } modify_external_catalogs: true
I20250901 01:52:27.476743 2323 raft_consensus.cc:491] T 3eba54e5ddd14c518f4cbd539268ae37 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:27.476886 2323 raft_consensus.cc:513] T 3eba54e5ddd14c518f4cbd539268ae37 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.477453 2323 leader_election.cc:290] T 3eba54e5ddd14c518f4cbd539268ae37 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:27.484171 858 leader_election.cc:336] T 3eba54e5ddd14c518f4cbd539268ae37 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:27.486923 2323 raft_consensus.cc:491] T b85d64e02cc646e380581ac9f59dee9e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:27.487051 2323 raft_consensus.cc:513] T b85d64e02cc646e380581ac9f59dee9e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.487555 2323 leader_election.cc:290] T b85d64e02cc646e380581ac9f59dee9e P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:27.493484 858 leader_election.cc:336] T b85d64e02cc646e380581ac9f59dee9e P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:27.494048 1485 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_c4148158ed7b48a3946559ba978e72fd" } modify_external_catalogs: true
I20250901 01:52:27.499819 2323 raft_consensus.cc:491] T 75345ea2a3dc40fc8c9f8090d8839d2d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:27.499941 2323 raft_consensus.cc:513] T 75345ea2a3dc40fc8c9f8090d8839d2d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.500501 2323 leader_election.cc:290] T 75345ea2a3dc40fc8c9f8090d8839d2d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:27.502010 2323 raft_consensus.cc:491] T b164b4a67a9143ceaff69b0ce49d63fe P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:27.502130 2323 raft_consensus.cc:513] T b164b4a67a9143ceaff69b0ce49d63fe P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:27.502624 2323 leader_election.cc:290] T b164b4a67a9143ceaff69b0ce49d63fe P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:27.502946 858 leader_election.cc:336] T 75345ea2a3dc40fc8c9f8090d8839d2d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:27.504359 858 leader_election.cc:336] T b164b4a67a9143ceaff69b0ce49d63fe P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:27.509936 1150 tablet_bootstrap.cc:492] T 8b9d79a25452410891c09fd3f7c4be6d P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:27.510105 1150 ts_tablet_manager.cc:1397] T 8b9d79a25452410891c09fd3f7c4be6d P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.054s user 0.006s sys 0.000s
I20250901 01:52:27.511740 1150 raft_consensus.cc:357] T 8b9d79a25452410891c09fd3f7c4be6d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.511933 1150 raft_consensus.cc:383] T 8b9d79a25452410891c09fd3f7c4be6d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:27.512005 1150 raft_consensus.cc:738] T 8b9d79a25452410891c09fd3f7c4be6d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:27.512218 1150 consensus_queue.cc:260] T 8b9d79a25452410891c09fd3f7c4be6d P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:27.515494 858 consensus_peers.cc:489] T 7a2f6e2c7ee2402ea310c058e046416c P 00c282f779c1477d920f24dc0dc16628 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:27.519352 1696 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 81319e373c9a4475abceeeedb507aabc
I20250901 01:52:27.525712 1058 raft_consensus.cc:2466] T f6a8b425968e49deaecfa3adacfd3f1f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00c282f779c1477d920f24dc0dc16628 in term 0.
I20250901 01:52:27.526273 1150 ts_tablet_manager.cc:1428] T 8b9d79a25452410891c09fd3f7c4be6d P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.016s user 0.000s sys 0.001s
I20250901 01:52:27.526674 1150 tablet_bootstrap.cc:492] T e8e7d2a72f704f52a8f8d46911a8a9e4 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:27.527393 2113 ts_tablet_manager.cc:1918] T b541a6e14f114ab29a864cca11806316 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:27.530162 1150 tablet_bootstrap.cc:654] T e8e7d2a72f704f52a8f8d46911a8a9e4 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:27.530143 2306 raft_consensus.cc:491] T f13b972e25214f168421439fe3943e8a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:27.530296 2306 raft_consensus.cc:513] T f13b972e25214f168421439fe3943e8a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.530897 2306 leader_election.cc:290] T f13b972e25214f168421439fe3943e8a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:27.531159 2113 log.cc:1199] T b541a6e14f114ab29a864cca11806316 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/b541a6e14f114ab29a864cca11806316
I20250901 01:52:27.531517 2113 ts_tablet_manager.cc:1939] T b541a6e14f114ab29a864cca11806316 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:27.533385 2306 raft_consensus.cc:491] T 8ed6616ce54548ba83ba299c4ecf3c38 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:27.533494 2306 raft_consensus.cc:513] T 8ed6616ce54548ba83ba299c4ecf3c38 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.533958 2306 leader_election.cc:290] T 8ed6616ce54548ba83ba299c4ecf3c38 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:27.534583 992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:40731, user_credentials={real_user=slave}} blocked reactor thread for 68071.4us
I20250901 01:52:27.534762 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 79) took 2495 ms. Trace:
I20250901 01:52:27.534919 2113 rpcz_store.cc:276] 0901 01:52:25.039048 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.039493 (+ 445us) service_pool.cc:225] Handling call
0901 01:52:27.455615 (+2416122us) ts_tablet_manager.cc:1145] Deleting tablet b541a6e14f114ab29a864cca11806316
0901 01:52:27.455627 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:27.522740 (+ 67113us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:27.527295 (+ 4555us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:27.534743 (+ 7448us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2415985}
W20250901 01:52:27.536441 858 consensus_peers.cc:489] T 8ca5cf4e99014c0d84a3acad4ef713af P 00c282f779c1477d920f24dc0dc16628 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:27.537173 2113 tablet_replica.cc:331] T b633b461dbf34e13bc6e1cc08e6d20c1 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:27.537402 2113 raft_consensus.cc:2241] T b633b461dbf34e13bc6e1cc08e6d20c1 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:27.537592 2113 raft_consensus.cc:2270] T b633b461dbf34e13bc6e1cc08e6d20c1 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:27.538151 2113 ts_tablet_manager.cc:1905] T b633b461dbf34e13bc6e1cc08e6d20c1 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:27.548336 1150 tablet_bootstrap.cc:492] T e8e7d2a72f704f52a8f8d46911a8a9e4 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:27.548569 1150 ts_tablet_manager.cc:1397] T e8e7d2a72f704f52a8f8d46911a8a9e4 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.003s
I20250901 01:52:27.549218 1151 tablet_bootstrap.cc:492] T f3962b471bdb49aea6e9fa95f1f9dfe2 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:27.549404 1151 ts_tablet_manager.cc:1397] T f3962b471bdb49aea6e9fa95f1f9dfe2 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.145s user 0.027s sys 0.014s
I20250901 01:52:27.549371 1150 raft_consensus.cc:357] T e8e7d2a72f704f52a8f8d46911a8a9e4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.549608 1150 raft_consensus.cc:383] T e8e7d2a72f704f52a8f8d46911a8a9e4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:27.549708 1150 raft_consensus.cc:738] T e8e7d2a72f704f52a8f8d46911a8a9e4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:27.549973 1150 consensus_queue.cc:260] T e8e7d2a72f704f52a8f8d46911a8a9e4 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:27.550172 992 consensus_peers.cc:489] T ad45c90ba0da43c9860745e534664a98 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:27.550144 1151 raft_consensus.cc:357] T f3962b471bdb49aea6e9fa95f1f9dfe2 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:27.550705 1150 ts_tablet_manager.cc:1428] T e8e7d2a72f704f52a8f8d46911a8a9e4 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250901 01:52:27.550295 1151 raft_consensus.cc:383] T f3962b471bdb49aea6e9fa95f1f9dfe2 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:27.552227 1151 raft_consensus.cc:738] T f3962b471bdb49aea6e9fa95f1f9dfe2 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:27.552217 859 leader_election.cc:304] T 71941b815d6840478c5acc1cbb187cc9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
W20250901 01:52:27.552580 859 leader_election.cc:343] T 482a9a2cb81f448bb2823892e35920d4 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:27.552536 1151 consensus_queue.cc:260] T f3962b471bdb49aea6e9fa95f1f9dfe2 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:27.552680 859 leader_election.cc:304] T 482a9a2cb81f448bb2823892e35920d4 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:27.553018 1151 ts_tablet_manager.cc:1428] T f3962b471bdb49aea6e9fa95f1f9dfe2 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
W20250901 01:52:27.553078 859 leader_election.cc:343] T cb7d7e1796c0456f8de98fb812b9e374 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:27.553158 859 leader_election.cc:304] T cb7d7e1796c0456f8de98fb812b9e374 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:27.553283 1151 tablet_bootstrap.cc:492] T 3d90c6ef06d1436d8e2bb1333e7d9985 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:27.554535 2357 raft_consensus.cc:2802] T 71941b815d6840478c5acc1cbb187cc9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:27.554602 2357 raft_consensus.cc:491] T 71941b815d6840478c5acc1cbb187cc9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:27.554654 2357 raft_consensus.cc:3058] T 71941b815d6840478c5acc1cbb187cc9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:27.557053 2357 raft_consensus.cc:513] T 71941b815d6840478c5acc1cbb187cc9 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.557632 2357 leader_election.cc:290] T 71941b815d6840478c5acc1cbb187cc9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:27.557776 2357 raft_consensus.cc:2747] T 482a9a2cb81f448bb2823892e35920d4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:27.560382 2307 raft_consensus.cc:491] T b11e8f77243e4764b96548d1132b8e5a P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00c282f779c1477d920f24dc0dc16628)
I20250901 01:52:27.560542 2307 raft_consensus.cc:513] T b11e8f77243e4764b96548d1132b8e5a P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:27.561024 2307 leader_election.cc:290] T b11e8f77243e4764b96548d1132b8e5a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:27.561419 2307 consensus_queue.cc:1035] T ad45c90ba0da43c9860745e534664a98 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:27.558480 2123 raft_consensus.cc:2747] T cb7d7e1796c0456f8de98fb812b9e374 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:27.559527 2354 tablet.cc:1722] T 616ff34e8d314e0aa0e11ce60be11789 P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:27.564395 2113 ts_tablet_manager.cc:1918] T b633b461dbf34e13bc6e1cc08e6d20c1 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:27.564525 2113 log.cc:1199] T b633b461dbf34e13bc6e1cc08e6d20c1 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/b633b461dbf34e13bc6e1cc08e6d20c1
I20250901 01:52:27.564894 2113 ts_tablet_manager.cc:1939] T b633b461dbf34e13bc6e1cc08e6d20c1 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:27.565793 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 80) took 2526 ms. Trace:
I20250901 01:52:27.565883 2113 rpcz_store.cc:276] 0901 01:52:25.039127 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.039657 (+ 530us) service_pool.cc:225] Handling call
0901 01:52:27.537119 (+2497462us) ts_tablet_manager.cc:1145] Deleting tablet b633b461dbf34e13bc6e1cc08e6d20c1
0901 01:52:27.537132 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:27.543930 (+ 6798us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:27.564278 (+ 20348us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:27.565760 (+ 1482us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2497334}
I20250901 01:52:27.566251 2118 ts_tablet_manager.cc:1918] T bb0cf5a6a27045a08df6845d7473184d P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:27.566454 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet b633b461dbf34e13bc6e1cc08e6d20c1 (table default.Table_d1302673d68f424ebb54b220911ce385 [id=453153553bea48319e132857f628d48a]) successfully deleted
I20250901 01:52:27.566809 1484 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 81ce448b148447ddbc41875d863e0518 (table default.Table_6e461d9cb0ae4dd095975bbc21533794 [id=49673fa73f8c436e9b8026efc4f3cbbc]). Expected version 1 got 0
I20250901 01:52:27.567128 1484 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet b164b4a67a9143ceaff69b0ce49d63fe (table default.Table_da2c70a8f2fe49a0a8184e88c0a9e0ea [id=c1b1e9e3876244f68d31c63ce7bdebf1]). Expected version 1 got 0
I20250901 01:52:27.566996 2118 log.cc:1199] T bb0cf5a6a27045a08df6845d7473184d P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/bb0cf5a6a27045a08df6845d7473184d
I20250901 01:52:27.567376 1484 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 9fbd44641d9e43c3b592e388e4adf350 (table default.Table_2fbca8463d6546c9b8f5b018fc60251d [id=6fdd3d817dc74551923021bd716ba58b]). Expected version 1 got 0
I20250901 01:52:27.567649 1484 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet d1f2d98712ff4ea6aec665512de068bd (table default.Table_82d91a1ba2eb4265896824028dec8d3a [id=3f7624d2901445e39155c12fa68e34e9]). Expected version 1 got 0
I20250901 01:52:27.567981 1484 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 1495de70f0844f82a0eea99c059f076b (table default.Table_af5338b09bbc4c879c7f4f573c32fd8d [id=70a5ce71d16b47d4bc6dae97be4548e7]). Expected version 3 got 0
I20250901 01:52:27.568284 1484 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 14b466b2a5a249aa841f1b8b06301edc (table default.Table_927955c1048f4e9c84b5b3a113a473f7 [id=78cd71321e9a480ca2d7d88327f157ee]). Expected version 1 got 0
I20250901 01:52:27.568536 1484 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet ed1b41340ce742a88c3e4fd85cd25ca6 (table default.Table_139f6d174d794a6e9a618e90587d0c0d [id=0baa4993b72e4a8d94ee3059c9d2db6c]). Expected version 1 got 0
I20250901 01:52:27.569265 2118 ts_tablet_manager.cc:1939] T bb0cf5a6a27045a08df6845d7473184d P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:27.569936 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8ed6616ce54548ba83ba299c4ecf3c38" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:27.570116 915 raft_consensus.cc:2466] T 8ed6616ce54548ba83ba299c4ecf3c38 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:27.570742 1151 tablet_bootstrap.cc:654] T 3d90c6ef06d1436d8e2bb1333e7d9985 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:27.570657 918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f13b972e25214f168421439fe3943e8a" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:27.570860 918 raft_consensus.cc:2466] T f13b972e25214f168421439fe3943e8a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:27.571229 922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "89d40f0331114490b51de3ba6d773571" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:27.571380 922 raft_consensus.cc:2466] T 89d40f0331114490b51de3ba6d773571 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:27.571854 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 54) took 2368 ms. Trace:
I20250901 01:52:27.571991 2118 rpcz_store.cc:276] 0901 01:52:25.202989 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.203024 (+ 35us) service_pool.cc:225] Handling call
0901 01:52:27.316504 (+2113480us) ts_tablet_manager.cc:1145] Deleting tablet bb0cf5a6a27045a08df6845d7473184d
0901 01:52:27.316513 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:27.332367 (+ 15854us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:27.565857 (+233490us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:27.571817 (+ 5960us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2113295}
W20250901 01:52:27.572710 858 leader_election.cc:336] T 71941b815d6840478c5acc1cbb187cc9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:27.573673 2113 tablet_replica.cc:331] T 20f7bfab18134e079e91c52c89468cc3 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:27.573455 2118 tablet_replica.cc:331] T a54af99bc10045e8b0b6868e5aed98b4 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:27.576097 2118 raft_consensus.cc:2241] T a54af99bc10045e8b0b6868e5aed98b4 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:27.577292 2118 raft_consensus.cc:2270] T a54af99bc10045e8b0b6868e5aed98b4 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:27.579448 2118 ts_tablet_manager.cc:1905] T a54af99bc10045e8b0b6868e5aed98b4 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:27.582679 992 leader_election.cc:304] T 8ed6616ce54548ba83ba299c4ecf3c38 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:27.582904 2306 raft_consensus.cc:2802] T 8ed6616ce54548ba83ba299c4ecf3c38 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:27.582983 2306 raft_consensus.cc:491] T 8ed6616ce54548ba83ba299c4ecf3c38 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:27.583014 992 leader_election.cc:304] T f13b972e25214f168421439fe3943e8a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:27.583047 2306 raft_consensus.cc:3058] T 8ed6616ce54548ba83ba299c4ecf3c38 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:27.583207 1993 raft_consensus.cc:2802] T f13b972e25214f168421439fe3943e8a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:27.583271 1993 raft_consensus.cc:491] T f13b972e25214f168421439fe3943e8a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:27.583253 992 leader_election.cc:304] T 89d40f0331114490b51de3ba6d773571 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:27.583385 1993 raft_consensus.cc:3058] T f13b972e25214f168421439fe3943e8a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:27.583516 2113 raft_consensus.cc:2241] T 20f7bfab18134e079e91c52c89468cc3 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:27.573078 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet bb0cf5a6a27045a08df6845d7473184d (table default.Table_b23a4778615d4597950ce1709ab00d37 [id=bf59a9514b7d49c3b3c1e9c6064ef369]) successfully deleted
I20250901 01:52:27.583839 2113 raft_consensus.cc:2270] T 20f7bfab18134e079e91c52c89468cc3 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:27.584065 2409 raft_consensus.cc:2802] T 89d40f0331114490b51de3ba6d773571 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:27.584125 2409 raft_consensus.cc:491] T 89d40f0331114490b51de3ba6d773571 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:27.584213 2409 raft_consensus.cc:3058] T 89d40f0331114490b51de3ba6d773571 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:27.584379 1484 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet d0c3c363dda444519a17dca63ad9caa7 (table default.Table_42422c44ca254d8ca989134c2433391f [id=8c1b263824ec4c15a5ec424364c78895]). Expected version 3 got 0
I20250901 01:52:27.584947 1484 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 615b2306fca84003b264da9d5f95ddab (table default.Table_302f6e7cf97d46b9a621eb852e6aefe0 [id=cecdd8d6ca6d445c9fd82decd2a95366]). Expected version 2 got 0
I20250901 01:52:27.585093 1484 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet fecf7d538b4c4c619182d5eccb44dd8b (table default.Table_1b5729d9768546fe99eaafc4e2604f42 [id=02d0056abe2146ac9af81ebf4447b99d]). Expected version 1 got 0
I20250901 01:52:27.585462 1484 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 7954393c8c1c4457ab8bb723c03a9c8b (table default.Table_b2083370690443c6aa50c453bd29cb9a [id=543f2074598d4c29972e564be1123fca]). Expected version 4 got 0
I20250901 01:52:27.585705 1484 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 89d40f0331114490b51de3ba6d773571 (table default.Table_4dacb3b1d2e94e42ac8937eb6eb58375 [id=8ee0aa1ba24042dd8ded004644913b9c]). Expected version 1 got 0
I20250901 01:52:27.585788 1993 raft_consensus.cc:513] T f13b972e25214f168421439fe3943e8a P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.585871 1484 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet ae21b9ae2e3b472eb67fb5f96388a50f (table default.Table_046289f393ff432aae0b033bdec1fb17 [id=db57b18a9ce24e068b362f5fbe940531]). Expected version 1 got 0
I20250901 01:52:27.586058 1484 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 858218c4427c41e980b2b5c969a5af40 (table default.Table_6d47266af0ed433eb1f2abbc81d18e69 [id=264cc469a73841e79538843479a165ab]). Expected version 1 got 0
I20250901 01:52:27.586185 1484 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 3618a3eb4f174c43805ce7dc0e5315cb (table default.Table_83f2c0acedbc4c6982caf5313646e90b [id=31a2e50c32644321825da40176127015]). Expected version 1 got 0
I20250901 01:52:27.586294 1993 leader_election.cc:290] T f13b972e25214f168421439fe3943e8a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:27.586509 1484 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet f80f13db501c42c7950499df8fcb6fc8 (table default.Table_ce86974fd8974de9a5a737954fac2ad7 [id=ff772d20b287476f94716379b86d2a71]). Expected version 1 got 0
I20250901 01:52:27.585743 2306 raft_consensus.cc:513] T 8ed6616ce54548ba83ba299c4ecf3c38 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.586998 2409 raft_consensus.cc:513] T 89d40f0331114490b51de3ba6d773571 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.587141 2306 leader_election.cc:290] T 8ed6616ce54548ba83ba299c4ecf3c38 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:27.587628 2409 leader_election.cc:290] T 89d40f0331114490b51de3ba6d773571 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:27.589165 1484 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 508406601c994e42a5c5b1f665e74957 (table default.Table_345b7f9fe8214aa6878f20a166125364 [id=4aabc328f6374ec5b1dea26f778208b2]). Expected version 1 got 0
I20250901 01:52:27.591184 1484 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 6c0abbfc8d904dce989ce51961e4237d (table default.Table_3f9422b08b0349b9a5466218f0fe1b06 [id=f0dfb3811e744bd4ba6d59195cebde6f]). Expected version 2 got 0
I20250901 01:52:27.550992 1150 tablet_bootstrap.cc:492] T 246b33338de647689d7404c70676e76f P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:27.594029 1484 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 44af4764d4f64e44aec84ccb84be6f5f (table default.Table_9896b980158948339a0b91f5882ed56b [id=a541b55be0dc420a8353ef4793b925b5]). Expected version 1 got 0
I20250901 01:52:27.594691 1484 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 93bcac1fd95d44dc927abfca81692921 (table default.Table_b8f8ee74648c406eaf589d3ab80934db [id=b8f69313439543be9c11eea88dfa69c5]). Expected version 1 got 0
I20250901 01:52:27.595829 1150 tablet_bootstrap.cc:654] T 246b33338de647689d7404c70676e76f P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:27.597867 1484 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet e9f1f892658b4736859ff507536920a1 (table default.Table_8825dd28afbe464193eb1b668cc9326a [id=9aa62cb634314a0a849612c54c1220d9]). Expected version 1 got 0
I20250901 01:52:27.598451 1484 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet d000ba05e12843b4925e622ab55471c0 (table default.Table_e0902dfbdd984f85b3fdac8d283731f7 [id=b43a92ed46164d199b445287ef039bc6]). Expected version 1 got 0
I20250901 01:52:27.599009 1484 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 7f46e9ddadaf4ff986d8c991178daa0c (table default.Table_9978497447394e5694ae928412634ecb [id=ea928501fe954cd0b81d38add4ef7128]). Expected version 2 got 0
I20250901 01:52:27.599366 2113 ts_tablet_manager.cc:1905] T 20f7bfab18134e079e91c52c89468cc3 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:27.599831 1484 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet aa424ac6ef1441e6b1cc2867b1c48186 (table default.Table_bbbde49530e34a76b7cb17988ac6baca [id=76928d8caa2f4134a014656cf2cc871a]). Expected version 1 got 0
I20250901 01:52:27.600392 1484 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 5f5396045979450193d461ba1334a5c9 (table default.Table_a89cc8e48563460baabc34c821ceae35 [id=9c3930e9fcfe4b49ba4bea67a2b0ad07]). Expected version 1 got 0
I20250901 01:52:27.601150 1484 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 33f637f6a8134215bb1995ec38440073 (table default.Table_3d5afc22c4e14d6990aea31564dfde20 [id=557afe29976d40f6bfb6ae4ca348f887]). Expected version 2 got 0
I20250901 01:52:27.602150 1484 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 61ddcf86c9f34726939cc116434f0822 (table default.Table_63d0f2ead9d946d39ce40162ebd769d6 [id=f9dcf4937b1143248bfa9d88053fce17]). Expected version 3 got 0
I20250901 01:52:27.602984 1484 catalog_manager.cc:5582] T 8ca5cf4e99014c0d84a3acad4ef713af P 00c282f779c1477d920f24dc0dc16628 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00c282f779c1477d920f24dc0dc16628 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "00c282f779c1477d920f24dc0dc16628" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: HEALTHY } } }
W20250901 01:52:27.603631 859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:39447, user_credentials={real_user=slave}} blocked reactor thread for 50551.1us
I20250901 01:52:27.604038 859 leader_election.cc:304] T 2599f98abfdc4871b573edfc2ad97d22 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:27.604799 1484 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 8ca5cf4e99014c0d84a3acad4ef713af (table default.Table_ed6fd284be6f4ae3a008e21e7ce17930 [id=303e76c918bc4c39aa47a4d24849a358]). Expected version 1 got 0
I20250901 01:52:27.605397 1484 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 7304937048ca48a7bd153f3cdcfaeff0 (table default.Table_1e59568bdcd14662ae9d5e98293d74bd [id=8051adf4041f4a2686a0b8e3850ef29c]). Expected version 1 got 0
I20250901 01:52:27.605461 859 leader_election.cc:304] T f6a8b425968e49deaecfa3adacfd3f1f P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:27.606671 1484 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet ad45c90ba0da43c9860745e534664a98 (table default.Table_1cd3f3fb6b244b3f8796bfca9f7622c8 [id=6fd739b0e725424ca844dbb652ea8028]). Expected version 2 got 0
I20250901 01:52:27.605867 2407 raft_consensus.cc:2802] T 2599f98abfdc4871b573edfc2ad97d22 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:27.609082 2357 raft_consensus.cc:2802] T f6a8b425968e49deaecfa3adacfd3f1f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:27.609153 2357 raft_consensus.cc:491] T f6a8b425968e49deaecfa3adacfd3f1f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:27.609207 2357 raft_consensus.cc:3058] T f6a8b425968e49deaecfa3adacfd3f1f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:27.609220 1150 tablet_bootstrap.cc:492] T 246b33338de647689d7404c70676e76f P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:27.609338 1150 ts_tablet_manager.cc:1397] T 246b33338de647689d7404c70676e76f P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.058s user 0.003s sys 0.000s
I20250901 01:52:27.610075 1150 raft_consensus.cc:357] T 246b33338de647689d7404c70676e76f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.610244 1150 raft_consensus.cc:383] T 246b33338de647689d7404c70676e76f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:27.610299 1150 raft_consensus.cc:738] T 246b33338de647689d7404c70676e76f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:27.610510 1150 consensus_queue.cc:260] T 246b33338de647689d7404c70676e76f P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.610738 2123 raft_consensus.cc:491] T 7954393c8c1c4457ab8bb723c03a9c8b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:27.610852 2123 raft_consensus.cc:513] T 7954393c8c1c4457ab8bb723c03a9c8b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.611027 1150 ts_tablet_manager.cc:1428] T 246b33338de647689d7404c70676e76f P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250901 01:52:27.611301 1150 tablet_bootstrap.cc:492] T c6f7c525f59b49938b697de98d4b5835 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:27.611374 2123 leader_election.cc:290] T 7954393c8c1c4457ab8bb723c03a9c8b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:27.613422 1151 tablet_bootstrap.cc:492] T 3d90c6ef06d1436d8e2bb1333e7d9985 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:27.613597 1151 ts_tablet_manager.cc:1397] T 3d90c6ef06d1436d8e2bb1333e7d9985 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.060s user 0.003s sys 0.000s
I20250901 01:52:27.613583 2123 raft_consensus.cc:491] T 8b9d79a25452410891c09fd3f7c4be6d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:27.613696 2123 raft_consensus.cc:513] T 8b9d79a25452410891c09fd3f7c4be6d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.614197 2123 leader_election.cc:290] T 8b9d79a25452410891c09fd3f7c4be6d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:27.614297 1151 raft_consensus.cc:357] T 3d90c6ef06d1436d8e2bb1333e7d9985 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:27.614409 858 leader_election.cc:336] T 7954393c8c1c4457ab8bb723c03a9c8b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:27.614449 1151 raft_consensus.cc:383] T 3d90c6ef06d1436d8e2bb1333e7d9985 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:27.614491 1151 raft_consensus.cc:738] T 3d90c6ef06d1436d8e2bb1333e7d9985 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:27.614653 1151 consensus_queue.cc:260] T 3d90c6ef06d1436d8e2bb1333e7d9985 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.615072 1151 ts_tablet_manager.cc:1428] T 3d90c6ef06d1436d8e2bb1333e7d9985 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250901 01:52:27.615376 1151 tablet_bootstrap.cc:492] T aec951f2cc84499b9162b9fa24d5440e P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:27.615726 2357 raft_consensus.cc:513] T f6a8b425968e49deaecfa3adacfd3f1f P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:27.616463 858 leader_election.cc:336] T 8b9d79a25452410891c09fd3f7c4be6d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:27.617053 1484 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet c6f7c525f59b49938b697de98d4b5835 (table default.Table_db61e055dbd7450898e82c4c0c4763be [id=aabca67f6f754820977ac9c4c20f3e4f]). Expected version 1 got 0
I20250901 01:52:27.617336 1151 tablet_bootstrap.cc:654] T aec951f2cc84499b9162b9fa24d5440e P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:27.617583 2357 leader_election.cc:290] T f6a8b425968e49deaecfa3adacfd3f1f P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:27.618228 858 leader_election.cc:336] T f6a8b425968e49deaecfa3adacfd3f1f P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:27.618413 1484 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 616ff34e8d314e0aa0e11ce60be11789 (table default.Table_0451d0afa26f4d5f8d4214a946eea04f [id=5ee13d46e06a4fa59b1070547573e523]). Expected version 1 got 0
I20250901 01:52:27.618131 2407 raft_consensus.cc:695] T 2599f98abfdc4871b573edfc2ad97d22 P 00c282f779c1477d920f24dc0dc16628 [term 1 LEADER]: Becoming Leader. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Running, Role: LEADER
I20250901 01:52:27.618875 2407 consensus_queue.cc:237] T 2599f98abfdc4871b573edfc2ad97d22 P 00c282f779c1477d920f24dc0dc16628 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.619424 1484 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 081e2630e284498aa819f85bf8326315 (table default.Table_9e7ca61fcc324a8984bd1e470162c61f [id=8199118617cb4707bc3ae9b1363750f7]). Expected version 2 got 0
I20250901 01:52:27.620066 2354 tablet.cc:1722] T 42c1156e8dd84c6fa1396b46de13eb6f P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:27.620769 1484 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet e87d1281f35648b0b115b67c95e244e7 (table default.Table_f840789a58984799bd8057128f6ea439 [id=4453959e273c473fadd737f56f7db3dd]). Expected version 1 got 0
I20250901 01:52:27.630200 1150 tablet_bootstrap.cc:654] T c6f7c525f59b49938b697de98d4b5835 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:27.635258 2118 ts_tablet_manager.cc:1918] T a54af99bc10045e8b0b6868e5aed98b4 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:27.635562 2118 log.cc:1199] T a54af99bc10045e8b0b6868e5aed98b4 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/a54af99bc10045e8b0b6868e5aed98b4
I20250901 01:52:27.636210 2118 ts_tablet_manager.cc:1939] T a54af99bc10045e8b0b6868e5aed98b4 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:27.637796 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 55) took 2434 ms. Trace:
I20250901 01:52:27.638087 2118 rpcz_store.cc:276] 0901 01:52:25.203324 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.203358 (+ 34us) service_pool.cc:225] Handling call
0901 01:52:27.573252 (+2369894us) ts_tablet_manager.cc:1145] Deleting tablet a54af99bc10045e8b0b6868e5aed98b4
0901 01:52:27.573261 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:27.623696 (+ 50435us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:27.635115 (+ 11419us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:27.637772 (+ 2657us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2369749}
I20250901 01:52:27.638669 2113 ts_tablet_manager.cc:1918] T 20f7bfab18134e079e91c52c89468cc3 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:27.638794 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet a54af99bc10045e8b0b6868e5aed98b4 (table default.Table_489b87a28dac48d093c241f301d5cccc [id=5fdd352d5e5e47a3a2bd8fd608a263dd]) successfully deleted
I20250901 01:52:27.638919 2118 tablet_replica.cc:331] T 7a2f6e2c7ee2402ea310c058e046416c P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:27.639290 2118 raft_consensus.cc:2241] T 7a2f6e2c7ee2402ea310c058e046416c P 00c282f779c1477d920f24dc0dc16628 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:27.639786 2118 raft_consensus.cc:2270] T 7a2f6e2c7ee2402ea310c058e046416c P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:27.641288 2118 ts_tablet_manager.cc:1905] T 7a2f6e2c7ee2402ea310c058e046416c P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:27.649320 1150 tablet_bootstrap.cc:492] T c6f7c525f59b49938b697de98d4b5835 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:27.649564 1150 ts_tablet_manager.cc:1397] T c6f7c525f59b49938b697de98d4b5835 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.000s
I20250901 01:52:27.650346 2118 ts_tablet_manager.cc:1918] T 7a2f6e2c7ee2402ea310c058e046416c P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:27.650578 2118 log.cc:1199] T 7a2f6e2c7ee2402ea310c058e046416c P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/7a2f6e2c7ee2402ea310c058e046416c
I20250901 01:52:27.650547 1150 raft_consensus.cc:357] T c6f7c525f59b49938b697de98d4b5835 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.650849 1150 raft_consensus.cc:383] T c6f7c525f59b49938b697de98d4b5835 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:27.650967 1150 raft_consensus.cc:738] T c6f7c525f59b49938b697de98d4b5835 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:27.651183 2118 ts_tablet_manager.cc:1939] T 7a2f6e2c7ee2402ea310c058e046416c P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:27.651269 1150 consensus_queue.cc:260] T c6f7c525f59b49938b697de98d4b5835 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.651685 2408 raft_consensus.cc:491] T c58edaa76ebf47dfb1d9d5fab6de7dfa P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:27.651818 2408 raft_consensus.cc:513] T c58edaa76ebf47dfb1d9d5fab6de7dfa P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.651971 1150 ts_tablet_manager.cc:1428] T c6f7c525f59b49938b697de98d4b5835 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250901 01:52:27.652248 1150 tablet_bootstrap.cc:492] T 268439ff8d3a4f7a89f3d6395f2079c5 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:27.652393 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 56) took 2448 ms. Trace:
I20250901 01:52:27.652537 2118 rpcz_store.cc:276] 0901 01:52:25.203622 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.203665 (+ 43us) service_pool.cc:225] Handling call
0901 01:52:27.638820 (+2435155us) ts_tablet_manager.cc:1145] Deleting tablet 7a2f6e2c7ee2402ea310c058e046416c
0901 01:52:27.638855 (+ 35us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:27.645829 (+ 6974us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:27.650205 (+ 4376us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:27.652353 (+ 2148us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2434403}
I20250901 01:52:27.653474 2408 leader_election.cc:290] T c58edaa76ebf47dfb1d9d5fab6de7dfa P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:27.653597 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 57) took 2442 ms. Trace:
I20250901 01:52:27.653846 1150 tablet_bootstrap.cc:654] T 268439ff8d3a4f7a89f3d6395f2079c5 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:27.654201 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 7a2f6e2c7ee2402ea310c058e046416c (table default.Table_a89cc8e48563460baabc34c821ceae35 [id=9c3930e9fcfe4b49ba4bea67a2b0ad07]) successfully deleted
W20250901 01:52:27.644769 992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:40561, user_credentials={real_user=slave}} blocked reactor thread for 61095.6us
I20250901 01:52:27.654000 2113 log.cc:1199] T 20f7bfab18134e079e91c52c89468cc3 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/20f7bfab18134e079e91c52c89468cc3
I20250901 01:52:27.646067 1151 tablet_bootstrap.cc:492] T aec951f2cc84499b9162b9fa24d5440e P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:27.660955 1151 ts_tablet_manager.cc:1397] T aec951f2cc84499b9162b9fa24d5440e P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.046s user 0.004s sys 0.000s
I20250901 01:52:27.661232 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3eba54e5ddd14c518f4cbd539268ae37" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:27.661585 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b85d64e02cc646e380581ac9f59dee9e" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:27.661585 1058 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "71941b815d6840478c5acc1cbb187cc9" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58"
I20250901 01:52:27.661782 1058 raft_consensus.cc:3058] T 71941b815d6840478c5acc1cbb187cc9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:27.661844 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b9d79a25452410891c09fd3f7c4be6d" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:27.661928 1054 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75345ea2a3dc40fc8c9f8090d8839d2d" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:27.662186 1055 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b164b4a67a9143ceaff69b0ce49d63fe" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:27.662375 992 leader_election.cc:336] T 8ed6616ce54548ba83ba299c4ecf3c38 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:27.662483 1051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f6a8b425968e49deaecfa3adacfd3f1f" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58"
I20250901 01:52:27.662420 1151 raft_consensus.cc:357] T aec951f2cc84499b9162b9fa24d5440e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.662724 1151 raft_consensus.cc:383] T aec951f2cc84499b9162b9fa24d5440e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:27.662819 1151 raft_consensus.cc:738] T aec951f2cc84499b9162b9fa24d5440e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:27.663233 1151 consensus_queue.cc:260] T aec951f2cc84499b9162b9fa24d5440e P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.662423 1053 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7954393c8c1c4457ab8bb723c03a9c8b" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:27.663797 1151 ts_tablet_manager.cc:1428] T aec951f2cc84499b9162b9fa24d5440e P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250901 01:52:27.662617 1051 raft_consensus.cc:3058] T f6a8b425968e49deaecfa3adacfd3f1f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:27.664448 1058 raft_consensus.cc:2466] T 71941b815d6840478c5acc1cbb187cc9 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c282f779c1477d920f24dc0dc16628 in term 1.
I20250901 01:52:27.666687 1051 raft_consensus.cc:2466] T f6a8b425968e49deaecfa3adacfd3f1f P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c282f779c1477d920f24dc0dc16628 in term 1.
I20250901 01:52:27.667071 1151 tablet_bootstrap.cc:492] T 5e2472ee5d6e4d3bbf1f24027ba531ca P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:27.667896 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c58edaa76ebf47dfb1d9d5fab6de7dfa" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:27.668058 1057 raft_consensus.cc:2466] T c58edaa76ebf47dfb1d9d5fab6de7dfa P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00c282f779c1477d920f24dc0dc16628 in term 0.
I20250901 01:52:27.668685 2113 ts_tablet_manager.cc:1939] T 20f7bfab18134e079e91c52c89468cc3 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:27.670217 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 82) took 2619 ms. Trace:
I20250901 01:52:27.670388 2113 rpcz_store.cc:276] 0901 01:52:25.050894 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.050975 (+ 81us) service_pool.cc:225] Handling call
0901 01:52:27.573627 (+2522652us) ts_tablet_manager.cc:1145] Deleting tablet 20f7bfab18134e079e91c52c89468cc3
0901 01:52:27.573637 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:27.635484 (+ 61847us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:27.638573 (+ 3089us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:27.670200 (+ 31627us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2481063}
I20250901 01:52:27.671679 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 20f7bfab18134e079e91c52c89468cc3 (table default.Table_18b93109e1224ea59b5c93825b3b27d6 [id=d2ef66bea68843bcb33819f88668437f]) successfully deleted
I20250901 01:52:27.669463 1151 tablet_bootstrap.cc:654] T 5e2472ee5d6e4d3bbf1f24027ba531ca P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:27.662621 992 consensus_peers.cc:489] T a289a66b2dfa44dc8c8547ff83c1716e P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250901 01:52:27.672087 992 leader_election.cc:336] T f13b972e25214f168421439fe3943e8a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:27.672358 992 consensus_peers.cc:489] T 3618a3eb4f174c43805ce7dc0e5315cb P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250901 01:52:27.672950 1465 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet 7442c6ff699141e1aff208ebdc9d192a on TS 0cf599f74322478d91475df234e58965 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
W20250901 01:52:27.673100 992 leader_election.cc:336] T 89d40f0331114490b51de3ba6d773571 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:27.653874 2118 rpcz_store.cc:276] 0901 01:52:25.210865 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.210915 (+ 50us) service_pool.cc:225] Handling call
0901 01:52:27.653036 (+2442121us) ts_tablet_manager.cc:1145] Deleting tablet 26c8ed083cea4dc0acca80d00b11b29b
0901 01:52:27.653403 (+ 367us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:27.653557 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2440384}
W20250901 01:52:27.654321 858 leader_election.cc:336] T c58edaa76ebf47dfb1d9d5fab6de7dfa P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:27.673344 992 consensus_peers.cc:489] T e249474dcbf642189560c91ec3a99fdf P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:27.673542 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 59) took 2458 ms. Trace:
I20250901 01:52:27.673626 2118 rpcz_store.cc:276] 0901 01:52:25.215192 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.215238 (+ 46us) service_pool.cc:225] Handling call
0901 01:52:27.673442 (+2458204us) ts_tablet_manager.cc:1145] Deleting tablet 17e8340dbd1a406cb782dad3fe88cc7e
0901 01:52:27.673450 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:27.673518 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2457602}
W20250901 01:52:27.673817 1472 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 26c8ed083cea4dc0acca80d00b11b29b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 26c8ed083cea4dc0acca80d00b11b29b already in progress: creating tablet
I20250901 01:52:27.673914 2118 tablet_replica.cc:331] T e25923b2daf9429683b2d44a75a56189 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
W20250901 01:52:27.674073 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 17e8340dbd1a406cb782dad3fe88cc7e because the tablet was not found. No further retry: Not found: Tablet not found: 17e8340dbd1a406cb782dad3fe88cc7e
I20250901 01:52:27.674103 2118 raft_consensus.cc:2241] T e25923b2daf9429683b2d44a75a56189 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:27.674245 2118 raft_consensus.cc:2270] T e25923b2daf9429683b2d44a75a56189 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:27.674750 2118 ts_tablet_manager.cc:1905] T e25923b2daf9429683b2d44a75a56189 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:27.675555 859 leader_election.cc:343] T 3eba54e5ddd14c518f4cbd539268ae37 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:27.675662 859 leader_election.cc:304] T 3eba54e5ddd14c518f4cbd539268ae37 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:27.676043 859 leader_election.cc:343] T b85d64e02cc646e380581ac9f59dee9e P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:27.676128 859 leader_election.cc:304] T b85d64e02cc646e380581ac9f59dee9e P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:27.677266 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 83) took 2625 ms. Trace:
I20250901 01:52:27.677351 2113 rpcz_store.cc:276] 0901 01:52:25.051861 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.052429 (+ 568us) service_pool.cc:225] Handling call
0901 01:52:27.673132 (+2620703us) ts_tablet_manager.cc:1145] Deleting tablet 56f2c4d2e6ab4bb4bdd6d025843589b6
0901 01:52:27.673140 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:27.677248 (+ 4108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2580367}
I20250901 01:52:27.677608 2407 consensus_queue.cc:1035] T 8ca5cf4e99014c0d84a3acad4ef713af P 00c282f779c1477d920f24dc0dc16628 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250901 01:52:27.677769 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 56f2c4d2e6ab4bb4bdd6d025843589b6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 56f2c4d2e6ab4bb4bdd6d025843589b6 already in progress: creating tablet
I20250901 01:52:27.677999 2407 raft_consensus.cc:2747] T 3eba54e5ddd14c518f4cbd539268ae37 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:27.678118 2407 raft_consensus.cc:2747] T b85d64e02cc646e380581ac9f59dee9e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:27.678583 1993 raft_consensus.cc:491] T ea846f88b3534beebd0001b7f17b5ca9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:27.678689 1993 raft_consensus.cc:513] T ea846f88b3534beebd0001b7f17b5ca9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.679211 1993 leader_election.cc:290] T ea846f88b3534beebd0001b7f17b5ca9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:27.679454 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "89d40f0331114490b51de3ba6d773571" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:27.679595 916 raft_consensus.cc:3058] T 89d40f0331114490b51de3ba6d773571 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:27.679996 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 84) took 2627 ms. Trace:
I20250901 01:52:27.680058 2113 rpcz_store.cc:276] 0901 01:52:25.052657 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.052830 (+ 173us) service_pool.cc:225] Handling call
0901 01:52:27.679903 (+2627073us) ts_tablet_manager.cc:1145] Deleting tablet 879fa66e55544118aafe504587393c66
0901 01:52:27.679912 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:27.679982 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2587093}
W20250901 01:52:27.680976 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 879fa66e55544118aafe504587393c66 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 879fa66e55544118aafe504587393c66 already in progress: creating tablet
I20250901 01:52:27.681042 922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f13b972e25214f168421439fe3943e8a" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:27.681174 922 raft_consensus.cc:3058] T f13b972e25214f168421439fe3943e8a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:27.681273 2113 tablet_replica.cc:331] T ef58f477ebeb49639ed7d071e1a62c10 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:27.681444 2113 raft_consensus.cc:2241] T ef58f477ebeb49639ed7d071e1a62c10 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:27.681576 2113 raft_consensus.cc:2270] T ef58f477ebeb49639ed7d071e1a62c10 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:27.682010 2113 ts_tablet_manager.cc:1905] T ef58f477ebeb49639ed7d071e1a62c10 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:27.682088 916 raft_consensus.cc:2466] T 89d40f0331114490b51de3ba6d773571 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:27.683439 922 raft_consensus.cc:2466] T f13b972e25214f168421439fe3943e8a P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:27.683692 918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8ed6616ce54548ba83ba299c4ecf3c38" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:27.683804 918 raft_consensus.cc:3058] T 8ed6616ce54548ba83ba299c4ecf3c38 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:27.686161 918 raft_consensus.cc:2466] T 8ed6616ce54548ba83ba299c4ecf3c38 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
E20250901 01:52:27.686807 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae4844183a3c4a04a775e6fd878391a3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_aee74198b51d4f089538c4f2bf3ce4dc'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:27.686919 1696 catalog_manager.cc:6205] Error selecting replicas for tablet be8dfbc466564842a2fdb4953eee939b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_75ba3ba70ca24a94920e7bce96190ff2'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:27.687005 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 96824e68bbf24b8287dbb683038ffaba, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_3c7c628f35fb4d02bb79a34995f12938'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:27.687077 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 7317eb7848ee44698038a4b404d4bf21, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b0e916c76a7a4766aa0887cc8691488f'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:27.687170 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9745582cebfe47b58c21d0191ab4cb9b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2f09538b6fbe401f909105decb56e1cb'. Need at least 3 replicas, but only 2 tablet servers are available
W20250901 01:52:27.687182 859 leader_election.cc:343] T 8b9d79a25452410891c09fd3f7c4be6d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
E20250901 01:52:27.687240 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 4abf9725be5d42f3893f07f0a9c7614c, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_8445fcd774874cadaf4acc3581eee417'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:27.687280 859 leader_election.cc:304] T 8b9d79a25452410891c09fd3f7c4be6d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
E20250901 01:52:27.687322 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 8a13346dcb3d4b6ab35c261829d50ae5, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_1bdd9041d74045cab6f23de11bf4dd09'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:27.687392 1696 catalog_manager.cc:6205] Error selecting replicas for tablet e2ed32b3292141278af28026528e96c2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_50ae3908a7944f70949c54be8fbf1564'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:27.687480 1696 catalog_manager.cc:6205] Error selecting replicas for tablet a2ca69fabadd4ad69e1670cae7a0c503, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_32c4716e3aa24a87a6b1472984b255c4'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:27.687561 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 76ba642c16734cdf969dbd3130639fda, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_faf7ef12324f454396e5ae64d64ebe51'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:27.687659 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f8f923ddcd2740bcb7e521f4f57c4034, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_5e65c5f068b84995a2d7a76fa65dda0f'. Need at least 3 replicas, but only 2 tablet servers are available
W20250901 01:52:27.687642 859 leader_election.cc:343] T 75345ea2a3dc40fc8c9f8090d8839d2d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
E20250901 01:52:27.687728 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 0813cc68d4b0488ba60a523e4479971b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d1302673d68f424ebb54b220911ce385'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:27.687724 859 leader_election.cc:304] T 75345ea2a3dc40fc8c9f8090d8839d2d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
E20250901 01:52:27.687836 1696 catalog_manager.cc:6205] Error selecting replicas for tablet b1ba0e19dc0b462e80c537bf2d183f5b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0e2cd82b00f1466eaef7f287402e644d'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:27.687914 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5f0e054f62de4cf6bc6f69ad6664b192, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ef6832d0845044ed9f165ca3541b4c3f'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:27.687992 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 787be682c0e44248a80d7037fb61dc6c, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7c12d66048a6470e8c227dc1d9f45fa4'. Need at least 3 replicas, but only 2 tablet servers are available
W20250901 01:52:27.688004 859 leader_election.cc:343] T b164b4a67a9143ceaff69b0ce49d63fe P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
E20250901 01:52:27.688059 1696 catalog_manager.cc:6205] Error selecting replicas for tablet cc3b8736b9094d7e892c80da4d5b20f6, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_93afd37289414655980b22934695bd55'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:27.688079 859 leader_election.cc:304] T b164b4a67a9143ceaff69b0ce49d63fe P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
E20250901 01:52:27.688275 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae9a4ad4fdfb4771ba23a91f7007447e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_5e84ae9a9c2a4084a1f9c5ffb4594de2'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:27.688351 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9074543dea9949e7a085717b40913348, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_53e30205c5fd488eb3b48c685a159d04'. Need at least 3 replicas, but only 2 tablet servers are available
W20250901 01:52:27.688346 859 leader_election.cc:343] T 7954393c8c1c4457ab8bb723c03a9c8b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
E20250901 01:52:27.688450 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 18498c9cb09141d08c1c5c0854b63ebd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77f4810bc83c4d948531da2a45a3792e'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:27.688438 859 leader_election.cc:304] T 7954393c8c1c4457ab8bb723c03a9c8b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
E20250901 01:52:27.688567 1696 catalog_manager.cc:6205] Error selecting replicas for tablet a6b539e160194da3a849d9e2f1ec9859, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_03fb48eaab9b42858ab7cefb722e43a8'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:27.688671 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 6ce9521b925b4f97bb4a810ce77e147e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_9e04f976a93f4d30baccf90fa3622dfe'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:27.688738 859 leader_election.cc:304] T 71941b815d6840478c5acc1cbb187cc9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:27.689059 859 leader_election.cc:304] T f6a8b425968e49deaecfa3adacfd3f1f P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:27.689350 2123 raft_consensus.cc:2747] T 8b9d79a25452410891c09fd3f7c4be6d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:27.689494 2123 raft_consensus.cc:2747] T 75345ea2a3dc40fc8c9f8090d8839d2d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:27.689618 2123 raft_consensus.cc:2747] T b164b4a67a9143ceaff69b0ce49d63fe P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:27.689718 2123 raft_consensus.cc:2747] T 7954393c8c1c4457ab8bb723c03a9c8b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:27.689826 2123 raft_consensus.cc:2802] T 71941b815d6840478c5acc1cbb187cc9 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:27.689904 2123 raft_consensus.cc:695] T 71941b815d6840478c5acc1cbb187cc9 P 00c282f779c1477d920f24dc0dc16628 [term 1 LEADER]: Becoming Leader. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Running, Role: LEADER
I20250901 01:52:27.690121 2123 consensus_queue.cc:237] T 71941b815d6840478c5acc1cbb187cc9 P 00c282f779c1477d920f24dc0dc16628 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.691071 2323 raft_consensus.cc:2802] T f6a8b425968e49deaecfa3adacfd3f1f P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:27.691170 2323 raft_consensus.cc:695] T f6a8b425968e49deaecfa3adacfd3f1f P 00c282f779c1477d920f24dc0dc16628 [term 1 LEADER]: Becoming Leader. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Running, Role: LEADER
I20250901 01:52:27.691365 2323 consensus_queue.cc:237] T f6a8b425968e49deaecfa3adacfd3f1f P 00c282f779c1477d920f24dc0dc16628 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.692036 1151 tablet_bootstrap.cc:492] T 5e2472ee5d6e4d3bbf1f24027ba531ca P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:27.692183 1151 ts_tablet_manager.cc:1397] T 5e2472ee5d6e4d3bbf1f24027ba531ca P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.000s
I20250901 01:52:27.693588 1151 raft_consensus.cc:357] T 5e2472ee5d6e4d3bbf1f24027ba531ca P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.694651 2118 ts_tablet_manager.cc:1918] T e25923b2daf9429683b2d44a75a56189 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:27.696468 1151 raft_consensus.cc:383] T 5e2472ee5d6e4d3bbf1f24027ba531ca P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:27.696640 1151 raft_consensus.cc:738] T 5e2472ee5d6e4d3bbf1f24027ba531ca P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:27.696913 1151 consensus_queue.cc:260] T 5e2472ee5d6e4d3bbf1f24027ba531ca P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.697521 1151 ts_tablet_manager.cc:1428] T 5e2472ee5d6e4d3bbf1f24027ba531ca P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250901 01:52:27.697659 2407 raft_consensus.cc:491] T becbaae4fd7e44faa679e9479465a7c4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:27.697798 2407 raft_consensus.cc:513] T becbaae4fd7e44faa679e9479465a7c4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.698302 2407 leader_election.cc:290] T becbaae4fd7e44faa679e9479465a7c4 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:27.699721 858 leader_election.cc:336] T becbaae4fd7e44faa679e9479465a7c4 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:27.700670 1151 tablet_bootstrap.cc:492] T 7954393c8c1c4457ab8bb723c03a9c8b P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:27.702498 1151 tablet_bootstrap.cc:654] T 7954393c8c1c4457ab8bb723c03a9c8b P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:27.705622 2113 ts_tablet_manager.cc:1918] T ef58f477ebeb49639ed7d071e1a62c10 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:27.705941 2113 log.cc:1199] T ef58f477ebeb49639ed7d071e1a62c10 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/ef58f477ebeb49639ed7d071e1a62c10
I20250901 01:52:27.706470 2113 ts_tablet_manager.cc:1939] T ef58f477ebeb49639ed7d071e1a62c10 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:27.707480 859 leader_election.cc:304] T c58edaa76ebf47dfb1d9d5fab6de7dfa P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:27.709089 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 85) took 2655 ms. Trace:
I20250901 01:52:27.709385 2113 rpcz_store.cc:276] 0901 01:52:25.053134 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.053184 (+ 50us) service_pool.cc:225] Handling call
0901 01:52:27.681228 (+2628044us) ts_tablet_manager.cc:1145] Deleting tablet ef58f477ebeb49639ed7d071e1a62c10
0901 01:52:27.681239 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:27.695759 (+ 14520us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:27.704107 (+ 8348us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:27.709063 (+ 4956us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2588366}
I20250901 01:52:27.709096 2420 raft_consensus.cc:2802] T c58edaa76ebf47dfb1d9d5fab6de7dfa P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:27.710290 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet ef58f477ebeb49639ed7d071e1a62c10 (table default.Table_7f507bbabca54260a4a5d9092c274e41 [id=6020a2081326445090e69ef48cad0d9d]) successfully deleted
I20250901 01:52:27.710184 2420 raft_consensus.cc:491] T c58edaa76ebf47dfb1d9d5fab6de7dfa P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:27.710944 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 86) took 2657 ms. Trace:
I20250901 01:52:27.711227 2420 raft_consensus.cc:3058] T c58edaa76ebf47dfb1d9d5fab6de7dfa P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:27.711496 2113 rpcz_store.cc:276] 0901 01:52:25.053505 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.053549 (+ 44us) service_pool.cc:225] Handling call
0901 01:52:27.710789 (+2657240us) ts_tablet_manager.cc:1145] Deleting tablet f98d1d8d660443cb8066a547e5d9b77f
0901 01:52:27.710801 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:27.710923 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2617888}
W20250901 01:52:27.712374 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet f98d1d8d660443cb8066a547e5d9b77f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f98d1d8d660443cb8066a547e5d9b77f already in progress: creating tablet
I20250901 01:52:27.712613 2123 raft_consensus.cc:491] T e2aa32450a584395ae07b49f76a7b26d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:27.712728 2123 raft_consensus.cc:513] T e2aa32450a584395ae07b49f76a7b26d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.712844 2113 tablet_replica.cc:331] T 60285f2ba037433999f5455fb9cc7acd P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:27.713268 2113 raft_consensus.cc:2241] T 60285f2ba037433999f5455fb9cc7acd P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:27.713326 2123 leader_election.cc:290] T e2aa32450a584395ae07b49f76a7b26d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:27.713779 2113 raft_consensus.cc:2270] T 60285f2ba037433999f5455fb9cc7acd P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:27.714351 917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ea846f88b3534beebd0001b7f17b5ca9" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:27.714529 917 raft_consensus.cc:2466] T ea846f88b3534beebd0001b7f17b5ca9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:27.715035 2113 ts_tablet_manager.cc:1905] T 60285f2ba037433999f5455fb9cc7acd P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:27.715302 858 leader_election.cc:336] T e2aa32450a584395ae07b49f76a7b26d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:27.716362 1993 consensus_queue.cc:1035] T 81c36693199945d98465579b509852e2 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:27.716542 2412 consensus_queue.cc:1035] T e249474dcbf642189560c91ec3a99fdf P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:27.716660 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "becbaae4fd7e44faa679e9479465a7c4" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:27.716930 1051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2aa32450a584395ae07b49f76a7b26d" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:27.717355 2412 consensus_queue.cc:1035] T 3618a3eb4f174c43805ce7dc0e5315cb P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250901 01:52:27.717978 992 leader_election.cc:343] T b11e8f77243e4764b96548d1132b8e5a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: b11e8f77243e4764b96548d1132b8e5a
I20250901 01:52:27.718981 992 leader_election.cc:304] T 89d40f0331114490b51de3ba6d773571 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:27.720690 2412 raft_consensus.cc:2802] T 89d40f0331114490b51de3ba6d773571 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:27.720814 2412 raft_consensus.cc:695] T 89d40f0331114490b51de3ba6d773571 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:27.721063 2412 consensus_queue.cc:237] T 89d40f0331114490b51de3ba6d773571 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.722505 992 leader_election.cc:304] T f13b972e25214f168421439fe3943e8a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:27.723213 2412 raft_consensus.cc:2802] T f13b972e25214f168421439fe3943e8a P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:27.723331 2412 raft_consensus.cc:695] T f13b972e25214f168421439fe3943e8a P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:27.723588 2412 consensus_queue.cc:237] T f13b972e25214f168421439fe3943e8a P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.724718 992 leader_election.cc:304] T 8ed6616ce54548ba83ba299c4ecf3c38 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:27.697664 2118 log.cc:1199] T e25923b2daf9429683b2d44a75a56189 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/e25923b2daf9429683b2d44a75a56189
I20250901 01:52:27.725855 2118 ts_tablet_manager.cc:1939] T e25923b2daf9429683b2d44a75a56189 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:27.726176 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 58) took 2512 ms. Trace:
I20250901 01:52:27.726272 2118 rpcz_store.cc:276] 0901 01:52:25.214148 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.214193 (+ 45us) service_pool.cc:225] Handling call
0901 01:52:27.673872 (+2459679us) ts_tablet_manager.cc:1145] Deleting tablet e25923b2daf9429683b2d44a75a56189
0901 01:52:27.673880 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:27.679304 (+ 5424us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:27.694534 (+ 15230us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:27.726159 (+ 31625us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2457669}
I20250901 01:52:27.729384 2113 ts_tablet_manager.cc:1918] T 60285f2ba037433999f5455fb9cc7acd P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:27.730080 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet e25923b2daf9429683b2d44a75a56189 (table default.Table_202b4c5ccc56474a8b2e7b9d7c4a28d6 [id=746845bd84c6444b8a1e5683eaba4a68]) successfully deleted
I20250901 01:52:27.731097 2420 raft_consensus.cc:513] T c58edaa76ebf47dfb1d9d5fab6de7dfa P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.732581 2113 log.cc:1199] T 60285f2ba037433999f5455fb9cc7acd P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/60285f2ba037433999f5455fb9cc7acd
I20250901 01:52:27.732726 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 60) took 2516 ms. Trace:
I20250901 01:52:27.732818 2118 rpcz_store.cc:276] 0901 01:52:25.215913 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.215952 (+ 39us) service_pool.cc:225] Handling call
0901 01:52:27.732576 (+2516624us) ts_tablet_manager.cc:1145] Deleting tablet f98d1d8d660443cb8066a547e5d9b77f
0901 01:52:27.732587 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:27.732708 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2516066}
I20250901 01:52:27.733215 2113 ts_tablet_manager.cc:1939] T 60285f2ba037433999f5455fb9cc7acd P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
W20250901 01:52:27.733860 1472 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet f98d1d8d660443cb8066a547e5d9b77f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f98d1d8d660443cb8066a547e5d9b77f already in progress: creating tablet
I20250901 01:52:27.734272 1150 tablet_bootstrap.cc:492] T 268439ff8d3a4f7a89f3d6395f2079c5 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:27.734400 1150 ts_tablet_manager.cc:1397] T 268439ff8d3a4f7a89f3d6395f2079c5 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.082s user 0.011s sys 0.006s
I20250901 01:52:27.734938 2409 raft_consensus.cc:2802] T 8ed6616ce54548ba83ba299c4ecf3c38 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:27.735062 2409 raft_consensus.cc:695] T 8ed6616ce54548ba83ba299c4ecf3c38 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:27.735328 2409 consensus_queue.cc:237] T 8ed6616ce54548ba83ba299c4ecf3c38 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.735644 1150 raft_consensus.cc:357] T 268439ff8d3a4f7a89f3d6395f2079c5 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.735857 1150 raft_consensus.cc:383] T 268439ff8d3a4f7a89f3d6395f2079c5 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:27.735954 1150 raft_consensus.cc:738] T 268439ff8d3a4f7a89f3d6395f2079c5 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:27.736227 1150 consensus_queue.cc:260] T 268439ff8d3a4f7a89f3d6395f2079c5 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.736981 1150 ts_tablet_manager.cc:1428] T 268439ff8d3a4f7a89f3d6395f2079c5 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250901 01:52:27.737290 1150 tablet_bootstrap.cc:492] T 81ce448b148447ddbc41875d863e0518 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:27.738652 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 88) took 2684 ms. Trace:
I20250901 01:52:27.738870 2113 rpcz_store.cc:276] 0901 01:52:25.054399 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.054452 (+ 53us) service_pool.cc:225] Handling call
0901 01:52:27.712801 (+2658349us) ts_tablet_manager.cc:1145] Deleting tablet 60285f2ba037433999f5455fb9cc7acd
0901 01:52:27.712810 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:27.725591 (+ 12781us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:27.729247 (+ 3656us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:27.738634 (+ 9387us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4288,"tablet-delete.queue_time_us":2619868}
I20250901 01:52:27.739104 1150 tablet_bootstrap.cc:654] T 81ce448b148447ddbc41875d863e0518 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:27.739464 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 60285f2ba037433999f5455fb9cc7acd (table default.Table_b582d9a2134b4184a195cfe0a8335a33 [id=cdf183c0a78b473b8d3428663415ab8d]) successfully deleted
I20250901 01:52:27.734663 992 leader_election.cc:304] T ea846f88b3534beebd0001b7f17b5ca9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:27.739923 2307 raft_consensus.cc:2802] T ea846f88b3534beebd0001b7f17b5ca9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:27.740001 2307 raft_consensus.cc:491] T ea846f88b3534beebd0001b7f17b5ca9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:27.740064 2307 raft_consensus.cc:3058] T ea846f88b3534beebd0001b7f17b5ca9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:27.740154 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 61) took 2523 ms. Trace:
I20250901 01:52:27.740229 2118 rpcz_store.cc:276] 0901 01:52:25.216585 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.216621 (+ 36us) service_pool.cc:225] Handling call
0901 01:52:27.740000 (+2523379us) ts_tablet_manager.cc:1145] Deleting tablet 50472a7aa2f04e0fbd65ac6c86c34b3e
0901 01:52:27.740010 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:27.740137 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2522317}
W20250901 01:52:27.740757 1472 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 50472a7aa2f04e0fbd65ac6c86c34b3e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 50472a7aa2f04e0fbd65ac6c86c34b3e already in progress: creating tablet
W20250901 01:52:27.741792 992 leader_election.cc:336] T 89d40f0331114490b51de3ba6d773571 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:27.746059 992 leader_election.cc:336] T 8ed6616ce54548ba83ba299c4ecf3c38 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:27.742599 2307 raft_consensus.cc:513] T ea846f88b3534beebd0001b7f17b5ca9 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.742204 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 87) took 2688 ms. Trace:
I20250901 01:52:27.746367 2113 rpcz_store.cc:276] 0901 01:52:25.054054 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.054593 (+ 539us) service_pool.cc:225] Handling call
0901 01:52:27.742061 (+2687468us) ts_tablet_manager.cc:1145] Deleting tablet e1c8fe96a6ba4edba4f09c00f7add75e
0901 01:52:27.742075 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:27.742185 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2649084}
I20250901 01:52:27.746695 2307 leader_election.cc:290] T ea846f88b3534beebd0001b7f17b5ca9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:27.747027 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet e1c8fe96a6ba4edba4f09c00f7add75e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e1c8fe96a6ba4edba4f09c00f7add75e already in progress: creating tablet
I20250901 01:52:27.747490 2113 tablet_replica.cc:331] T dd694cfa09564d3da5d279c0861a4c32 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:27.747754 2113 raft_consensus.cc:2241] T dd694cfa09564d3da5d279c0861a4c32 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:27.747957 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 62) took 2530 ms. Trace:
I20250901 01:52:27.748065 2118 rpcz_store.cc:276] 0901 01:52:25.217838 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.217896 (+ 58us) service_pool.cc:225] Handling call
0901 01:52:27.742204 (+2524308us) ts_tablet_manager.cc:1145] Deleting tablet 764bfeb2a7d14c40815b5ec1b07bdd62
0901 01:52:27.747805 (+ 5601us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:27.747932 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2521398}
W20250901 01:52:27.748003 992 leader_election.cc:336] T f13b972e25214f168421439fe3943e8a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:27.748538 1472 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 764bfeb2a7d14c40815b5ec1b07bdd62 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 764bfeb2a7d14c40815b5ec1b07bdd62 already in progress: creating tablet
I20250901 01:52:27.748900 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 63) took 2528 ms. Trace:
I20250901 01:52:27.748975 2118 rpcz_store.cc:276] 0901 01:52:25.220901 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.220937 (+ 36us) service_pool.cc:225] Handling call
0901 01:52:27.748496 (+2527559us) ts_tablet_manager.cc:1145] Deleting tablet 56f2c4d2e6ab4bb4bdd6d025843589b6
0901 01:52:27.748800 (+ 304us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:27.748882 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2525360}
W20250901 01:52:27.749723 1472 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 56f2c4d2e6ab4bb4bdd6d025843589b6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 56f2c4d2e6ab4bb4bdd6d025843589b6 already in progress: creating tablet
I20250901 01:52:27.750054 2113 raft_consensus.cc:2270] T dd694cfa09564d3da5d279c0861a4c32 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250901 01:52:27.750312 858 leader_election.cc:336] T c58edaa76ebf47dfb1d9d5fab6de7dfa P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:27.751437 2420 leader_election.cc:290] T c58edaa76ebf47dfb1d9d5fab6de7dfa P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:27.751607 1151 tablet_bootstrap.cc:492] T 7954393c8c1c4457ab8bb723c03a9c8b P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:27.751780 1151 ts_tablet_manager.cc:1397] T 7954393c8c1c4457ab8bb723c03a9c8b P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.051s user 0.001s sys 0.002s
I20250901 01:52:27.752384 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 65) took 2528 ms. Trace:
I20250901 01:52:27.752532 2118 rpcz_store.cc:276] 0901 01:52:25.223453 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.223496 (+ 43us) service_pool.cc:225] Handling call
0901 01:52:27.749171 (+2525675us) ts_tablet_manager.cc:1145] Deleting tablet 4fca1435d82447f289fda4035aa2ed89
0901 01:52:27.752274 (+ 3103us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:27.752368 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2525502}
I20250901 01:52:27.752859 2118 tablet_replica.cc:331] T 0dbdcb76eeba4fefa56e7835c23c0316 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:27.753106 2118 raft_consensus.cc:2241] T 0dbdcb76eeba4fefa56e7835c23c0316 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:27.753295 2118 raft_consensus.cc:2270] T 0dbdcb76eeba4fefa56e7835c23c0316 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:27.753296 1151 raft_consensus.cc:357] T 7954393c8c1c4457ab8bb723c03a9c8b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.753537 1151 raft_consensus.cc:383] T 7954393c8c1c4457ab8bb723c03a9c8b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:27.753630 1151 raft_consensus.cc:738] T 7954393c8c1c4457ab8bb723c03a9c8b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:27.753865 1151 consensus_queue.cc:260] T 7954393c8c1c4457ab8bb723c03a9c8b P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.754395 1151 ts_tablet_manager.cc:1428] T 7954393c8c1c4457ab8bb723c03a9c8b P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20250901 01:52:27.750452 992 leader_election.cc:336] T b11e8f77243e4764b96548d1132b8e5a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:27.754568 992 leader_election.cc:304] T b11e8f77243e4764b96548d1132b8e5a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:27.755555 2307 raft_consensus.cc:2747] T b11e8f77243e4764b96548d1132b8e5a P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250901 01:52:27.755702 1472 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 4fca1435d82447f289fda4035aa2ed89 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4fca1435d82447f289fda4035aa2ed89 already in progress: creating tablet
I20250901 01:52:27.756103 2113 ts_tablet_manager.cc:1905] T dd694cfa09564d3da5d279c0861a4c32 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:27.756701 859 leader_election.cc:343] T becbaae4fd7e44faa679e9479465a7c4 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:27.756837 859 leader_election.cc:304] T becbaae4fd7e44faa679e9479465a7c4 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:27.757189 859 leader_election.cc:343] T e2aa32450a584395ae07b49f76a7b26d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:27.757272 859 leader_election.cc:304] T e2aa32450a584395ae07b49f76a7b26d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:27.757633 2123 raft_consensus.cc:2747] T becbaae4fd7e44faa679e9479465a7c4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:27.757779 2123 raft_consensus.cc:2747] T e2aa32450a584395ae07b49f76a7b26d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:27.754745 1151 tablet_bootstrap.cc:492] T e87d1281f35648b0b115b67c95e244e7 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:27.759510 1151 tablet_bootstrap.cc:654] T e87d1281f35648b0b115b67c95e244e7 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:27.762540 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c58edaa76ebf47dfb1d9d5fab6de7dfa" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58"
I20250901 01:52:27.762691 1057 raft_consensus.cc:3058] T c58edaa76ebf47dfb1d9d5fab6de7dfa P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:27.765303 1057 raft_consensus.cc:2466] T c58edaa76ebf47dfb1d9d5fab6de7dfa P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c282f779c1477d920f24dc0dc16628 in term 1.
W20250901 01:52:27.768225 992 leader_election.cc:336] T ea846f88b3534beebd0001b7f17b5ca9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:27.768491 922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ea846f88b3534beebd0001b7f17b5ca9" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:27.768698 922 raft_consensus.cc:3058] T ea846f88b3534beebd0001b7f17b5ca9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:27.771907 922 raft_consensus.cc:2466] T ea846f88b3534beebd0001b7f17b5ca9 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:27.772492 859 leader_election.cc:304] T c58edaa76ebf47dfb1d9d5fab6de7dfa P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:27.772737 2357 raft_consensus.cc:2802] T c58edaa76ebf47dfb1d9d5fab6de7dfa P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:27.772838 2357 raft_consensus.cc:695] T c58edaa76ebf47dfb1d9d5fab6de7dfa P 00c282f779c1477d920f24dc0dc16628 [term 1 LEADER]: Becoming Leader. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Running, Role: LEADER
I20250901 01:52:27.773068 2357 consensus_queue.cc:237] T c58edaa76ebf47dfb1d9d5fab6de7dfa P 00c282f779c1477d920f24dc0dc16628 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.776850 2409 consensus_queue.cc:1035] T a289a66b2dfa44dc8c8547ff83c1716e P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:27.777906 992 leader_election.cc:304] T ea846f88b3534beebd0001b7f17b5ca9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:27.778118 2409 raft_consensus.cc:2802] T ea846f88b3534beebd0001b7f17b5ca9 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:27.778218 2409 raft_consensus.cc:695] T ea846f88b3534beebd0001b7f17b5ca9 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:27.778424 2409 consensus_queue.cc:237] T ea846f88b3534beebd0001b7f17b5ca9 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.784415 2412 consensus_queue.cc:1035] T a8483e7212d94daba7e5401256c33aef P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250901 01:52:27.785466 992 consensus_peers.cc:489] T 81c36693199945d98465579b509852e2 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:27.791392 2113 ts_tablet_manager.cc:1918] T dd694cfa09564d3da5d279c0861a4c32 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:27.791538 2113 log.cc:1199] T dd694cfa09564d3da5d279c0861a4c32 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/dd694cfa09564d3da5d279c0861a4c32
I20250901 01:52:27.791989 2113 ts_tablet_manager.cc:1939] T dd694cfa09564d3da5d279c0861a4c32 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
W20250901 01:52:27.794674 992 leader_election.cc:336] T ea846f88b3534beebd0001b7f17b5ca9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:27.800174 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 1495de70f0844f82a0eea99c059f076b (table default.Table_af5338b09bbc4c879c7f4f573c32fd8d [id=70a5ce71d16b47d4bc6dae97be4548e7]). Expected version 3 got 0
W20250901 01:52:27.800925 992 consensus_peers.cc:489] T 831e42de95854541a49ae6b3e039d3e9 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250901 01:52:27.803598 1150 tablet_bootstrap.cc:492] T 81ce448b148447ddbc41875d863e0518 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:27.804845 1150 ts_tablet_manager.cc:1397] T 81ce448b148447ddbc41875d863e0518 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.068s user 0.002s sys 0.002s
I20250901 01:52:27.806453 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 89) took 2751 ms. Trace:
I20250901 01:52:27.806838 1150 raft_consensus.cc:357] T 81ce448b148447ddbc41875d863e0518 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.806996 2323 raft_consensus.cc:491] T ed1b41340ce742a88c3e4fd85cd25ca6 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:27.808529 1150 raft_consensus.cc:383] T 81ce448b148447ddbc41875d863e0518 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:27.808588 2323 raft_consensus.cc:513] T ed1b41340ce742a88c3e4fd85cd25ca6 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.809244 2323 leader_election.cc:290] T ed1b41340ce742a88c3e4fd85cd25ca6 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:27.804204 2118 ts_tablet_manager.cc:1905] T 0dbdcb76eeba4fefa56e7835c23c0316 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:27.810058 1492 catalog_manager.cc:5582] T 15e4d972e8254cf899c3e0b71b6e2bd9 P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } }
W20250901 01:52:27.810703 858 leader_election.cc:336] T ed1b41340ce742a88c3e4fd85cd25ca6 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:27.810779 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ed1b41340ce742a88c3e4fd85cd25ca6" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:27.812024 859 leader_election.cc:343] T ed1b41340ce742a88c3e4fd85cd25ca6 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:27.812140 859 leader_election.cc:304] T ed1b41340ce742a88c3e4fd85cd25ca6 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:27.812060 2113 rpcz_store.cc:276] 0901 01:52:25.055203 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.055460 (+ 257us) service_pool.cc:225] Handling call
0901 01:52:27.747434 (+2691974us) ts_tablet_manager.cc:1145] Deleting tablet dd694cfa09564d3da5d279c0861a4c32
0901 01:52:27.747450 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:27.772997 (+ 25547us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:27.791278 (+ 18281us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:27.806421 (+ 15143us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2654421}
I20250901 01:52:27.812551 2323 raft_consensus.cc:2747] T ed1b41340ce742a88c3e4fd85cd25ca6 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:27.813112 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet dd694cfa09564d3da5d279c0861a4c32 (table default.Table_7183dbd577624461af0f861fa4f63d6d [id=a785d19b66e546b5901b6d7e93106161]) successfully deleted
I20250901 01:52:27.816221 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 90) took 2760 ms. Trace:
I20250901 01:52:27.816439 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet ed1b41340ce742a88c3e4fd85cd25ca6 (table default.Table_139f6d174d794a6e9a618e90587d0c0d [id=0baa4993b72e4a8d94ee3059c9d2db6c]). Expected version 1 got 0
I20250901 01:52:27.816587 2113 rpcz_store.cc:276] 0901 01:52:25.055613 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.055661 (+ 48us) service_pool.cc:225] Handling call
0901 01:52:27.813417 (+2757756us) ts_tablet_manager.cc:1145] Deleting tablet 4fca1435d82447f289fda4035aa2ed89
0901 01:52:27.814180 (+ 763us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:27.816205 (+ 2025us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2720357}
I20250901 01:52:27.817181 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 14b466b2a5a249aa841f1b8b06301edc (table default.Table_927955c1048f4e9c84b5b3a113a473f7 [id=78cd71321e9a480ca2d7d88327f157ee]). Expected version 1 got 0
I20250901 01:52:27.817924 2113 tablet_replica.cc:331] T 4410da9e95454ecf95faf80221b0a71c P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:27.818370 2113 raft_consensus.cc:2241] T 4410da9e95454ecf95faf80221b0a71c P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
W20250901 01:52:27.817667 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 4fca1435d82447f289fda4035aa2ed89 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4fca1435d82447f289fda4035aa2ed89 already in progress: creating tablet
I20250901 01:52:27.818842 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet e87d1281f35648b0b115b67c95e244e7 (table default.Table_f840789a58984799bd8057128f6ea439 [id=4453959e273c473fadd737f56f7db3dd]). Expected version 1 got 0
I20250901 01:52:27.821359 2113 raft_consensus.cc:2270] T 4410da9e95454ecf95faf80221b0a71c P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:27.821750 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 7954393c8c1c4457ab8bb723c03a9c8b (table default.Table_b2083370690443c6aa50c453bd29cb9a [id=543f2074598d4c29972e564be1123fca]). Expected version 4 got 0
I20250901 01:52:27.823608 2113 ts_tablet_manager.cc:1905] T 4410da9e95454ecf95faf80221b0a71c P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:27.823962 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 6c0abbfc8d904dce989ce51961e4237d (table default.Table_3f9422b08b0349b9a5466218f0fe1b06 [id=f0dfb3811e744bd4ba6d59195cebde6f]). Expected version 2 got 0
I20250901 01:52:27.824335 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet c6f7c525f59b49938b697de98d4b5835 (table default.Table_db61e055dbd7450898e82c4c0c4763be [id=aabca67f6f754820977ac9c4c20f3e4f]). Expected version 1 got 0
I20250901 01:52:27.825114 1492 catalog_manager.cc:5582] T 081e2630e284498aa819f85bf8326315 P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: UNKNOWN } } }
I20250901 01:52:27.808602 1150 raft_consensus.cc:738] T 81ce448b148447ddbc41875d863e0518 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:27.831174 1150 consensus_queue.cc:260] T 81ce448b148447ddbc41875d863e0518 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.831841 1150 ts_tablet_manager.cc:1428] T 81ce448b148447ddbc41875d863e0518 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.026s user 0.001s sys 0.001s
I20250901 01:52:27.832031 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 081e2630e284498aa819f85bf8326315 (table default.Table_9e7ca61fcc324a8984bd1e470162c61f [id=8199118617cb4707bc3ae9b1363750f7]). Expected version 2 got 0
I20250901 01:52:27.832157 1150 tablet_bootstrap.cc:492] T 410d44e5f1974ed69e5dee65e98856a4 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:27.833828 1150 tablet_bootstrap.cc:654] T 410d44e5f1974ed69e5dee65e98856a4 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:27.847522 992 consensus_peers.cc:489] T a54af99bc10045e8b0b6868e5aed98b4 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Couldn't send request to peer 00c282f779c1477d920f24dc0dc16628. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a54af99bc10045e8b0b6868e5aed98b4. This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:27.834398 1492 catalog_manager.cc:5582] T 81c36693199945d98465579b509852e2 P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: UNKNOWN } } }
I20250901 01:52:27.848798 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 81ce448b148447ddbc41875d863e0518 (table default.Table_6e461d9cb0ae4dd095975bbc21533794 [id=49673fa73f8c436e9b8026efc4f3cbbc]). Expected version 1 got 0
I20250901 01:52:27.849187 2118 ts_tablet_manager.cc:1918] T 0dbdcb76eeba4fefa56e7835c23c0316 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:27.850062 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet d000ba05e12843b4925e622ab55471c0 (table default.Table_e0902dfbdd984f85b3fdac8d283731f7 [id=b43a92ed46164d199b445287ef039bc6]). Expected version 1 got 0
I20250901 01:52:27.850667 2118 log.cc:1199] T 0dbdcb76eeba4fefa56e7835c23c0316 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/0dbdcb76eeba4fefa56e7835c23c0316
I20250901 01:52:27.851169 970 heartbeater.cc:344] Connected to a master server at 127.0.104.62:43871
I20250901 01:52:27.851054 2118 ts_tablet_manager.cc:1939] T 0dbdcb76eeba4fefa56e7835c23c0316 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:27.853504 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 68) took 2629 ms. Trace:
I20250901 01:52:27.853611 2118 rpcz_store.cc:276] 0901 01:52:25.224182 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.224243 (+ 61us) service_pool.cc:225] Handling call
0901 01:52:27.752820 (+2528577us) ts_tablet_manager.cc:1145] Deleting tablet 0dbdcb76eeba4fefa56e7835c23c0316
0901 01:52:27.752828 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:27.804021 (+ 51193us) spinlock_profiling.cc:243] Waited 49.9 ms on lock 0x608000c5a150. stack: 00007facf6cb87d6 00007facf6cb8590 00007facf5cab3ad 0000000000972d94 0000000000972d5d 000000000097272a 00007fad063cccf8 00007fad063d2834 00007fad063d250d 0000000000a774dc 00007facf6bde74e 00007fad03924573 00007fad03a1b7a4 000000000096bf76 000000000096be51 0000000000af4f31
0901 01:52:27.814894 (+ 10873us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:27.849071 (+ 34177us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:27.853488 (+ 4417us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":114764032,"tablet-delete.queue_time_us":2528431}
I20250901 01:52:27.854121 2118 tablet_replica.cc:331] T 670d90a37b934579b353bb4e4820127b P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:27.854081 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 0dbdcb76eeba4fefa56e7835c23c0316 (table default.Table_97f8d5c204174c918a795498e47d404d [id=d74c256799d64024b869f8076b857f63]) successfully deleted
W20250901 01:52:27.854768 992 consensus_peers.cc:489] T f62d96c37cca4143a4b098b7c69979e6 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250901 01:52:27.854784 2118 raft_consensus.cc:2241] T 670d90a37b934579b353bb4e4820127b P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:27.856768 2118 raft_consensus.cc:2270] T 670d90a37b934579b353bb4e4820127b P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:27.859856 2118 ts_tablet_manager.cc:1905] T 670d90a37b934579b353bb4e4820127b P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:27.861171 1150 tablet_bootstrap.cc:492] T 410d44e5f1974ed69e5dee65e98856a4 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:27.861431 1150 ts_tablet_manager.cc:1397] T 410d44e5f1974ed69e5dee65e98856a4 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.001s
I20250901 01:52:27.862625 1150 raft_consensus.cc:357] T 410d44e5f1974ed69e5dee65e98856a4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.862920 1150 raft_consensus.cc:383] T 410d44e5f1974ed69e5dee65e98856a4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:27.863051 1150 raft_consensus.cc:738] T 410d44e5f1974ed69e5dee65e98856a4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:27.863472 1150 consensus_queue.cc:260] T 410d44e5f1974ed69e5dee65e98856a4 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.864631 1150 ts_tablet_manager.cc:1428] T 410d44e5f1974ed69e5dee65e98856a4 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250901 01:52:27.865839 1150 tablet_bootstrap.cc:492] T 858218c4427c41e980b2b5c969a5af40 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
W20250901 01:52:27.871066 992 consensus_peers.cc:489] T c763d252598745a088344aa0eb5fa3d3 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250901 01:52:27.874444 1150 tablet_bootstrap.cc:654] T 858218c4427c41e980b2b5c969a5af40 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:27.875591 2118 ts_tablet_manager.cc:1918] T 670d90a37b934579b353bb4e4820127b P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:27.875721 2118 log.cc:1199] T 670d90a37b934579b353bb4e4820127b P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/670d90a37b934579b353bb4e4820127b
I20250901 01:52:27.877286 2118 ts_tablet_manager.cc:1939] T 670d90a37b934579b353bb4e4820127b P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:27.859463 1492 catalog_manager.cc:5582] T a8483e7212d94daba7e5401256c33aef P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } }
I20250901 01:52:27.880568 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 616ff34e8d314e0aa0e11ce60be11789 (table default.Table_0451d0afa26f4d5f8d4214a946eea04f [id=5ee13d46e06a4fa59b1070547573e523]). Expected version 1 got 0
I20250901 01:52:27.881227 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet d0c3c363dda444519a17dca63ad9caa7 (table default.Table_42422c44ca254d8ca989134c2433391f [id=8c1b263824ec4c15a5ec424364c78895]). Expected version 3 got 0
I20250901 01:52:27.881702 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet e9f1f892658b4736859ff507536920a1 (table default.Table_8825dd28afbe464193eb1b668cc9326a [id=9aa62cb634314a0a849612c54c1220d9]). Expected version 1 got 0
I20250901 01:52:27.881939 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 44af4764d4f64e44aec84ccb84be6f5f (table default.Table_9896b980158948339a0b91f5882ed56b [id=a541b55be0dc420a8353ef4793b925b5]). Expected version 1 got 0
I20250901 01:52:27.882279 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 5f5396045979450193d461ba1334a5c9 (table default.Table_a89cc8e48563460baabc34c821ceae35 [id=9c3930e9fcfe4b49ba4bea67a2b0ad07]). Expected version 1 got 0
I20250901 01:52:27.882723 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 69) took 2658 ms. Trace:
I20250901 01:52:27.883116 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 9fbd44641d9e43c3b592e388e4adf350 (table default.Table_2fbca8463d6546c9b8f5b018fc60251d [id=6fdd3d817dc74551923021bd716ba58b]). Expected version 1 got 0
I20250901 01:52:27.883323 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet b164b4a67a9143ceaff69b0ce49d63fe (table default.Table_da2c70a8f2fe49a0a8184e88c0a9e0ea [id=c1b1e9e3876244f68d31c63ce7bdebf1]). Expected version 1 got 0
I20250901 01:52:27.883782 2118 rpcz_store.cc:276] 0901 01:52:25.224286 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.224541 (+ 255us) service_pool.cc:225] Handling call
0901 01:52:27.854074 (+2629533us) ts_tablet_manager.cc:1145] Deleting tablet 670d90a37b934579b353bb4e4820127b
0901 01:52:27.854084 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:27.863310 (+ 9226us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:27.875487 (+ 12177us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:27.882703 (+ 7216us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2629383}
W20250901 01:52:27.883955 992 consensus_peers.cc:489] T a8483e7212d94daba7e5401256c33aef P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:27.884296 2118 tablet_replica.cc:331] T 1286c0ab85704837a99b6e5a62184d77 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:27.884544 2118 raft_consensus.cc:2241] T 1286c0ab85704837a99b6e5a62184d77 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:27.885196 1151 tablet_bootstrap.cc:492] T e87d1281f35648b0b115b67c95e244e7 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:27.885423 2118 raft_consensus.cc:2270] T 1286c0ab85704837a99b6e5a62184d77 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:27.889281 1151 ts_tablet_manager.cc:1397] T e87d1281f35648b0b115b67c95e244e7 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.135s user 0.003s sys 0.015s
I20250901 01:52:27.890121 1151 raft_consensus.cc:357] T e87d1281f35648b0b115b67c95e244e7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.886263 2118 ts_tablet_manager.cc:1905] T 1286c0ab85704837a99b6e5a62184d77 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:27.884227 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 670d90a37b934579b353bb4e4820127b (table default.Table_2f09538b6fbe401f909105decb56e1cb [id=a90d1f10531042c3bb7a196b8be61c5f]) successfully deleted
I20250901 01:52:27.893178 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet ae21b9ae2e3b472eb67fb5f96388a50f (table default.Table_046289f393ff432aae0b033bdec1fb17 [id=db57b18a9ce24e068b362f5fbe940531]). Expected version 1 got 0
I20250901 01:52:27.893806 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 93bcac1fd95d44dc927abfca81692921 (table default.Table_b8f8ee74648c406eaf589d3ab80934db [id=b8f69313439543be9c11eea88dfa69c5]). Expected version 1 got 0
I20250901 01:52:27.896144 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 508406601c994e42a5c5b1f665e74957 (table default.Table_345b7f9fe8214aa6878f20a166125364 [id=4aabc328f6374ec5b1dea26f778208b2]). Expected version 1 got 0
I20250901 01:52:27.896589 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 7f46e9ddadaf4ff986d8c991178daa0c (table default.Table_9978497447394e5694ae928412634ecb [id=ea928501fe954cd0b81d38add4ef7128]). Expected version 2 got 0
I20250901 01:52:27.897755 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 33f637f6a8134215bb1995ec38440073 (table default.Table_3d5afc22c4e14d6990aea31564dfde20 [id=557afe29976d40f6bfb6ae4ca348f887]). Expected version 2 got 0
I20250901 01:52:27.899619 1492 catalog_manager.cc:5582] T f80f13db501c42c7950499df8fcb6fc8 P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: UNKNOWN } } }
I20250901 01:52:27.900113 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet f80f13db501c42c7950499df8fcb6fc8 (table default.Table_ce86974fd8974de9a5a737954fac2ad7 [id=ff772d20b287476f94716379b86d2a71]). Expected version 1 got 0
I20250901 01:52:27.900450 1492 catalog_manager.cc:5582] T 3618a3eb4f174c43805ce7dc0e5315cb P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: UNKNOWN } } }
I20250901 01:52:27.902091 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 3618a3eb4f174c43805ce7dc0e5315cb (table default.Table_83f2c0acedbc4c6982caf5313646e90b [id=31a2e50c32644321825da40176127015]). Expected version 1 got 0
I20250901 01:52:27.903070 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet d1f2d98712ff4ea6aec665512de068bd (table default.Table_82d91a1ba2eb4265896824028dec8d3a [id=3f7624d2901445e39155c12fa68e34e9]). Expected version 1 got 0
I20250901 01:52:27.903470 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 615b2306fca84003b264da9d5f95ddab (table default.Table_302f6e7cf97d46b9a621eb852e6aefe0 [id=cecdd8d6ca6d445c9fd82decd2a95366]). Expected version 2 got 0
I20250901 01:52:27.904460 1151 raft_consensus.cc:383] T e87d1281f35648b0b115b67c95e244e7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:27.904547 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 7304937048ca48a7bd153f3cdcfaeff0 (table default.Table_1e59568bdcd14662ae9d5e98293d74bd [id=8051adf4041f4a2686a0b8e3850ef29c]). Expected version 1 got 0
I20250901 01:52:27.904594 1151 raft_consensus.cc:738] T e87d1281f35648b0b115b67c95e244e7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:27.904799 1151 consensus_queue.cc:260] T e87d1281f35648b0b115b67c95e244e7 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.905486 1151 ts_tablet_manager.cc:1428] T e87d1281f35648b0b115b67c95e244e7 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
I20250901 01:52:27.905813 1151 tablet_bootstrap.cc:492] T 7a607fac57994001b2304e5d81c9decc P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:27.905766 1492 catalog_manager.cc:5582] T ad45c90ba0da43c9860745e534664a98 P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: UNKNOWN } } }
I20250901 01:52:27.907716 1151 tablet_bootstrap.cc:654] T 7a607fac57994001b2304e5d81c9decc P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:27.887301 2113 ts_tablet_manager.cc:1918] T 4410da9e95454ecf95faf80221b0a71c P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 01:52:27.908520 2113 log.cc:1199] T 4410da9e95454ecf95faf80221b0a71c P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/4410da9e95454ecf95faf80221b0a71c
I20250901 01:52:27.908996 2113 ts_tablet_manager.cc:1939] T 4410da9e95454ecf95faf80221b0a71c P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:27.911000 1150 tablet_bootstrap.cc:492] T 858218c4427c41e980b2b5c969a5af40 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:27.911265 1150 ts_tablet_manager.cc:1397] T 858218c4427c41e980b2b5c969a5af40 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.046s user 0.005s sys 0.002s
I20250901 01:52:27.912580 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet ad45c90ba0da43c9860745e534664a98 (table default.Table_1cd3f3fb6b244b3f8796bfca9f7622c8 [id=6fd739b0e725424ca844dbb652ea8028]). Expected version 2 got 0
I20250901 01:52:27.912500 1150 raft_consensus.cc:357] T 858218c4427c41e980b2b5c969a5af40 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.912894 1150 raft_consensus.cc:383] T 858218c4427c41e980b2b5c969a5af40 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:27.912946 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet fecf7d538b4c4c619182d5eccb44dd8b (table default.Table_1b5729d9768546fe99eaafc4e2604f42 [id=02d0056abe2146ac9af81ebf4447b99d]). Expected version 1 got 0
I20250901 01:52:27.913015 2118 ts_tablet_manager.cc:1918] T 1286c0ab85704837a99b6e5a62184d77 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:27.913103 2118 log.cc:1199] T 1286c0ab85704837a99b6e5a62184d77 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/1286c0ab85704837a99b6e5a62184d77
I20250901 01:52:27.913132 1150 raft_consensus.cc:738] T 858218c4427c41e980b2b5c969a5af40 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:27.913400 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 91) took 2857 ms. Trace:
I20250901 01:52:27.913445 2118 ts_tablet_manager.cc:1939] T 1286c0ab85704837a99b6e5a62184d77 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:27.913507 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 8ca5cf4e99014c0d84a3acad4ef713af (table default.Table_ed6fd284be6f4ae3a008e21e7ce17930 [id=303e76c918bc4c39aa47a4d24849a358]). Expected version 1 got 0
I20250901 01:52:27.913533 2113 rpcz_store.cc:276] 0901 01:52:25.056208 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.056552 (+ 344us) service_pool.cc:225] Handling call
0901 01:52:27.817869 (+2761317us) ts_tablet_manager.cc:1145] Deleting tablet 4410da9e95454ecf95faf80221b0a71c
0901 01:52:27.817879 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:27.851839 (+ 33960us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:27.887203 (+ 35364us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:27.913378 (+ 26175us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2724785}
I20250901 01:52:27.913748 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 72) took 2689 ms. Trace:
I20250901 01:52:27.913873 2113 tablet_replica.cc:331] T b11e8f77243e4764b96548d1132b8e5a P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:27.913816 1150 consensus_queue.cc:260] T 858218c4427c41e980b2b5c969a5af40 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.914032 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 4410da9e95454ecf95faf80221b0a71c (table default.Table_420b0979d6fd42da9b320456f4ccc9ce [id=d2004563c4374790954327f05c305cc6]) successfully deleted
I20250901 01:52:27.914083 2113 raft_consensus.cc:2241] T b11e8f77243e4764b96548d1132b8e5a P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:27.914270 2113 raft_consensus.cc:2270] T b11e8f77243e4764b96548d1132b8e5a P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:27.914808 2113 ts_tablet_manager.cc:1905] T b11e8f77243e4764b96548d1132b8e5a P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:27.915014 1150 ts_tablet_manager.cc:1428] T 858218c4427c41e980b2b5c969a5af40 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250901 01:52:27.916702 1150 tablet_bootstrap.cc:492] T da5f69c5b0ca4257ad2ff7918ee2af2e P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:27.917099 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 61ddcf86c9f34726939cc116434f0822 (table default.Table_63d0f2ead9d946d39ce40162ebd769d6 [id=f9dcf4937b1143248bfa9d88053fce17]). Expected version 3 got 0
I20250901 01:52:27.913823 2118 rpcz_store.cc:276] 0901 01:52:25.224701 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.224795 (+ 94us) service_pool.cc:225] Handling call
0901 01:52:27.884242 (+2659447us) ts_tablet_manager.cc:1145] Deleting tablet 1286c0ab85704837a99b6e5a62184d77
0901 01:52:27.884251 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:27.909944 (+ 25693us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:27.912921 (+ 2977us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:27.913729 (+ 808us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2659314}
I20250901 01:52:27.918641 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 858218c4427c41e980b2b5c969a5af40 (table default.Table_6d47266af0ed433eb1f2abbc81d18e69 [id=264cc469a73841e79538843479a165ab]). Expected version 1 got 0
I20250901 01:52:27.918851 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 1286c0ab85704837a99b6e5a62184d77 (table default.Table_c9401a0bf1b84645bdfd72bc9bcbe5b6 [id=be7660807d4745d19346469b223bea60]) successfully deleted
I20250901 01:52:27.918968 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 75) took 2693 ms. Trace:
I20250901 01:52:27.919206 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 89d40f0331114490b51de3ba6d773571 (table default.Table_4dacb3b1d2e94e42ac8937eb6eb58375 [id=8ee0aa1ba24042dd8ded004644913b9c]). Expected version 1 got 0
I20250901 01:52:27.920015 1483 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 85322a864d154f299f2d670fdd2a7bfb
W20250901 01:52:27.920217 1483 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 85322a864d154f299f2d670fdd2a7bfb: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:27.920918 901 tablet_service.cc:1515] Processing DeleteTablet for tablet 85322a864d154f299f2d670fdd2a7bfb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:27 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:27.921036 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 85322a864d154f299f2d670fdd2a7bfb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:27 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:27.921015 1483 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_749fcd4bf97d439c9fd49b138a52a3fd" } new_table_name: "default.Table_1adc70cccd2f4cb3a79df96861cb4181" modify_external_catalogs: true
I20250901 01:52:27.921334 1485 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet cb7d7e1796c0456f8de98fb812b9e374
W20250901 01:52:27.921514 1485 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet cb7d7e1796c0456f8de98fb812b9e374: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:27.922087 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_253971144e19425aa98d98c51ccdc504"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:27.919046 2118 rpcz_store.cc:276] 0901 01:52:25.225138 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.225199 (+ 61us) service_pool.cc:225] Handling call
0901 01:52:27.918852 (+2693653us) ts_tablet_manager.cc:1145] Deleting tablet 2ff914352c7c43de800f3d54ecf63072
0901 01:52:27.918862 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:27.918952 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2693477}
W20250901 01:52:27.924837 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 2ff914352c7c43de800f3d54ecf63072 because the tablet was not found. No further retry: Not found: Tablet not found: 2ff914352c7c43de800f3d54ecf63072
I20250901 01:52:27.925009 901 tablet_service.cc:1515] Processing DeleteTablet for tablet cb7d7e1796c0456f8de98fb812b9e374 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:27 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:27.925233 2118 tablet_replica.cc:331] T 822382a1781b4304bce29355ce83730e P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:27.925273 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet cb7d7e1796c0456f8de98fb812b9e374 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:27 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:27.925567 2118 raft_consensus.cc:2241] T 822382a1781b4304bce29355ce83730e P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:27.925745 2118 raft_consensus.cc:2270] T 822382a1781b4304bce29355ce83730e P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:27.925781 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e77fcc52e6d54ebdb9129addade6db5c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:27.926266 2118 ts_tablet_manager.cc:1905] T 822382a1781b4304bce29355ce83730e P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:27.926544 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_56a369ad5c7449008166ad00b325b824"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:27.927320 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9d422aa3837f47268356b2cd8ab732ca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:27.927413 1150 tablet_bootstrap.cc:654] T da5f69c5b0ca4257ad2ff7918ee2af2e P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:27.927955 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3256df71020d4cd49645bfa5233fabe1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:27.928916 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2a6b44f0a24f45378873b5243a3abe56"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:27.929678 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ce0f0a5ac4d34483bcd61b65892c9a1b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:27.930234 1151 tablet_bootstrap.cc:492] T 7a607fac57994001b2304e5d81c9decc P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:27.930389 1151 ts_tablet_manager.cc:1397] T 7a607fac57994001b2304e5d81c9decc P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.000s
I20250901 01:52:27.930619 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_bb2b5420ed084a27899f8b9f3ce23d34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:27.931142 1151 raft_consensus.cc:357] T 7a607fac57994001b2304e5d81c9decc P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.931324 1151 raft_consensus.cc:383] T 7a607fac57994001b2304e5d81c9decc P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:27.931380 1151 raft_consensus.cc:738] T 7a607fac57994001b2304e5d81c9decc P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:27.931413 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b1eee6632a074b97ad1239328e59ac7f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:27.931576 1151 consensus_queue.cc:260] T 7a607fac57994001b2304e5d81c9decc P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.932067 1151 ts_tablet_manager.cc:1428] T 7a607fac57994001b2304e5d81c9decc P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250901 01:52:27.932428 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_022337b9ae9746cdaa05426a9c5684de"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:27.932999 1151 tablet_bootstrap.cc:492] T 482a9a2cb81f448bb2823892e35920d4 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:27.933154 1485 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_3c7c628f35fb4d02bb79a34995f12938" } modify_external_catalogs: true
I20250901 01:52:27.933328 1485 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_3c7c628f35fb4d02bb79a34995f12938" } modify_external_catalogs: true
I20250901 01:52:27.934945 1151 tablet_bootstrap.cc:654] T 482a9a2cb81f448bb2823892e35920d4 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:27.935319 1482 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 45f6e8ba99b74898a2922b36768c6c36 consensus state has no leader
W20250901 01:52:27.935417 1486 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7954393c8c1c4457ab8bb723c03a9c8b consensus state has no leader
I20250901 01:52:27.935660 1486 master_service.cc:948] ReplaceTablet: received request to replace tablet 0cba5329ab31440999faec599d704633 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:27.935758 1482 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_1d63a9f86fe4488b92d5d607b102328c" } new_table_name: "default.Table_f610cb75906d4c5cb3b57cfdb80ecefc" modify_external_catalogs: true
I20250901 01:52:27.936259 1489 master_service.cc:948] ReplaceTablet: received request to replace tablet d10c193452954c198bba686169b8025b from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:27.936542 1491 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 81319e373c9a4475abceeeedb507aabc
I20250901 01:52:27.936726 1491 master_service.cc:948] ReplaceTablet: received request to replace tablet 50e3da53433f4c14b57db8a733fa53bc from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:27.937004 1490 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 3697821cd29446bb908b020599df0db9
W20250901 01:52:27.937175 1490 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 3697821cd29446bb908b020599df0db9: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:27.937726 1490 master_service.cc:948] ReplaceTablet: received request to replace tablet 9fbd44641d9e43c3b592e388e4adf350 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:27.941385 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet 3697821cd29446bb908b020599df0db9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:27 UTC) from {username='slave'} at 127.0.0.1:35946
W20250901 01:52:27.942677 992 consensus_peers.cc:489] T 4fd0e4f9d86c4362af96ce9964b72328 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250901 01:52:27.942823 901 tablet_service.cc:1515] Processing DeleteTablet for tablet 3697821cd29446bb908b020599df0db9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:27 UTC) from {username='slave'} at 127.0.0.1:34472
W20250901 01:52:27.944279 992 consensus_peers.cc:489] T bb0cf5a6a27045a08df6845d7473184d P fd27ab6665304aaea53adbd102dd4f58 -> Peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Couldn't send request to peer 00c282f779c1477d920f24dc0dc16628. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bb0cf5a6a27045a08df6845d7473184d. This is attempt 1: this message will repeat every 5th retry.
W20250901 01:52:27.945288 1483 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0e4009963cfe42d6ba164fc828897f09 consensus state has no leader
I20250901 01:52:27.945642 1483 master_service.cc:948] ReplaceTablet: received request to replace tablet 7f46e9ddadaf4ff986d8c991178daa0c from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:27.953639 1151 tablet_bootstrap.cc:492] T 482a9a2cb81f448bb2823892e35920d4 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:27.953828 1151 ts_tablet_manager.cc:1397] T 482a9a2cb81f448bb2823892e35920d4 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.002s
I20250901 01:52:27.953944 2323 raft_consensus.cc:491] T e87d1281f35648b0b115b67c95e244e7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:27.954083 2323 raft_consensus.cc:513] T e87d1281f35648b0b115b67c95e244e7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.954646 2323 leader_election.cc:290] T e87d1281f35648b0b115b67c95e244e7 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:27.954790 1151 raft_consensus.cc:357] T 482a9a2cb81f448bb2823892e35920d4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.955027 1151 raft_consensus.cc:383] T 482a9a2cb81f448bb2823892e35920d4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:27.955097 1151 raft_consensus.cc:738] T 482a9a2cb81f448bb2823892e35920d4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:27.955271 1151 consensus_queue.cc:260] T 482a9a2cb81f448bb2823892e35920d4 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:27.955823 858 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111) [suppressed 116 similar messages]
I20250901 01:52:27.955855 1151 ts_tablet_manager.cc:1428] T 482a9a2cb81f448bb2823892e35920d4 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250901 01:52:27.956135 1151 tablet_bootstrap.cc:492] T 6789505328b1409ea288e8b4eaf94935 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:27.957567 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e87d1281f35648b0b115b67c95e244e7" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:27.957726 1057 raft_consensus.cc:2466] T e87d1281f35648b0b115b67c95e244e7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00c282f779c1477d920f24dc0dc16628 in term 0.
W20250901 01:52:27.959447 858 leader_election.cc:336] T e87d1281f35648b0b115b67c95e244e7 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:27.960060 1482 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5d68ea97c71942c7b55b967ba870e0d7 consensus state has no leader
I20250901 01:52:27.960529 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_addc03378d6b49f795d373c6cdc47dec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:27.961488 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2a84ba2380874c58b8d4fb3af6857068"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:27.962242 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c0d36515bf764658872a6b1745f93e85"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:27.962934 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_711cf44017ab4607bd5522cc266f5748"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:27.964176 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_14c56b7dfad84001b7f607e4bffd56e1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:27.966826 859 leader_election.cc:304] T e87d1281f35648b0b115b67c95e244e7 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
W20250901 01:52:27.967005 992 consensus_peers.cc:489] T 081e2630e284498aa819f85bf8326315 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:27.968889 2118 ts_tablet_manager.cc:1918] T 822382a1781b4304bce29355ce83730e P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:27.968995 2118 log.cc:1199] T 822382a1781b4304bce29355ce83730e P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/822382a1781b4304bce29355ce83730e
I20250901 01:52:27.969372 2118 ts_tablet_manager.cc:1939] T 822382a1781b4304bce29355ce83730e P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:27.969733 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 78) took 2744 ms. Trace:
I20250901 01:52:27.969803 2118 rpcz_store.cc:276] 0901 01:52:25.225673 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.225716 (+ 43us) service_pool.cc:225] Handling call
0901 01:52:27.925058 (+2699342us) ts_tablet_manager.cc:1145] Deleting tablet 822382a1781b4304bce29355ce83730e
0901 01:52:27.925069 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:27.937633 (+ 12564us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:27.968769 (+ 31136us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:27.969715 (+ 946us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2699163}
I20250901 01:52:27.970213 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 822382a1781b4304bce29355ce83730e (table default.Table_b300f31bdab14e24ad40f265ac4de311 [id=4a54bc023da7481fbb0ce573be49f4c3]) successfully deleted
I20250901 01:52:27.970613 1151 tablet_bootstrap.cc:654] T 6789505328b1409ea288e8b4eaf94935 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:27.970919 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7ee3917ccc724b0eb1c77e753f6cf358"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:27.971028 2323 raft_consensus.cc:2802] T e87d1281f35648b0b115b67c95e244e7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:27.971109 2323 raft_consensus.cc:491] T e87d1281f35648b0b115b67c95e244e7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:27.971207 2323 raft_consensus.cc:3058] T e87d1281f35648b0b115b67c95e244e7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:27.971662 2118 tablet_replica.cc:331] T 579b9cdf819f4609a532e96bec093fd5 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:27.971858 2118 raft_consensus.cc:2241] T 579b9cdf819f4609a532e96bec093fd5 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:27.972020 2118 raft_consensus.cc:2270] T 579b9cdf819f4609a532e96bec093fd5 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:27.972226 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3533c8c4c7be4495bd0417ff83c9963d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:27.972616 2118 ts_tablet_manager.cc:1905] T 579b9cdf819f4609a532e96bec093fd5 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:27.974038 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b88d9ab8abd14781b2f7f1ec4b568d58"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:27.974900 2323 raft_consensus.cc:513] T e87d1281f35648b0b115b67c95e244e7 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:27.977351 858 leader_election.cc:336] T e87d1281f35648b0b115b67c95e244e7 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:27.977818 2323 leader_election.cc:290] T e87d1281f35648b0b115b67c95e244e7 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:27.978040 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e87d1281f35648b0b115b67c95e244e7" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58"
I20250901 01:52:27.978209 1057 raft_consensus.cc:3058] T e87d1281f35648b0b115b67c95e244e7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:27.980988 1057 raft_consensus.cc:2466] T e87d1281f35648b0b115b67c95e244e7 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c282f779c1477d920f24dc0dc16628 in term 1.
I20250901 01:52:27.981410 859 leader_election.cc:304] T e87d1281f35648b0b115b67c95e244e7 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:27.981899 2323 raft_consensus.cc:2802] T e87d1281f35648b0b115b67c95e244e7 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:27.981962 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4df6aa57648e47efb9dbb3bc3d1e1c3a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:27.982285 2323 raft_consensus.cc:695] T e87d1281f35648b0b115b67c95e244e7 P 00c282f779c1477d920f24dc0dc16628 [term 1 LEADER]: Becoming Leader. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Running, Role: LEADER
I20250901 01:52:27.982885 2357 raft_consensus.cc:491] T 410d44e5f1974ed69e5dee65e98856a4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:27.982939 2113 ts_tablet_manager.cc:1918] T b11e8f77243e4764b96548d1132b8e5a P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250901 01:52:27.983035 2113 log.cc:1199] T b11e8f77243e4764b96548d1132b8e5a P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/b11e8f77243e4764b96548d1132b8e5a
I20250901 01:52:27.982985 2357 raft_consensus.cc:513] T 410d44e5f1974ed69e5dee65e98856a4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.983556 2357 leader_election.cc:290] T 410d44e5f1974ed69e5dee65e98856a4 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:27.982758 2323 consensus_queue.cc:237] T e87d1281f35648b0b115b67c95e244e7 P 00c282f779c1477d920f24dc0dc16628 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.985234 1488 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: default.Table_a4ebd8ab87c842f082da5cd04d1bddac [id=fd6d8e92d6e64a89a7d38aa761376955] alter complete (version 1)
W20250901 01:52:27.985752 858 leader_election.cc:336] T 410d44e5f1974ed69e5dee65e98856a4 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:27.984331 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a1d404f0e1464038bde23f6cd5403b1e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:27.988926 1150 tablet_bootstrap.cc:492] T da5f69c5b0ca4257ad2ff7918ee2af2e P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:27.989625 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f5ea9be7a4e2490bacf26802f288f7ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:27.990361 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_492cf7cca33d4d96ac9198c0dc9ef375"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:27.991083 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2f5290490e6b46969e02d74fe9885c5b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:27.990619 1150 ts_tablet_manager.cc:1397] T da5f69c5b0ca4257ad2ff7918ee2af2e P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.074s user 0.004s sys 0.008s
I20250901 01:52:27.991770 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_38a7881b705c4b71a87ab6c8f0af14f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:27.997618 1150 raft_consensus.cc:357] T da5f69c5b0ca4257ad2ff7918ee2af2e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:27.998472 1150 raft_consensus.cc:383] T da5f69c5b0ca4257ad2ff7918ee2af2e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:27.999504 2323 raft_consensus.cc:491] T 6789505328b1409ea288e8b4eaf94935 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:27.999529 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_fef9bf141cf0480d9edf095bd20732cd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:27.999755 2323 raft_consensus.cc:513] T 6789505328b1409ea288e8b4eaf94935 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:28.000871 2323 leader_election.cc:290] T 6789505328b1409ea288e8b4eaf94935 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:28.002030 858 leader_election.cc:336] T 6789505328b1409ea288e8b4eaf94935 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:28.004585 2118 ts_tablet_manager.cc:1918] T 579b9cdf819f4609a532e96bec093fd5 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:28.005882 1485 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 96824e68bbf24b8287dbb683038ffaba
I20250901 01:52:27.999507 1150 raft_consensus.cc:738] T da5f69c5b0ca4257ad2ff7918ee2af2e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:28.010619 2118 log.cc:1199] T 579b9cdf819f4609a532e96bec093fd5 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/579b9cdf819f4609a532e96bec093fd5
I20250901 01:52:28.011276 2118 ts_tablet_manager.cc:1939] T 579b9cdf819f4609a532e96bec093fd5 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:28.011188 1150 consensus_queue.cc:260] T da5f69c5b0ca4257ad2ff7918ee2af2e P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:28.012168 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 79) took 2786 ms. Trace:
I20250901 01:52:28.013885 1150 ts_tablet_manager.cc:1428] T da5f69c5b0ca4257ad2ff7918ee2af2e P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.022s user 0.001s sys 0.002s
I20250901 01:52:28.016896 2118 rpcz_store.cc:276] 0901 01:52:25.225806 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.225927 (+ 121us) service_pool.cc:225] Handling call
0901 01:52:27.971606 (+2745679us) ts_tablet_manager.cc:1145] Deleting tablet 579b9cdf819f4609a532e96bec093fd5
0901 01:52:27.971616 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:27.981627 (+ 10011us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:28.004441 (+ 22814us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:28.012114 (+ 7673us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2745528}
I20250901 01:52:28.017324 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 579b9cdf819f4609a532e96bec093fd5 (table default.Table_b0e916c76a7a4766aa0887cc8691488f [id=cf61137a852c48748a29f833f1860da7]) successfully deleted
I20250901 01:52:28.017571 1150 tablet_bootstrap.cc:492] T dabebc77f72e4bb19e264be2f496202a P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
W20250901 01:52:28.017800 1488 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 435b1cca5a7d4d49959a310cb250b2a0 (table default.Table_b989608c88204be2b6f706a9426c47bb [id=126b298fef6c4653952d672adff1db7e], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet 435b1cca5a7d4d49959a310cb250b2a0 consensus state has no leader
I20250901 01:52:28.021754 1484 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: default.Table_e53182d58e5244a9b7ba33cb48c939ba [id=cc26c89db2484a6fad919231116285f2] alter complete (version 3)
I20250901 01:52:28.023022 1150 tablet_bootstrap.cc:654] T dabebc77f72e4bb19e264be2f496202a P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:28.025090 2323 raft_consensus.cc:491] T 3d90c6ef06d1436d8e2bb1333e7d9985 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:28.025220 2323 raft_consensus.cc:513] T 3d90c6ef06d1436d8e2bb1333e7d9985 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:28.025880 2323 leader_election.cc:290] T 3d90c6ef06d1436d8e2bb1333e7d9985 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:28.027477 858 leader_election.cc:336] T 3d90c6ef06d1436d8e2bb1333e7d9985 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:28.017606 2118 tablet_replica.cc:331] T d77aadd7291647a0bae7c421f244c252 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:28.028440 2118 raft_consensus.cc:2241] T d77aadd7291647a0bae7c421f244c252 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:28.029044 2118 raft_consensus.cc:2270] T d77aadd7291647a0bae7c421f244c252 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:28.031589 2118 ts_tablet_manager.cc:1905] T d77aadd7291647a0bae7c421f244c252 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:28.035414 969 heartbeater.cc:499] Master 127.0.104.60:44231 was elected leader, sending a full tablet report...
I20250901 01:52:28.042137 2412 raft_consensus.cc:491] T 7a607fac57994001b2304e5d81c9decc P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:28.042285 2412 raft_consensus.cc:513] T 7a607fac57994001b2304e5d81c9decc P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:28.043395 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "410d44e5f1974ed69e5dee65e98856a4" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:28.043840 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6789505328b1409ea288e8b4eaf94935" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:28.044835 859 leader_election.cc:343] T 410d44e5f1974ed69e5dee65e98856a4 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:28.044951 859 leader_election.cc:304] T 410d44e5f1974ed69e5dee65e98856a4 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:28.044989 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d90c6ef06d1436d8e2bb1333e7d9985" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:28.045318 859 leader_election.cc:343] T 6789505328b1409ea288e8b4eaf94935 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:28.045446 859 leader_election.cc:304] T 6789505328b1409ea288e8b4eaf94935 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:28.045840 2357 raft_consensus.cc:2747] T 410d44e5f1974ed69e5dee65e98856a4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:28.045993 2357 raft_consensus.cc:2747] T 6789505328b1409ea288e8b4eaf94935 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:28.046368 2412 leader_election.cc:290] T 7a607fac57994001b2304e5d81c9decc P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:28.045116 1057 raft_consensus.cc:2466] T 3d90c6ef06d1436d8e2bb1333e7d9985 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00c282f779c1477d920f24dc0dc16628 in term 0.
I20250901 01:52:28.047497 2323 raft_consensus.cc:491] T 33f637f6a8134215bb1995ec38440073 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:28.047631 2323 raft_consensus.cc:513] T 33f637f6a8134215bb1995ec38440073 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:28.048166 2323 leader_election.cc:290] T 33f637f6a8134215bb1995ec38440073 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:28.048839 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet 8dd587304e6342edbe96c9897b40e5c9 with delete_type TABLET_DATA_DELETED (replaced by tablet 05630a72ddae4de0b890dece2feacc31) from {username='slave'} at 127.0.0.1:35946
W20250901 01:52:28.049228 992 consensus_peers.cc:489] T 822382a1781b4304bce29355ce83730e P fd27ab6665304aaea53adbd102dd4f58 -> Peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Couldn't send request to peer 00c282f779c1477d920f24dc0dc16628. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 822382a1781b4304bce29355ce83730e. This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:28.048926 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet f50f33de1b654663b9ef8a91e3436f11 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:13 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:28.049889 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "33f637f6a8134215bb1995ec38440073" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:28.049993 859 leader_election.cc:304] T 3d90c6ef06d1436d8e2bb1333e7d9985 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:28.050345 2323 raft_consensus.cc:2802] T 3d90c6ef06d1436d8e2bb1333e7d9985 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:28.050417 2323 raft_consensus.cc:491] T 3d90c6ef06d1436d8e2bb1333e7d9985 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250901 01:52:28.050455 858 leader_election.cc:336] T 33f637f6a8134215bb1995ec38440073 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:28.050489 2323 raft_consensus.cc:3058] T 3d90c6ef06d1436d8e2bb1333e7d9985 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
W20250901 01:52:28.050639 992 leader_election.cc:343] T 7a607fac57994001b2304e5d81c9decc P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 7a607fac57994001b2304e5d81c9decc
I20250901 01:52:28.052956 2323 raft_consensus.cc:513] T 3d90c6ef06d1436d8e2bb1333e7d9985 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:28.053892 2323 leader_election.cc:290] T 3d90c6ef06d1436d8e2bb1333e7d9985 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:28.054198 859 leader_election.cc:343] T 33f637f6a8134215bb1995ec38440073 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:28.054280 859 leader_election.cc:304] T 33f637f6a8134215bb1995ec38440073 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:28.054602 2323 raft_consensus.cc:2747] T 33f637f6a8134215bb1995ec38440073 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:28.055233 858 leader_election.cc:336] T 3d90c6ef06d1436d8e2bb1333e7d9985 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:28.055963 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 74b709cb5ec64006b2f12bead8217d80 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:35946
W20250901 01:52:28.056491 992 leader_election.cc:336] T 7a607fac57994001b2304e5d81c9decc P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:28.056607 992 leader_election.cc:304] T 7a607fac57994001b2304e5d81c9decc P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:28.056856 2412 raft_consensus.cc:2747] T 7a607fac57994001b2304e5d81c9decc P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:28.057281 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d90c6ef06d1436d8e2bb1333e7d9985" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58"
I20250901 01:52:28.057436 1057 raft_consensus.cc:3058] T 3d90c6ef06d1436d8e2bb1333e7d9985 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:28.060288 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 64fb46b942ff4e4c8749986ecc3dfe65 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:28.060667 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 66a48a8ebe154b2cac89c234eaa378ff with delete_type TABLET_DATA_DELETED (replaced by tablet 15e4d972e8254cf899c3e0b71b6e2bd9) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:28.061245 1151 tablet_bootstrap.cc:492] T 6789505328b1409ea288e8b4eaf94935 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:28.061420 1151 ts_tablet_manager.cc:1397] T 6789505328b1409ea288e8b4eaf94935 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.105s user 0.009s sys 0.005s
I20250901 01:52:28.061683 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e742ee3cfef24a14b4935e2b666f989d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.061892 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_6f736b31ed0f4492a59b5e614bfd1ed1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.063241 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_69ec6944366a4c7993aebae2724fa2e1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.063401 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9dc418daf2614c04bb356f6c7600b11f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.064146 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_640696d2ce704cfda65df7ae5c0a23fd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.064597 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3a5fae3c5b0d4d20bfaa52ed6a5b46f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.065083 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8855fcf6afa74a5992a11d7c50228139"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.065214 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_771c6930c9e8468e8f42cbabd429c700"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.061702 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_baeb5546a5bd4ab9bbaf7ba89615f13c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.072954 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ce68920ac676416c952a6751b146d209"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.073855 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_03f8dd1c4b7547c090d781bdd05433d2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.074518 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b205da3ad8274db58ef24597afafbfad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.075143 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5465f7e66f6b492e998ec7544d4f8ffe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.075734 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a765e78d51aa459aac877a4ef617f36d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.076401 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4856c7a279f54b8b847c3cace79617c5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.082218 1150 tablet_bootstrap.cc:492] T dabebc77f72e4bb19e264be2f496202a P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:28.082414 1150 ts_tablet_manager.cc:1397] T dabebc77f72e4bb19e264be2f496202a P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.065s user 0.005s sys 0.006s
I20250901 01:52:28.076984 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_49cde8670f9f4b19b0d11f6e32cbf66b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.085472 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_414204fa44f546be9965f6f5cd484ec5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.086156 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9f0caabe35204d5ca0fb5e021e61da44"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.086828 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9a4a09b3cfc84f6e859456c1d8f0979f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.087541 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8aabc90c980e45edbc98bdd3a2b31aa0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.087680 1150 raft_consensus.cc:357] T dabebc77f72e4bb19e264be2f496202a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:28.087936 1150 raft_consensus.cc:383] T dabebc77f72e4bb19e264be2f496202a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:28.088006 1150 raft_consensus.cc:738] T dabebc77f72e4bb19e264be2f496202a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:28.088171 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_eb94cb674bbf40b19bb32261adfc9638"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.088318 1150 consensus_queue.cc:260] T dabebc77f72e4bb19e264be2f496202a P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:28.077195 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e90db847053c4d5caa71ec1ba59eab72"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.089465 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ce1eaea2f97c481389170dd1284addbf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.064168 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1d0b45b9307f4482b0eb0e812f7f7a70"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.090168 1150 ts_tablet_manager.cc:1428] T dabebc77f72e4bb19e264be2f496202a P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250901 01:52:28.092250 1150 tablet_bootstrap.cc:492] T 26c8ed083cea4dc0acca80d00b11b29b P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:28.098357 2118 ts_tablet_manager.cc:1918] T d77aadd7291647a0bae7c421f244c252 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:28.099610 2118 log.cc:1199] T d77aadd7291647a0bae7c421f244c252 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/d77aadd7291647a0bae7c421f244c252
I20250901 01:52:28.102504 2118 ts_tablet_manager.cc:1939] T d77aadd7291647a0bae7c421f244c252 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:28.103525 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e396ab940d284673855a71c9308f14d7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.104493 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4105974cf9d340c182187ca783a064ec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.105715 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_00b9d8218de24ef79497d18f18bfa417"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.106509 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e3f168c3c8794baeb04068c469d2c01c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.106415 1150 tablet_bootstrap.cc:654] T 26c8ed083cea4dc0acca80d00b11b29b P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:28.107689 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_700c476c15ce47e18f79b7916eaa7c1e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.108444 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5730fbd18a204a9b8713ff320bca7908"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.109344 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_cc983530257a49f784b9ff8581946613"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.110136 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b57a506ecb3f4837923d13fc7898dfe1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.110451 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 81) took 2884 ms. Trace:
I20250901 01:52:28.111159 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b99e8b898e334fd498aa25412af3c7f2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.111034 2118 rpcz_store.cc:276] 0901 01:52:25.226055 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.226104 (+ 49us) service_pool.cc:225] Handling call
0901 01:52:28.017557 (+2791453us) ts_tablet_manager.cc:1145] Deleting tablet d77aadd7291647a0bae7c421f244c252
0901 01:52:28.017566 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:28.050858 (+ 33292us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:28.097604 (+ 46746us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:28.110062 (+ 12458us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2791281}
I20250901 01:52:28.112721 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet d77aadd7291647a0bae7c421f244c252 (table default.Table_eacb38fbc98046dfac27344cfbed0a7c [id=015239c5e105476ca8bf6198560f0feb]) successfully deleted
I20250901 01:52:28.113894 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 64) took 2890 ms. Trace:
W20250901 01:52:28.118263 992 consensus_peers.cc:489] T d77aadd7291647a0bae7c421f244c252 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Couldn't send request to peer 00c282f779c1477d920f24dc0dc16628. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d77aadd7291647a0bae7c421f244c252. This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:28.118827 2118 rpcz_store.cc:276] 0901 01:52:25.223254 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.223295 (+ 41us) service_pool.cc:225] Handling call
0901 01:52:28.113107 (+2889812us) ts_tablet_manager.cc:1145] Deleting tablet e1c8fe96a6ba4edba4f09c00f7add75e
0901 01:52:28.113117 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:28.113715 (+ 598us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2840215}
W20250901 01:52:28.120010 1472 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet e1c8fe96a6ba4edba4f09c00f7add75e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e1c8fe96a6ba4edba4f09c00f7add75e already in progress: creating tablet
I20250901 01:52:28.120263 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c47e93f4b6f24bf99979966c402550df"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.120766 2118 tablet_replica.cc:331] T 46bfbdff3f904715a8c24907b20fcab0 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:28.121222 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_811d5170cf854a92b1b84d4e37dcf9cc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.122005 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_47d73161a93849079bd8867ebe863001"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.121940 2118 raft_consensus.cc:2241] T 46bfbdff3f904715a8c24907b20fcab0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:28.122812 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_937100eb52fa4b40843ec75b1a4fe601"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.122769 2118 raft_consensus.cc:2270] T 46bfbdff3f904715a8c24907b20fcab0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:28.123694 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_22cfd7713fe44cb78734286b7fdf7041"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.124819 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9f243449f70f45c59dd317cbd67d68bb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.125669 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f56155c4aee84bf2bc8b9aa5557470b5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.125785 2118 ts_tablet_manager.cc:1905] T 46bfbdff3f904715a8c24907b20fcab0 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:28.126431 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e8c92f230e854f429e590481650c2e72"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.127317 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9f3482f753534530954dd83a419d73a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.132021 2323 raft_consensus.cc:491] T e8e7d2a72f704f52a8f8d46911a8a9e4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:28.132161 2323 raft_consensus.cc:513] T e8e7d2a72f704f52a8f8d46911a8a9e4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:28.132766 2323 leader_election.cc:290] T e8e7d2a72f704f52a8f8d46911a8a9e4 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:28.134907 858 leader_election.cc:336] T e8e7d2a72f704f52a8f8d46911a8a9e4 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:28.135311 1491 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_1bdd9041d74045cab6f23de11bf4dd09" } modify_external_catalogs: true
I20250901 01:52:28.135548 1491 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_1bdd9041d74045cab6f23de11bf4dd09" } modify_external_catalogs: true
I20250901 01:52:28.136890 1485 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_f611c1794e564a0496a761da46171d0e" } modify_external_catalogs: true
I20250901 01:52:28.137065 1485 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_f611c1794e564a0496a761da46171d0e" } modify_external_catalogs: true
I20250901 01:52:28.140892 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet f3962b471bdb49aea6e9fa95f1f9dfe2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:17 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:28.141094 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet b6daeff0070a4db08e90020752d21c29 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:14 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:28.141194 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 4410da9e95454ecf95faf80221b0a71c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:24 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:28.141302 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 0cb3262973934c8689b851adad60afe0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:16 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:28.141439 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet 3fda93767e494a4684978c8522b829dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:15 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:28.141561 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet b85d64e02cc646e380581ac9f59dee9e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:28.141664 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet 088ef4861dc14f81ae470c3aaee5b4e1 with delete_type TABLET_DATA_DELETED (replaced by tablet 6ce9521b925b4f97bb4a810ce77e147e) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:28.141794 1038 tablet_service.cc:1515] Processing DeleteTablet for tablet e2aa32450a584395ae07b49f76a7b26d with delete_type TABLET_DATA_DELETED (replaced by tablet 7304937048ca48a7bd153f3cdcfaeff0) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:28.141943 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet e8e7d2a72f704f52a8f8d46911a8a9e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:28.142068 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 75345ea2a3dc40fc8c9f8090d8839d2d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:28.142187 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 5a8c21aee9f84216b6f90538a5bf4c17 with delete_type TABLET_DATA_DELETED (replaced by tablet 42c1156e8dd84c6fa1396b46de13eb6f) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:28.142324 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet b541a6e14f114ab29a864cca11806316 with delete_type TABLET_DATA_DELETED (replaced by tablet 2599f98abfdc4871b573edfc2ad97d22) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:28.142428 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet b633b461dbf34e13bc6e1cc08e6d20c1 with delete_type TABLET_DATA_DELETED (replaced by tablet 4855448edb4b40c184838aa7f81c112b) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:28.142536 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 50472a7aa2f04e0fbd65ac6c86c34b3e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:20 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:28.142675 1035 tablet_service.cc:1515] Processing DeleteTablet for tablet 20f7bfab18134e079e91c52c89468cc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:17 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:28.142822 1036 tablet_service.cc:1515] Processing DeleteTablet for tablet 56f2c4d2e6ab4bb4bdd6d025843589b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:28.142887 1150 tablet_bootstrap.cc:492] T 26c8ed083cea4dc0acca80d00b11b29b P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:28.142963 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 879fa66e55544118aafe504587393c66 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:23 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:28.143083 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet ef58f477ebeb49639ed7d071e1a62c10 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:13 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:28.143065 1150 ts_tablet_manager.cc:1397] T 26c8ed083cea4dc0acca80d00b11b29b P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.051s user 0.003s sys 0.002s
I20250901 01:52:28.143175 1034 tablet_service.cc:1515] Processing DeleteTablet for tablet f98d1d8d660443cb8066a547e5d9b77f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:23 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:28.143265 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet e1c8fe96a6ba4edba4f09c00f7add75e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:28.143353 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 60285f2ba037433999f5455fb9cc7acd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:13 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:28.143509 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet dd694cfa09564d3da5d279c0861a4c32 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:12 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:28.143674 1033 tablet_service.cc:1515] Processing DeleteTablet for tablet 4fca1435d82447f289fda4035aa2ed89 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:20 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:28.144232 1150 raft_consensus.cc:357] T 26c8ed083cea4dc0acca80d00b11b29b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:28.144454 1150 raft_consensus.cc:383] T 26c8ed083cea4dc0acca80d00b11b29b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:28.144527 1150 raft_consensus.cc:738] T 26c8ed083cea4dc0acca80d00b11b29b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:28.144878 1150 consensus_queue.cc:260] T 26c8ed083cea4dc0acca80d00b11b29b P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:28.145222 1051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e8e7d2a72f704f52a8f8d46911a8a9e4" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:28.145242 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7ce6ebaccdea4919a09034dcb5b3fcc3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.145848 1150 ts_tablet_manager.cc:1428] T 26c8ed083cea4dc0acca80d00b11b29b P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250901 01:52:28.146288 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_64806c5992544d858968ef335eb2eef6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:28.149890 859 leader_election.cc:343] T e8e7d2a72f704f52a8f8d46911a8a9e4 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:28.150003 859 leader_election.cc:304] T e8e7d2a72f704f52a8f8d46911a8a9e4 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:28.150206 2323 raft_consensus.cc:2747] T e8e7d2a72f704f52a8f8d46911a8a9e4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:28.152153 1150 tablet_bootstrap.cc:492] T 617cc33d0d23439cb5f1e0c31bf11281 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:28.155015 2113 ts_tablet_manager.cc:1939] T b11e8f77243e4764b96548d1132b8e5a P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:28.155794 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 92) took 3096 ms. Trace:
I20250901 01:52:28.155879 2113 rpcz_store.cc:276] 0901 01:52:25.059555 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.059612 (+ 57us) service_pool.cc:225] Handling call
0901 01:52:27.913829 (+2854217us) ts_tablet_manager.cc:1145] Deleting tablet b11e8f77243e4764b96548d1132b8e5a
0901 01:52:27.913839 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:27.953008 (+ 39169us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:27.982841 (+ 29833us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:28.155773 (+172932us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2820666}
I20250901 01:52:28.156723 1150 tablet_bootstrap.cc:654] T 617cc33d0d23439cb5f1e0c31bf11281 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:28.158047 2113 tablet_replica.cc:331] T 49d30da95b14400b845270122487a2a0 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:28.158226 2113 raft_consensus.cc:2241] T 49d30da95b14400b845270122487a2a0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:28.159014 2113 raft_consensus.cc:2270] T 49d30da95b14400b845270122487a2a0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:28.159507 2113 ts_tablet_manager.cc:1905] T 49d30da95b14400b845270122487a2a0 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:28.162220 1151 raft_consensus.cc:357] T 6789505328b1409ea288e8b4eaf94935 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:28.162397 1151 raft_consensus.cc:383] T 6789505328b1409ea288e8b4eaf94935 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:28.163182 1151 raft_consensus.cc:738] T 6789505328b1409ea288e8b4eaf94935 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:28.163383 1151 consensus_queue.cc:260] T 6789505328b1409ea288e8b4eaf94935 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:28.164146 1151 ts_tablet_manager.cc:1428] T 6789505328b1409ea288e8b4eaf94935 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.102s user 0.001s sys 0.001s
I20250901 01:52:28.166420 1057 raft_consensus.cc:2466] T 3d90c6ef06d1436d8e2bb1333e7d9985 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c282f779c1477d920f24dc0dc16628 in term 1.
W20250901 01:52:28.166782 1488 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.3:51295 (request call id 181) took 1974 ms (client timeout 1000 ms). Trace:
I20250901 01:52:28.164410 1151 tablet_bootstrap.cc:492] T 05630a72ddae4de0b890dece2feacc31 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:28.169819 2459 tablet.cc:1722] T 5adf6b492640417ca071c94702b48cb9 P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250901 01:52:28.170092 1151 tablet_bootstrap.cc:654] T 05630a72ddae4de0b890dece2feacc31 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:28.171346 859 leader_election.cc:304] T 3d90c6ef06d1436d8e2bb1333e7d9985 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:28.171617 2323 raft_consensus.cc:2802] T 3d90c6ef06d1436d8e2bb1333e7d9985 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:28.171736 2323 raft_consensus.cc:695] T 3d90c6ef06d1436d8e2bb1333e7d9985 P 00c282f779c1477d920f24dc0dc16628 [term 1 LEADER]: Becoming Leader. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Running, Role: LEADER
I20250901 01:52:28.171948 2323 consensus_queue.cc:237] T 3d90c6ef06d1436d8e2bb1333e7d9985 P 00c282f779c1477d920f24dc0dc16628 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:28.187290 2357 raft_consensus.cc:491] T 268439ff8d3a4f7a89f3d6395f2079c5 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:28.187438 2357 raft_consensus.cc:513] T 268439ff8d3a4f7a89f3d6395f2079c5 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:28.188002 2357 leader_election.cc:290] T 268439ff8d3a4f7a89f3d6395f2079c5 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:28.189610 858 leader_election.cc:336] T 268439ff8d3a4f7a89f3d6395f2079c5 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:28.189425 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "268439ff8d3a4f7a89f3d6395f2079c5" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:28.176393 2113 ts_tablet_manager.cc:1918] T 49d30da95b14400b845270122487a2a0 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250901 01:52:28.190423 859 leader_election.cc:343] T 268439ff8d3a4f7a89f3d6395f2079c5 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:28.190528 859 leader_election.cc:304] T 268439ff8d3a4f7a89f3d6395f2079c5 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:28.190896 2357 raft_consensus.cc:2747] T 268439ff8d3a4f7a89f3d6395f2079c5 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:28.191823 2113 log.cc:1199] T 49d30da95b14400b845270122487a2a0 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/49d30da95b14400b845270122487a2a0
I20250901 01:52:28.192544 2113 ts_tablet_manager.cc:1939] T 49d30da95b14400b845270122487a2a0 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:28.193915 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 94) took 3114 ms. Trace:
I20250901 01:52:28.196166 2113 rpcz_store.cc:276] 0901 01:52:25.079360 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.079426 (+ 66us) service_pool.cc:225] Handling call
0901 01:52:28.156169 (+3076743us) ts_tablet_manager.cc:1145] Deleting tablet 49d30da95b14400b845270122487a2a0
0901 01:52:28.156179 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:28.173447 (+ 17268us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:28.176276 (+ 2829us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:28.193879 (+ 17603us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3062966}
I20250901 01:52:28.196899 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 49d30da95b14400b845270122487a2a0 (table default.Table_c96393e11f3a4cca8202ccc2df6635e0 [id=091e1ea029a4404e87561de62e038043]) successfully deleted
I20250901 01:52:28.197290 2113 tablet_replica.cc:331] T a774cc6ae3924d299127fd9ffb01ca57 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:28.198925 2357 raft_consensus.cc:491] T 26c8ed083cea4dc0acca80d00b11b29b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:28.199062 2357 raft_consensus.cc:513] T 26c8ed083cea4dc0acca80d00b11b29b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:28.199609 2357 leader_election.cc:290] T 26c8ed083cea4dc0acca80d00b11b29b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:28.200028 2113 raft_consensus.cc:2241] T a774cc6ae3924d299127fd9ffb01ca57 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250901 01:52:28.201524 858 leader_election.cc:336] T 26c8ed083cea4dc0acca80d00b11b29b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:28.201050 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26c8ed083cea4dc0acca80d00b11b29b" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:28.203464 859 leader_election.cc:343] T 26c8ed083cea4dc0acca80d00b11b29b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:28.203562 859 leader_election.cc:304] T 26c8ed083cea4dc0acca80d00b11b29b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:28.203908 2357 raft_consensus.cc:2747] T 26c8ed083cea4dc0acca80d00b11b29b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:28.204078 2113 raft_consensus.cc:2270] T a774cc6ae3924d299127fd9ffb01ca57 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250901 01:52:28.166914 1488 rpcz_store.cc:269] 0901 01:52:26.192328 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.192379 (+ 51us) service_pool.cc:225] Handling call
0901 01:52:27.121634 (+929255us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:27.121641 (+ 7us) write_op.cc:454] Released schema lock
0901 01:52:28.166742 (+1045101us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0901 01:52:26.553902 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:26.554026 (+ 124us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:26.554033 (+ 7us) write_op.cc:435] Acquired schema lock
0901 01:52:26.554036 (+ 3us) tablet.cc:660] Decoding operations
0901 01:52:26.554230 (+ 194us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:26.554245 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:26.554251 (+ 6us) tablet.cc:683] Acquiring locks for 4 operations
0901 01:52:26.554388 (+ 137us) tablet.cc:699] Row locks acquired
0901 01:52:26.554391 (+ 3us) write_op.cc:260] PREPARE: finished
0901 01:52:26.554431 (+ 40us) write_op.cc:270] Start()
0901 01:52:26.554468 (+ 37us) write_op.cc:276] Timestamp: P: 1756691546554327 usec, L: 0
0901 01:52:26.554471 (+ 3us) op_driver.cc:348] REPLICATION: starting
0901 01:52:26.554711 (+ 240us) log.cc:844] Serialized 1650 byte log entry
0901 01:52:26.565244 (+ 10533us) op_driver.cc:464] REPLICATION: finished
0901 01:52:26.565359 (+ 115us) write_op.cc:301] APPLY: starting
0901 01:52:26.565376 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:26.565454 (+ 78us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:26.565482 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:26.565568 (+ 86us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:26.565612 (+ 44us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0901 01:52:26.565619 (+ 7us) write_op.cc:312] APPLY: finished
0901 01:52:26.566206 (+ 587us) log.cc:844] Serialized 41 byte log entry
0901 01:52:26.566280 (+ 74us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:26.566303 (+ 23us) write_op.cc:454] Released schema lock
0901 01:52:26.566311 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0901 01:52:26.566386 (+ 75us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:26.566393 (+ 7us) write_op.cc:454] Released schema lock
Related trace 'op':
0901 01:52:26.643645 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:26.643745 (+ 100us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:26.643750 (+ 5us) write_op.cc:435] Acquired schema lock
0901 01:52:26.643752 (+ 2us) tablet.cc:660] Decoding operations
0901 01:52:26.643824 (+ 72us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:26.643832 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:26.643836 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0901 01:52:26.643878 (+ 42us) tablet.cc:699] Row locks acquired
0901 01:52:26.643880 (+ 2us) write_op.cc:260] PREPARE: finished
0901 01:52:26.643908 (+ 28us) write_op.cc:270] Start()
0901 01:52:26.643933 (+ 25us) write_op.cc:276] Timestamp: P: 1756691546643797 usec, L: 0
0901 01:52:26.643935 (+ 2us) op_driver.cc:348] REPLICATION: starting
0901 01:52:26.644067 (+ 132us) log.cc:844] Serialized 356 byte log entry
0901 01:52:26.651338 (+ 7271us) op_driver.cc:464] REPLICATION: finished
0901 01:52:26.651446 (+ 108us) write_op.cc:301] APPLY: starting
0901 01:52:26.651457 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:26.651492 (+ 35us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:26.651536 (+ 44us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:26.651578 (+ 42us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:26.651605 (+ 27us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0901 01:52:26.651613 (+ 8us) write_op.cc:312] APPLY: finished
0901 01:52:26.651672 (+ 59us) log.cc:844] Serialized 23 byte log entry
0901 01:52:26.651724 (+ 52us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:26.651739 (+ 15us) write_op.cc:454] Released schema lock
0901 01:52:26.651744 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0901 01:52:26.651808 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:26.651812 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0901 01:52:27.043340 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:27.043470 (+ 130us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:27.043475 (+ 5us) write_op.cc:435] Acquired schema lock
0901 01:52:27.043478 (+ 3us) tablet.cc:660] Decoding operations
0901 01:52:27.044554 (+ 1076us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:27.044563 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:27.044567 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0901 01:52:27.044621 (+ 54us) tablet.cc:699] Row locks acquired
0901 01:52:27.044624 (+ 3us) write_op.cc:260] PREPARE: finished
0901 01:52:27.044658 (+ 34us) write_op.cc:270] Start()
0901 01:52:27.044686 (+ 28us) write_op.cc:276] Timestamp: P: 1756691547044640 usec, L: 0
0901 01:52:27.044689 (+ 3us) op_driver.cc:348] REPLICATION: starting
0901 01:52:27.045193 (+ 504us) log.cc:844] Serialized 356 byte log entry
0901 01:52:27.050449 (+ 5256us) op_driver.cc:464] REPLICATION: finished
0901 01:52:27.050993 (+ 544us) write_op.cc:301] APPLY: starting
0901 01:52:27.051003 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:27.051032 (+ 29us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:27.051055 (+ 23us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:27.051092 (+ 37us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:27.051119 (+ 27us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0901 01:52:27.051127 (+ 8us) write_op.cc:312] APPLY: finished
0901 01:52:27.051177 (+ 50us) log.cc:844] Serialized 23 byte log entry
0901 01:52:27.051214 (+ 37us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:27.051228 (+ 14us) write_op.cc:454] Released schema lock
0901 01:52:27.051235 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0901 01:52:27.193684 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:27.193779 (+ 95us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:27.193785 (+ 6us) write_op.cc:435] Acquired schema lock
0901 01:52:27.193789 (+ 4us) tablet.cc:660] Decoding operations
0901 01:52:27.193901 (+ 112us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:27.193915 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:27.193922 (+ 7us) tablet.cc:683] Acquiring locks for 1 operations
0901 01:52:27.193981 (+ 59us) tablet.cc:699] Row locks acquired
0901 01:52:27.193985 (+ 4us) write_op.cc:260] PREPARE: finished
0901 01:52:27.194018 (+ 33us) write_op.cc:270] Start()
0901 01:52:27.194048 (+ 30us) write_op.cc:276] Timestamp: P: 1756691547193984 usec, L: 0
0901 01:52:27.194051 (+ 3us) op_driver.cc:348] REPLICATION: starting
0901 01:52:27.196320 (+ 2269us) log.cc:844] Serialized 356 byte log entry
0901 01:52:27.196529 (+ 209us) spinlock_profiling.cc:243] Waited 1.08 ms on lock 0x616000040c54. stack: 00007f0d64ec57d6 00007f0d64ec5590 00007f0d63eb83ad 0000000000972d94 0000000000972d5d 000000000097272a 00007f0d70921d62 00007f0d71cca2d5 00007f0d71cc8df9 00007f0d71cd1220 00007f0d71cd102d 0000000000a774dc 00007f0d64f0bb5f 00007f0d64f0ee2b 00007f0d64f0eca8 0000000000a774dc
0901 01:52:27.240165 (+ 43636us) op_driver.cc:464] REPLICATION: finished
0901 01:52:27.240341 (+ 176us) write_op.cc:301] APPLY: starting
0901 01:52:27.240359 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:27.240401 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:27.240403 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:27.240488 (+ 85us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:27.240524 (+ 36us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0901 01:52:27.240532 (+ 8us) write_op.cc:312] APPLY: finished
0901 01:52:27.240633 (+ 101us) log.cc:844] Serialized 23 byte log entry
0901 01:52:27.240700 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:27.240717 (+ 17us) write_op.cc:454] Released schema lock
0901 01:52:27.240724 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0901 01:52:27.240794 (+ 70us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:27.240798 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0901 01:52:27.309318 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:27.309415 (+ 97us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:27.309420 (+ 5us) write_op.cc:435] Acquired schema lock
0901 01:52:27.309422 (+ 2us) tablet.cc:660] Decoding operations
0901 01:52:27.309504 (+ 82us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:27.309516 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:27.309523 (+ 7us) tablet.cc:683] Acquiring locks for 1 operations
0901 01:52:27.309581 (+ 58us) tablet.cc:699] Row locks acquired
0901 01:52:27.309584 (+ 3us) write_op.cc:260] PREPARE: finished
0901 01:52:27.309612 (+ 28us) write_op.cc:270] Start()
0901 01:52:27.309638 (+ 26us) write_op.cc:276] Timestamp: P: 1756691547309569 usec, L: 0
0901 01:52:27.309642 (+ 4us) op_driver.cc:348] REPLICATION: starting
0901 01:52:27.309776 (+ 134us) log.cc:844] Serialized 356 byte log entry
0901 01:52:27.325439 (+ 15663us) op_driver.cc:464] REPLICATION: finished
0901 01:52:27.325545 (+ 106us) write_op.cc:301] APPLY: starting
0901 01:52:27.325571 (+ 26us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:27.325612 (+ 41us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:27.325644 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:27.325699 (+ 55us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:27.325730 (+ 31us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0901 01:52:27.325738 (+ 8us) write_op.cc:312] APPLY: finished
0901 01:52:27.325811 (+ 73us) log.cc:844] Serialized 23 byte log entry
0901 01:52:27.325865 (+ 54us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:27.325882 (+ 17us) write_op.cc:454] Released schema lock
0901 01:52:27.325888 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0901 01:52:27.325950 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:27.325953 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":236,"child_traces":[["op",{"apply.queue_time_us":33,"apply.run_cpu_time_us":603,"apply.run_wall_time_us":1019,"num_ops":4,"prepare.queue_time_us":5347,"prepare.run_cpu_time_us":927,"prepare.run_wall_time_us":928,"raft.queue_time_us":692,"raft.run_cpu_time_us":363,"raft.run_wall_time_us":364,"replication_time_us":7056}],["op",{"apply.queue_time_us":67,"apply.run_cpu_time_us":351,"apply.run_wall_time_us":351,"num_ops":1,"prepare.queue_time_us":44,"prepare.run_cpu_time_us":482,"prepare.run_wall_time_us":482,"replication_time_us":7373}],["op",{"apply.queue_time_us":507,"apply.run_cpu_time_us":287,"apply.run_wall_time_us":287,"num_ops":1,"prepare.queue_time_us":52,"prepare.run_cpu_time_us":741,"prepare.run_wall_time_us":1918,"replication_time_us":5735,"spinlock_wait_cycles":721152}],["op",{"apply.queue_time_us":85,"apply.run_cpu_time_us":449,"apply.run_wall_time_us":450,"num_ops":1,"prepare.queue_time_us":42,"prepare.run_cpu_time_us":1018,"prepare.run_wall_time_us":2914,"raft.queue_time_us":385,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":190,"replication_time_us":46071,"spinlock_wait_cycles":2482560}],["op",{"apply.queue_time_us":63,"apply.run_cpu_time_us":395,"apply.run_wall_time_us":395,"num_ops":1,"prepare.queue_time_us":38,"prepare.run_cpu_time_us":513,"prepare.run_wall_time_us":514,"replication_time_us":15769}]]}
I20250901 01:52:28.211751 1483 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 7f46e9ddadaf4ff986d8c991178daa0c
I20250901 01:52:28.211797 1490 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 9fbd44641d9e43c3b592e388e4adf350
W20250901 01:52:28.212031 1490 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 9fbd44641d9e43c3b592e388e4adf350: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:28.212353 1490 catalog_manager.cc:6652] ReplaceTablet: tablet 9fbd44641d9e43c3b592e388e4adf350 deleted and replaced by tablet dcc60bb000f440f1aca9133928b5ce20
I20250901 01:52:28.213054 1486 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 0cba5329ab31440999faec599d704633
W20250901 01:52:28.213250 1486 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 0cba5329ab31440999faec599d704633: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:28.213559 1486 catalog_manager.cc:6652] ReplaceTablet: tablet 0cba5329ab31440999faec599d704633 deleted and replaced by tablet 2f61edb50d7a42198bfd70639c7bf4de
I20250901 01:52:28.214165 2123 raft_consensus.cc:491] T 246b33338de647689d7404c70676e76f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:28.214283 2123 raft_consensus.cc:513] T 246b33338de647689d7404c70676e76f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:28.214833 2123 leader_election.cc:290] T 246b33338de647689d7404c70676e76f P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:28.215953 901 tablet_service.cc:1515] Processing DeleteTablet for tablet 9fbd44641d9e43c3b592e388e4adf350 with delete_type TABLET_DATA_DELETED (replaced by tablet dcc60bb000f440f1aca9133928b5ce20) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:28.215968 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 0cba5329ab31440999faec599d704633 with delete_type TABLET_DATA_DELETED (replaced by tablet 2f61edb50d7a42198bfd70639c7bf4de) from {username='slave'} at 127.0.0.1:34472
W20250901 01:52:28.217229 858 leader_election.cc:336] T 246b33338de647689d7404c70676e76f P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:28.217340 858 consensus_peers.cc:489] T 629db80a8ab444e582628578155638fd P 00c282f779c1477d920f24dc0dc16628 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250901 01:52:28.217525 1483 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 7f46e9ddadaf4ff986d8c991178daa0c: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:28.218056 1483 catalog_manager.cc:6652] ReplaceTablet: tablet 7f46e9ddadaf4ff986d8c991178daa0c deleted and replaced by tablet 7d7346ef9a2e4a35874605256ce41c6a
I20250901 01:52:28.218773 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 7f46e9ddadaf4ff986d8c991178daa0c with delete_type TABLET_DATA_DELETED (replaced by tablet 7d7346ef9a2e4a35874605256ce41c6a) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:28.220979 1033 tablet_service.cc:1515] Processing DeleteTablet for tablet 9fbd44641d9e43c3b592e388e4adf350 with delete_type TABLET_DATA_DELETED (replaced by tablet dcc60bb000f440f1aca9133928b5ce20) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:28.221845 1033 tablet_service.cc:1515] Processing DeleteTablet for tablet 0cba5329ab31440999faec599d704633 with delete_type TABLET_DATA_DELETED (replaced by tablet 2f61edb50d7a42198bfd70639c7bf4de) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:28.222067 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 7f46e9ddadaf4ff986d8c991178daa0c with delete_type TABLET_DATA_DELETED (replaced by tablet 7d7346ef9a2e4a35874605256ce41c6a) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:28.222576 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "246b33338de647689d7404c70676e76f" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:28.223482 2113 ts_tablet_manager.cc:1905] T a774cc6ae3924d299127fd9ffb01ca57 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:28.227918 859 leader_election.cc:343] T 246b33338de647689d7404c70676e76f P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:28.228050 859 leader_election.cc:304] T 246b33338de647689d7404c70676e76f P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:28.228440 2357 raft_consensus.cc:2747] T 246b33338de647689d7404c70676e76f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:28.229135 1151 tablet_bootstrap.cc:492] T 05630a72ddae4de0b890dece2feacc31 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:28.229357 1151 ts_tablet_manager.cc:1397] T 05630a72ddae4de0b890dece2feacc31 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.065s user 0.003s sys 0.003s
I20250901 01:52:28.230207 1151 raft_consensus.cc:357] T 05630a72ddae4de0b890dece2feacc31 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:28.230510 1151 raft_consensus.cc:383] T 05630a72ddae4de0b890dece2feacc31 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:28.230643 1151 raft_consensus.cc:738] T 05630a72ddae4de0b890dece2feacc31 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:28.231000 1151 consensus_queue.cc:260] T 05630a72ddae4de0b890dece2feacc31 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:28.231918 1151 ts_tablet_manager.cc:1428] T 05630a72ddae4de0b890dece2feacc31 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250901 01:52:28.232800 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_06ba4518130a4c4f888fb500219ed511"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.235191 2463 tablet.cc:1722] T 5adf6b492640417ca071c94702b48cb9 P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250901 01:52:28.235276 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_cb95252d00f44e28adbb12493ea743c1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.236173 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1ce7fc247fc84de49bddbf7dba58cebb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.236917 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_242d1633fc03439e9317b6190ffe2112"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:28.239722 992 consensus_peers.cc:489] T 1286c0ab85704837a99b6e5a62184d77 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250901 01:52:28.240147 2409 consensus_queue.cc:1035] T 081e2630e284498aa819f85bf8326315 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:28.247241 1103 heartbeater.cc:499] Master 127.0.104.60:44231 was elected leader, sending a full tablet report...
I20250901 01:52:28.247996 1489 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_3940656768d5406fb0e10a71f8f77003" } new_table_name: "default.Table_e7126b48b6034b0f81ce1c1bef8ffb51" modify_external_catalogs: true
I20250901 01:52:28.239419 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_319f5197ebba471f973d4970f61b9d8c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.250231 1490 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_4a412a59b6db421eb0187ffd02ecf541" } modify_external_catalogs: true
I20250901 01:52:28.250419 1490 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_4a412a59b6db421eb0187ffd02ecf541" } modify_external_catalogs: true
I20250901 01:52:28.239953 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b67463095af34540ac7ad8578c98c00f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.252326 1486 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_5e32028e03b04431b57a5d3a653afda4" } new_table_name: "default.Table_97526c9a6c9f4acbb6dedd78475571e0" modify_external_catalogs: true
I20250901 01:52:28.245842 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7277c935c6194d1cb697c6c41469d06d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.256984 1483 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_8770c960d4bd4c49b043ffe889b4b87b" } new_table_name: "default.Table_156d1feafd5246ff8ff1bb7b9ae444a5" modify_external_catalogs: true
I20250901 01:52:28.268994 1488 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_9e7ca61fcc324a8984bd1e470162c61f" } new_table_name: "default.Table_b3e694397be14321a668e46c09c51a71" modify_external_catalogs: true
I20250901 01:52:28.269852 2118 ts_tablet_manager.cc:1918] T 46bfbdff3f904715a8c24907b20fcab0 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:28.270715 2118 log.cc:1199] T 46bfbdff3f904715a8c24907b20fcab0 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/46bfbdff3f904715a8c24907b20fcab0
I20250901 01:52:28.271098 2409 consensus_queue.cc:1035] T eaa521c52a2f4408b59770bd2a05907c P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:28.271695 2118 ts_tablet_manager.cc:1939] T 46bfbdff3f904715a8c24907b20fcab0 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:28.280463 2409 raft_consensus.cc:491] T 5e2472ee5d6e4d3bbf1f24027ba531ca P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:28.280669 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 66) took 3056 ms. Trace:
I20250901 01:52:28.280618 2409 raft_consensus.cc:513] T 5e2472ee5d6e4d3bbf1f24027ba531ca P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:28.280922 2118 rpcz_store.cc:276] 0901 01:52:25.223892 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.223931 (+ 39us) service_pool.cc:225] Handling call
0901 01:52:28.120301 (+2896370us) ts_tablet_manager.cc:1145] Deleting tablet 46bfbdff3f904715a8c24907b20fcab0
0901 01:52:28.120627 (+ 326us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:28.168501 (+ 47874us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:28.269470 (+100969us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:28.280648 (+ 11178us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2846702}
I20250901 01:52:28.281255 2409 leader_election.cc:290] T 5e2472ee5d6e4d3bbf1f24027ba531ca P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:28.281744 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 46bfbdff3f904715a8c24907b20fcab0 (table default.Table_a210833682434f928e7da36d2fb4e09c [id=e71440216cc74a5a87964a8e456f784c]) successfully deleted
W20250901 01:52:28.282446 992 consensus_peers.cc:489] T eaa521c52a2f4408b59770bd2a05907c P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:28.283185 2323 raft_consensus.cc:491] T 81ce448b148447ddbc41875d863e0518 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:28.283314 2323 raft_consensus.cc:513] T 81ce448b148447ddbc41875d863e0518 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:28.283701 2118 tablet_replica.cc:331] T 20f7bfab18134e079e91c52c89468cc3 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
W20250901 01:52:28.285518 858 leader_election.cc:336] T 81ce448b148447ddbc41875d863e0518 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:28.285862 2118 raft_consensus.cc:2241] T 20f7bfab18134e079e91c52c89468cc3 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:28.286079 2118 raft_consensus.cc:2270] T 20f7bfab18134e079e91c52c89468cc3 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:28.286749 2118 ts_tablet_manager.cc:1905] T 20f7bfab18134e079e91c52c89468cc3 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:28.284134 2323 leader_election.cc:290] T 81ce448b148447ddbc41875d863e0518 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:28.295993 2118 ts_tablet_manager.cc:1918] T 20f7bfab18134e079e91c52c89468cc3 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:28.296430 2118 log.cc:1199] T 20f7bfab18134e079e91c52c89468cc3 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/20f7bfab18134e079e91c52c89468cc3
I20250901 01:52:28.296885 2118 ts_tablet_manager.cc:1939] T 20f7bfab18134e079e91c52c89468cc3 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:28.297255 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 67) took 3073 ms. Trace:
W20250901 01:52:28.292902 992 leader_election.cc:343] T 5e2472ee5d6e4d3bbf1f24027ba531ca P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 5e2472ee5d6e4d3bbf1f24027ba531ca
I20250901 01:52:28.297401 2118 rpcz_store.cc:276] 0901 01:52:25.224062 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.224117 (+ 55us) service_pool.cc:225] Handling call
0901 01:52:28.281580 (+3057463us) ts_tablet_manager.cc:1145] Deleting tablet 20f7bfab18134e079e91c52c89468cc3
0901 01:52:28.283662 (+ 2082us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:28.292981 (+ 9319us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:28.295903 (+ 2922us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:28.297234 (+ 1331us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3007937}
I20250901 01:52:28.298043 2118 tablet_replica.cc:331] T 50e3da53433f4c14b57db8a733fa53bc P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:28.297993 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 20f7bfab18134e079e91c52c89468cc3 (table default.Table_18b93109e1224ea59b5c93825b3b27d6 [id=d2ef66bea68843bcb33819f88668437f]) successfully deleted
I20250901 01:52:28.298345 2118 raft_consensus.cc:2241] T 50e3da53433f4c14b57db8a733fa53bc P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:28.298573 2118 raft_consensus.cc:2270] T 50e3da53433f4c14b57db8a733fa53bc P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:28.299170 2118 ts_tablet_manager.cc:1905] T 50e3da53433f4c14b57db8a733fa53bc P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:28.306157 2118 ts_tablet_manager.cc:1918] T 50e3da53433f4c14b57db8a733fa53bc P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:28.306334 2118 log.cc:1199] T 50e3da53433f4c14b57db8a733fa53bc P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/50e3da53433f4c14b57db8a733fa53bc
E20250901 01:52:28.306638 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae4844183a3c4a04a775e6fd878391a3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_aee74198b51d4f089538c4f2bf3ce4dc'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:28.306862 2118 ts_tablet_manager.cc:1939] T 50e3da53433f4c14b57db8a733fa53bc P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
W20250901 01:52:28.306936 1489 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 61d8f4252767490f842fb28f3edc1a96 consensus state has no leader
E20250901 01:52:28.306910 1696 catalog_manager.cc:6205] Error selecting replicas for tablet be8dfbc466564842a2fdb4953eee939b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_75ba3ba70ca24a94920e7bce96190ff2'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:28.307237 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 7317eb7848ee44698038a4b404d4bf21, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b0e916c76a7a4766aa0887cc8691488f'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:28.307256 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 70) took 3082 ms. Trace:
E20250901 01:52:28.307430 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9745582cebfe47b58c21d0191ab4cb9b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2f09538b6fbe401f909105decb56e1cb'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:28.307595 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 4abf9725be5d42f3893f07f0a9c7614c, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_8445fcd774874cadaf4acc3581eee417'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:28.307503 2118 rpcz_store.cc:276] 0901 01:52:25.224453 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.224493 (+ 40us) service_pool.cc:225] Handling call
0901 01:52:28.297995 (+3073502us) ts_tablet_manager.cc:1145] Deleting tablet 50e3da53433f4c14b57db8a733fa53bc
0901 01:52:28.298006 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:28.302420 (+ 4414us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:28.306071 (+ 3651us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:28.307236 (+ 1165us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3024120}
E20250901 01:52:28.307824 1696 catalog_manager.cc:6205] Error selecting replicas for tablet e2ed32b3292141278af28026528e96c2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_50ae3908a7944f70949c54be8fbf1564'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:28.307937 2118 tablet_replica.cc:331] T 4855448edb4b40c184838aa7f81c112b P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
E20250901 01:52:28.307991 1696 catalog_manager.cc:6205] Error selecting replicas for tablet a2ca69fabadd4ad69e1670cae7a0c503, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_32c4716e3aa24a87a6b1472984b255c4'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:28.307996 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 50e3da53433f4c14b57db8a733fa53bc (table default.Table_4ce20bf6ad224378861e083f6a480edf [id=cd8eab7eafac490c896a2a9be18e426f]) successfully deleted
E20250901 01:52:28.308203 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 76ba642c16734cdf969dbd3130639fda, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_faf7ef12324f454396e5ae64d64ebe51'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:28.308326 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f8f923ddcd2740bcb7e521f4f57c4034, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_5e65c5f068b84995a2d7a76fa65dda0f'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:28.308858 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 0813cc68d4b0488ba60a523e4479971b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d1302673d68f424ebb54b220911ce385'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:28.309026 1696 catalog_manager.cc:6205] Error selecting replicas for tablet b1ba0e19dc0b462e80c537bf2d183f5b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0e2cd82b00f1466eaef7f287402e644d'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:28.308146 2118 raft_consensus.cc:2241] T 4855448edb4b40c184838aa7f81c112b P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
E20250901 01:52:28.309216 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5f0e054f62de4cf6bc6f69ad6664b192, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ef6832d0845044ed9f165ca3541b4c3f'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:28.309373 2118 raft_consensus.cc:2270] T 4855448edb4b40c184838aa7f81c112b P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
E20250901 01:52:28.309401 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 787be682c0e44248a80d7037fb61dc6c, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7c12d66048a6470e8c227dc1d9f45fa4'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:28.309552 1696 catalog_manager.cc:6205] Error selecting replicas for tablet cc3b8736b9094d7e892c80da4d5b20f6, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_93afd37289414655980b22934695bd55'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:28.309713 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae9a4ad4fdfb4771ba23a91f7007447e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_5e84ae9a9c2a4084a1f9c5ffb4594de2'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:28.309890 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9074543dea9949e7a085717b40913348, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_53e30205c5fd488eb3b48c685a159d04'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:28.309962 2118 ts_tablet_manager.cc:1905] T 4855448edb4b40c184838aa7f81c112b P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
E20250901 01:52:28.310038 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 18498c9cb09141d08c1c5c0854b63ebd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77f4810bc83c4d948531da2a45a3792e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:28.310217 1696 catalog_manager.cc:6205] Error selecting replicas for tablet a6b539e160194da3a849d9e2f1ec9859, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_03fb48eaab9b42858ab7cefb722e43a8'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:28.310381 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 6ce9521b925b4f97bb4a810ce77e147e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_9e04f976a93f4d30baccf90fa3622dfe'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:28.310735 1489 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_9978497447394e5694ae928412634ecb" } modify_external_catalogs: true
I20250901 01:52:28.310950 1489 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_9978497447394e5694ae928412634ecb" } modify_external_catalogs: true
I20250901 01:52:28.320775 2118 ts_tablet_manager.cc:1918] T 4855448edb4b40c184838aa7f81c112b P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:28.320958 2118 log.cc:1199] T 4855448edb4b40c184838aa7f81c112b P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/4855448edb4b40c184838aa7f81c112b
I20250901 01:52:28.320999 2412 raft_consensus.cc:491] T f3962b471bdb49aea6e9fa95f1f9dfe2 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:28.321266 2412 raft_consensus.cc:513] T f3962b471bdb49aea6e9fa95f1f9dfe2 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:28.321372 2118 ts_tablet_manager.cc:1939] T 4855448edb4b40c184838aa7f81c112b P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:28.321770 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 86) took 3047 ms. Trace:
I20250901 01:52:28.321882 2118 rpcz_store.cc:276] 0901 01:52:25.274901 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.274946 (+ 45us) service_pool.cc:225] Handling call
0901 01:52:28.307895 (+3032949us) ts_tablet_manager.cc:1145] Deleting tablet 4855448edb4b40c184838aa7f81c112b
0901 01:52:28.307904 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:28.313307 (+ 5403us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:28.320672 (+ 7365us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:28.321752 (+ 1080us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3032570}
I20250901 01:52:28.322016 2412 leader_election.cc:290] T f3962b471bdb49aea6e9fa95f1f9dfe2 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:28.322391 2118 tablet_replica.cc:331] T c49d1143d54b4c319739488906b0e271 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:28.322387 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 4855448edb4b40c184838aa7f81c112b (table default.Table_d1302673d68f424ebb54b220911ce385 [id=453153553bea48319e132857f628d48a]) successfully deleted
I20250901 01:52:28.322448 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "81ce448b148447ddbc41875d863e0518" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:28.322615 2118 raft_consensus.cc:2241] T c49d1143d54b4c319739488906b0e271 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:28.322798 2118 raft_consensus.cc:2270] T c49d1143d54b4c319739488906b0e271 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:28.323414 2118 ts_tablet_manager.cc:1905] T c49d1143d54b4c319739488906b0e271 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:28.325346 859 leader_election.cc:343] T 81ce448b148447ddbc41875d863e0518 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:28.325487 859 leader_election.cc:304] T 81ce448b148447ddbc41875d863e0518 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:28.326095 2323 raft_consensus.cc:2747] T 81ce448b148447ddbc41875d863e0518 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:28.331851 992 consensus_peers.cc:489] T a6445b29591548be9f9577db1ba0bfa5 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250901 01:52:28.331959 992 consensus_peers.cc:489] T 3a8231590ea643b0a1f83eec4d9a8191 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250901 01:52:28.332015 992 leader_election.cc:336] T 5e2472ee5d6e4d3bbf1f24027ba531ca P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:28.332087 992 leader_election.cc:304] T 5e2472ee5d6e4d3bbf1f24027ba531ca P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:28.332320 2412 raft_consensus.cc:2747] T 5e2472ee5d6e4d3bbf1f24027ba531ca P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:28.333134 992 leader_election.cc:343] T f3962b471bdb49aea6e9fa95f1f9dfe2 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: f3962b471bdb49aea6e9fa95f1f9dfe2
I20250901 01:52:28.338002 1150 tablet_bootstrap.cc:492] T 617cc33d0d23439cb5f1e0c31bf11281 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:28.338246 1150 ts_tablet_manager.cc:1397] T 617cc33d0d23439cb5f1e0c31bf11281 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.186s user 0.012s sys 0.014s
I20250901 01:52:28.339479 1150 raft_consensus.cc:357] T 617cc33d0d23439cb5f1e0c31bf11281 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:28.339780 1150 raft_consensus.cc:383] T 617cc33d0d23439cb5f1e0c31bf11281 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:28.339905 1150 raft_consensus.cc:738] T 617cc33d0d23439cb5f1e0c31bf11281 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:28.340251 1150 consensus_queue.cc:260] T 617cc33d0d23439cb5f1e0c31bf11281 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:28.341095 1150 ts_tablet_manager.cc:1428] T 617cc33d0d23439cb5f1e0c31bf11281 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250901 01:52:28.342243 1150 tablet_bootstrap.cc:492] T 4b4aa063ffe948e69e47f03102581899 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
W20250901 01:52:28.349418 992 leader_election.cc:336] T f3962b471bdb49aea6e9fa95f1f9dfe2 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:28.349591 992 leader_election.cc:304] T f3962b471bdb49aea6e9fa95f1f9dfe2 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:28.350003 2409 raft_consensus.cc:2747] T f3962b471bdb49aea6e9fa95f1f9dfe2 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:28.350158 2408 consensus_queue.cc:1035] T 641a83a3e50d451189c33c7b7b4ca070 P 00c282f779c1477d920f24dc0dc16628 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:28.351172 2409 raft_consensus.cc:491] T 154574e17f214cc0ad56e6e6be3af899 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250901 01:52:28.351400 858 consensus_peers.cc:489] T 641a83a3e50d451189c33c7b7b4ca070 P 00c282f779c1477d920f24dc0dc16628 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:28.351382 2409 raft_consensus.cc:513] T 154574e17f214cc0ad56e6e6be3af899 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:28.352085 2409 leader_election.cc:290] T 154574e17f214cc0ad56e6e6be3af899 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:28.355984 1150 tablet_bootstrap.cc:654] T 4b4aa063ffe948e69e47f03102581899 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:28.364619 970 heartbeater.cc:344] Connected to a master server at 127.0.104.62:43871
W20250901 01:52:28.366147 992 consensus_peers.cc:489] T 670d90a37b934579b353bb4e4820127b P fd27ab6665304aaea53adbd102dd4f58 -> Peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Couldn't send request to peer 00c282f779c1477d920f24dc0dc16628. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 670d90a37b934579b353bb4e4820127b. This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:28.366883 2118 ts_tablet_manager.cc:1918] T c49d1143d54b4c319739488906b0e271 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:28.367427 2118 log.cc:1199] T c49d1143d54b4c319739488906b0e271 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/c49d1143d54b4c319739488906b0e271
I20250901 01:52:28.368670 2118 ts_tablet_manager.cc:1939] T c49d1143d54b4c319739488906b0e271 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:28.369014 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 85) took 3095 ms. Trace:
I20250901 01:52:28.369218 2113 ts_tablet_manager.cc:1918] T a774cc6ae3924d299127fd9ffb01ca57 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:28.369371 2113 log.cc:1199] T a774cc6ae3924d299127fd9ffb01ca57 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/a774cc6ae3924d299127fd9ffb01ca57
I20250901 01:52:28.369735 2113 ts_tablet_manager.cc:1939] T a774cc6ae3924d299127fd9ffb01ca57 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:28.370570 2118 rpcz_store.cc:276] 0901 01:52:25.273183 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.273232 (+ 49us) service_pool.cc:225] Handling call
0901 01:52:28.322339 (+3049107us) ts_tablet_manager.cc:1145] Deleting tablet c49d1143d54b4c319739488906b0e271
0901 01:52:28.322348 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:28.327828 (+ 5480us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:28.366751 (+ 38923us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:28.368999 (+ 2248us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3046896}
I20250901 01:52:28.371096 2118 tablet_replica.cc:331] T f8a5886ddb644a86b76df44aab309057 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:28.371088 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet c49d1143d54b4c319739488906b0e271 (table default.Table_583e53c2e86544e9be68eb27ab6068a3 [id=f376eaf65bb549429aa71b03fca81647]) successfully deleted
I20250901 01:52:28.371728 2118 raft_consensus.cc:2241] T f8a5886ddb644a86b76df44aab309057 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:28.372736 2118 raft_consensus.cc:2270] T f8a5886ddb644a86b76df44aab309057 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:28.372417 1150 tablet_bootstrap.cc:492] T 4b4aa063ffe948e69e47f03102581899 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:28.373540 1150 ts_tablet_manager.cc:1397] T 4b4aa063ffe948e69e47f03102581899 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.032s user 0.006s sys 0.000s
I20250901 01:52:28.374442 1150 raft_consensus.cc:357] T 4b4aa063ffe948e69e47f03102581899 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:28.374708 1150 raft_consensus.cc:383] T 4b4aa063ffe948e69e47f03102581899 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:28.374799 1150 raft_consensus.cc:738] T 4b4aa063ffe948e69e47f03102581899 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:28.375074 1150 consensus_queue.cc:260] T 4b4aa063ffe948e69e47f03102581899 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:28.376169 1150 ts_tablet_manager.cc:1428] T 4b4aa063ffe948e69e47f03102581899 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250901 01:52:28.376935 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 95) took 3297 ms. Trace:
I20250901 01:52:28.377084 2113 rpcz_store.cc:276] 0901 01:52:25.079817 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.079871 (+ 54us) service_pool.cc:225] Handling call
0901 01:52:28.197225 (+3117354us) ts_tablet_manager.cc:1145] Deleting tablet a774cc6ae3924d299127fd9ffb01ca57
0901 01:52:28.197236 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:28.227190 (+ 29954us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:28.369074 (+141884us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:28.376912 (+ 7838us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3103969}
I20250901 01:52:28.377624 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet a774cc6ae3924d299127fd9ffb01ca57 (table default.Table_1159dbb70cc940df8b509db6e70c350a [id=1d0bc46740444f1fbf2129270e806259]) successfully deleted
I20250901 01:52:28.377987 2113 tablet_replica.cc:331] T a7ff996482164776b468cf305989866b P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:28.378229 2113 raft_consensus.cc:2241] T a7ff996482164776b468cf305989866b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:28.378399 2113 raft_consensus.cc:2270] T a7ff996482164776b468cf305989866b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:28.378549 1150 tablet_bootstrap.cc:492] T 61ddcf86c9f34726939cc116434f0822 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:28.378933 2113 ts_tablet_manager.cc:1905] T a7ff996482164776b468cf305989866b P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:28.380438 2118 ts_tablet_manager.cc:1905] T f8a5886ddb644a86b76df44aab309057 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:28.384474 992 leader_election.cc:343] T 154574e17f214cc0ad56e6e6be3af899 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 154574e17f214cc0ad56e6e6be3af899
I20250901 01:52:28.385113 1150 tablet_bootstrap.cc:654] T 61ddcf86c9f34726939cc116434f0822 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:28.396908 992 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111) [suppressed 356 similar messages]
W20250901 01:52:28.397135 992 leader_election.cc:336] T 154574e17f214cc0ad56e6e6be3af899 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:28.397223 992 leader_election.cc:304] T 154574e17f214cc0ad56e6e6be3af899 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:28.397428 1761 raft_consensus.cc:2747] T 154574e17f214cc0ad56e6e6be3af899 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:28.407895 2113 ts_tablet_manager.cc:1918] T a7ff996482164776b468cf305989866b P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:28.413484 1150 tablet_bootstrap.cc:492] T 61ddcf86c9f34726939cc116434f0822 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:28.413692 1150 ts_tablet_manager.cc:1397] T 61ddcf86c9f34726939cc116434f0822 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.035s user 0.004s sys 0.000s
I20250901 01:52:28.414609 1150 raft_consensus.cc:357] T 61ddcf86c9f34726939cc116434f0822 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:28.414855 1150 raft_consensus.cc:383] T 61ddcf86c9f34726939cc116434f0822 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:28.414935 1150 raft_consensus.cc:738] T 61ddcf86c9f34726939cc116434f0822 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:28.415668 1150 consensus_queue.cc:260] T 61ddcf86c9f34726939cc116434f0822 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:28.417338 1482 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 81ce448b148447ddbc41875d863e0518 (table default.Table_6e461d9cb0ae4dd095975bbc21533794 [id=49673fa73f8c436e9b8026efc4f3cbbc]). Expected version 1 got 0
I20250901 01:52:28.423547 1482 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet b164b4a67a9143ceaff69b0ce49d63fe (table default.Table_da2c70a8f2fe49a0a8184e88c0a9e0ea [id=c1b1e9e3876244f68d31c63ce7bdebf1]). Expected version 1 got 0
I20250901 01:52:28.423974 1482 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet d1f2d98712ff4ea6aec665512de068bd (table default.Table_82d91a1ba2eb4265896824028dec8d3a [id=3f7624d2901445e39155c12fa68e34e9]). Expected version 1 got 0
I20250901 01:52:28.424266 1482 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 33f637f6a8134215bb1995ec38440073 (table default.Table_3d5afc22c4e14d6990aea31564dfde20 [id=557afe29976d40f6bfb6ae4ca348f887]). Expected version 2 got 0
I20250901 01:52:28.408496 2409 raft_consensus.cc:491] T 500e3c33598b45a4afa983da7cb0845a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:28.424562 2409 raft_consensus.cc:513] T 500e3c33598b45a4afa983da7cb0845a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:28.424664 1482 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet d0c3c363dda444519a17dca63ad9caa7 (table default.Table_42422c44ca254d8ca989134c2433391f [id=8c1b263824ec4c15a5ec424364c78895]). Expected version 3 got 0
I20250901 01:52:28.424971 1482 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 615b2306fca84003b264da9d5f95ddab (table default.Table_302f6e7cf97d46b9a621eb852e6aefe0 [id=cecdd8d6ca6d445c9fd82decd2a95366]). Expected version 2 got 0
I20250901 01:52:28.425151 1482 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet fecf7d538b4c4c619182d5eccb44dd8b (table default.Table_1b5729d9768546fe99eaafc4e2604f42 [id=02d0056abe2146ac9af81ebf4447b99d]). Expected version 1 got 0
I20250901 01:52:28.425209 2409 leader_election.cc:290] T 500e3c33598b45a4afa983da7cb0845a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:28.408691 992 consensus_peers.cc:489] T f8a5886ddb644a86b76df44aab309057 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Couldn't send request to peer 00c282f779c1477d920f24dc0dc16628. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:28.425659 1482 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 5adf6b492640417ca071c94702b48cb9 (table default.Table_99f0fb91a6994b2abf5af0c8f701159b [id=5d14706774d345c48ca3575feabb17b0]). Expected version 3 got 2
I20250901 01:52:28.425930 1482 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 7954393c8c1c4457ab8bb723c03a9c8b (table default.Table_7050639ebd0147959ac4a251f534eb96 [id=543f2074598d4c29972e564be1123fca]). Expected version 5 got 0
I20250901 01:52:28.426771 1482 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 89d40f0331114490b51de3ba6d773571 (table default.Table_4dacb3b1d2e94e42ac8937eb6eb58375 [id=8ee0aa1ba24042dd8ded004644913b9c]). Expected version 1 got 0
I20250901 01:52:28.427011 1482 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 858218c4427c41e980b2b5c969a5af40 (table default.Table_6d47266af0ed433eb1f2abbc81d18e69 [id=264cc469a73841e79538843479a165ab]). Expected version 1 got 0
W20250901 01:52:28.427907 992 leader_election.cc:336] T 500e3c33598b45a4afa983da7cb0845a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:28.427505 1482 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 3618a3eb4f174c43805ce7dc0e5315cb (table default.Table_83f2c0acedbc4c6982caf5313646e90b [id=31a2e50c32644321825da40176127015]). Expected version 1 got 0
I20250901 01:52:28.428516 1482 catalog_manager.cc:5582] T 71941b815d6840478c5acc1cbb187cc9 P 00c282f779c1477d920f24dc0dc16628 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00c282f779c1477d920f24dc0dc16628 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "00c282f779c1477d920f24dc0dc16628" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: HEALTHY } } }
I20250901 01:52:28.429028 1482 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet f80f13db501c42c7950499df8fcb6fc8 (table default.Table_ce86974fd8974de9a5a737954fac2ad7 [id=ff772d20b287476f94716379b86d2a71]). Expected version 1 got 0
I20250901 01:52:28.429600 1482 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 508406601c994e42a5c5b1f665e74957 (table default.Table_345b7f9fe8214aa6878f20a166125364 [id=4aabc328f6374ec5b1dea26f778208b2]). Expected version 1 got 0
I20250901 01:52:28.429903 1482 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 6c0abbfc8d904dce989ce51961e4237d (table default.Table_3f9422b08b0349b9a5466218f0fe1b06 [id=f0dfb3811e744bd4ba6d59195cebde6f]). Expected version 2 got 0
I20250901 01:52:28.430248 1482 catalog_manager.cc:5582] T f6a8b425968e49deaecfa3adacfd3f1f P 00c282f779c1477d920f24dc0dc16628 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00c282f779c1477d920f24dc0dc16628 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "00c282f779c1477d920f24dc0dc16628" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: HEALTHY } } }
I20250901 01:52:28.430756 1482 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 45f6e8ba99b74898a2922b36768c6c36 (table default.Table_938421db3cf74cf8b878089bac931818 [id=9ce26dcf307149b595a43b6ac4562fba]). Expected version 1 got 0
I20250901 01:52:28.430801 2113 log.cc:1199] T a7ff996482164776b468cf305989866b P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/a7ff996482164776b468cf305989866b
I20250901 01:52:28.430984 1482 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 0e4009963cfe42d6ba164fc828897f09 (table default.Table_1adc70cccd2f4cb3a79df96861cb4181 [id=24fffc7ebf5a47bdb37211910301acf7]). Expected version 1 got 0
I20250901 01:52:28.431422 2113 ts_tablet_manager.cc:1939] T a7ff996482164776b468cf305989866b P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:28.431459 1482 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 14b466b2a5a249aa841f1b8b06301edc (table default.Table_927955c1048f4e9c84b5b3a113a473f7 [id=78cd71321e9a480ca2d7d88327f157ee]). Expected version 1 got 0
I20250901 01:52:28.431650 1104 heartbeater.cc:344] Connected to a master server at 127.0.104.62:43871
I20250901 01:52:28.431849 1482 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 44af4764d4f64e44aec84ccb84be6f5f (table default.Table_9896b980158948339a0b91f5882ed56b [id=a541b55be0dc420a8353ef4793b925b5]). Expected version 1 got 0
I20250901 01:52:28.432034 1482 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 1495de70f0844f82a0eea99c059f076b (table default.Table_af5338b09bbc4c879c7f4f573c32fd8d [id=70a5ce71d16b47d4bc6dae97be4548e7]). Expected version 3 got 0
I20250901 01:52:28.432431 1482 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 93bcac1fd95d44dc927abfca81692921 (table default.Table_b8f8ee74648c406eaf589d3ab80934db [id=b8f69313439543be9c11eea88dfa69c5]). Expected version 1 got 0
I20250901 01:52:28.432653 1482 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet e9f1f892658b4736859ff507536920a1 (table default.Table_8825dd28afbe464193eb1b668cc9326a [id=9aa62cb634314a0a849612c54c1220d9]). Expected version 1 got 0
I20250901 01:52:28.432910 1482 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet ae21b9ae2e3b472eb67fb5f96388a50f (table default.Table_046289f393ff432aae0b033bdec1fb17 [id=db57b18a9ce24e068b362f5fbe940531]). Expected version 1 got 0
I20250901 01:52:28.433163 1482 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 5d68ea97c71942c7b55b967ba870e0d7 (table default.Table_f610cb75906d4c5cb3b57cfdb80ecefc [id=36915a3e843340ba8ef3578ac35a9af4]). Expected version 1 got 0
I20250901 01:52:28.433341 1482 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet d000ba05e12843b4925e622ab55471c0 (table default.Table_e0902dfbdd984f85b3fdac8d283731f7 [id=b43a92ed46164d199b445287ef039bc6]). Expected version 1 got 0
I20250901 01:52:28.417819 2118 ts_tablet_manager.cc:1918] T f8a5886ddb644a86b76df44aab309057 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:28.433687 1482 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 5f5396045979450193d461ba1334a5c9 (table default.Table_a89cc8e48563460baabc34c821ceae35 [id=9c3930e9fcfe4b49ba4bea67a2b0ad07]). Expected version 1 got 0
I20250901 01:52:28.434162 1482 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 61ddcf86c9f34726939cc116434f0822 (table default.Table_63d0f2ead9d946d39ce40162ebd769d6 [id=f9dcf4937b1143248bfa9d88053fce17]). Expected version 3 got 0
I20250901 01:52:28.434589 1482 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 8ca5cf4e99014c0d84a3acad4ef713af (table default.Table_ed6fd284be6f4ae3a008e21e7ce17930 [id=303e76c918bc4c39aa47a4d24849a358]). Expected version 1 got 0
I20250901 01:52:28.426918 922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "500e3c33598b45a4afa983da7cb0845a" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:28.434798 922 raft_consensus.cc:2466] T 500e3c33598b45a4afa983da7cb0845a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:28.434885 1482 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 7304937048ca48a7bd153f3cdcfaeff0 (table default.Table_1e59568bdcd14662ae9d5e98293d74bd [id=8051adf4041f4a2686a0b8e3850ef29c]). Expected version 1 got 0
I20250901 01:52:28.418407 1150 ts_tablet_manager.cc:1428] T 61ddcf86c9f34726939cc116434f0822 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250901 01:52:28.435204 1482 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet ad45c90ba0da43c9860745e534664a98 (table default.Table_1cd3f3fb6b244b3f8796bfca9f7622c8 [id=6fd739b0e725424ca844dbb652ea8028]). Expected version 2 got 0
I20250901 01:52:28.435593 1150 tablet_bootstrap.cc:492] T 4fca1435d82447f289fda4035aa2ed89 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:28.435752 1482 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet c6f7c525f59b49938b697de98d4b5835 (table default.Table_db61e055dbd7450898e82c4c0c4763be [id=aabca67f6f754820977ac9c4c20f3e4f]). Expected version 1 got 0
I20250901 01:52:28.436703 1482 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 081e2630e284498aa819f85bf8326315 (table default.Table_9e7ca61fcc324a8984bd1e470162c61f [id=8199118617cb4707bc3ae9b1363750f7]). Expected version 2 got 0
I20250901 01:52:28.437146 1482 catalog_manager.cc:5582] T e87d1281f35648b0b115b67c95e244e7 P 00c282f779c1477d920f24dc0dc16628 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00c282f779c1477d920f24dc0dc16628 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "00c282f779c1477d920f24dc0dc16628" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: HEALTHY } } }
I20250901 01:52:28.437570 1482 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet e87d1281f35648b0b115b67c95e244e7 (table default.Table_f840789a58984799bd8057128f6ea439 [id=4453959e273c473fadd737f56f7db3dd]). Expected version 1 got 0
I20250901 01:52:28.437626 1150 tablet_bootstrap.cc:654] T 4fca1435d82447f289fda4035aa2ed89 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:28.437747 992 leader_election.cc:304] T 500e3c33598b45a4afa983da7cb0845a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:28.437996 1482 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet ed1b41340ce742a88c3e4fd85cd25ca6 (table default.Table_139f6d174d794a6e9a618e90587d0c0d [id=0baa4993b72e4a8d94ee3059c9d2db6c]). Expected version 1 got 0
I20250901 01:52:28.438028 2409 raft_consensus.cc:2802] T 500e3c33598b45a4afa983da7cb0845a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:28.438097 2409 raft_consensus.cc:491] T 500e3c33598b45a4afa983da7cb0845a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:28.438169 2409 raft_consensus.cc:3058] T 500e3c33598b45a4afa983da7cb0845a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:28.440758 2409 raft_consensus.cc:513] T 500e3c33598b45a4afa983da7cb0845a P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:28.441300 2409 leader_election.cc:290] T 500e3c33598b45a4afa983da7cb0845a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:28.442842 2118 log.cc:1199] T f8a5886ddb644a86b76df44aab309057 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/f8a5886ddb644a86b76df44aab309057
I20250901 01:52:28.443279 2118 ts_tablet_manager.cc:1939] T f8a5886ddb644a86b76df44aab309057 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:28.443646 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 71) took 3219 ms. Trace:
I20250901 01:52:28.443747 2118 rpcz_store.cc:276] 0901 01:52:25.224572 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.224746 (+ 174us) service_pool.cc:225] Handling call
0901 01:52:28.371048 (+3146302us) ts_tablet_manager.cc:1145] Deleting tablet f8a5886ddb644a86b76df44aab309057
0901 01:52:28.371058 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:28.383438 (+ 12380us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:28.417714 (+ 34276us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:28.443629 (+ 25915us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3089169}
W20250901 01:52:28.447876 992 leader_election.cc:336] T 500e3c33598b45a4afa983da7cb0845a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:28.451433 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 96) took 3371 ms. Trace:
I20250901 01:52:28.452760 2409 raft_consensus.cc:491] T 6789505328b1409ea288e8b4eaf94935 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:28.445391 922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "500e3c33598b45a4afa983da7cb0845a" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:28.453513 922 raft_consensus.cc:3058] T 500e3c33598b45a4afa983da7cb0845a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:28.453557 2409 raft_consensus.cc:513] T 6789505328b1409ea288e8b4eaf94935 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:28.454252 2409 leader_election.cc:290] T 6789505328b1409ea288e8b4eaf94935 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:28.454336 2323 raft_consensus.cc:491] T 615b2306fca84003b264da9d5f95ddab P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:28.454458 2323 raft_consensus.cc:513] T 615b2306fca84003b264da9d5f95ddab P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:28.454725 2113 rpcz_store.cc:276] 0901 01:52:25.080329 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.080384 (+ 55us) service_pool.cc:225] Handling call
0901 01:52:28.377941 (+3297557us) ts_tablet_manager.cc:1145] Deleting tablet a7ff996482164776b468cf305989866b
0901 01:52:28.377950 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:28.382182 (+ 4232us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:28.407783 (+ 25601us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:28.451416 (+ 43633us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3284527}
I20250901 01:52:28.455030 2323 leader_election.cc:290] T 615b2306fca84003b264da9d5f95ddab P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:28.455114 2113 tablet_replica.cc:331] T a54af99bc10045e8b0b6868e5aed98b4 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:28.455297 2113 raft_consensus.cc:2241] T a54af99bc10045e8b0b6868e5aed98b4 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:28.455274 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet a7ff996482164776b468cf305989866b (table default.Table_1f8f7c00026749f593d89bc9e92f71cc [id=37883dea90384a2a8aa832201d66b7b6]) successfully deleted
I20250901 01:52:28.455638 2113 raft_consensus.cc:2270] T a54af99bc10045e8b0b6868e5aed98b4 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:28.456250 922 raft_consensus.cc:2466] T 500e3c33598b45a4afa983da7cb0845a P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:28.456317 2113 ts_tablet_manager.cc:1905] T a54af99bc10045e8b0b6868e5aed98b4 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:28.457747 1483 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet aec951f2cc84499b9162b9fa24d5440e consensus state has no leader
I20250901 01:52:28.458222 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet f8a5886ddb644a86b76df44aab309057 (table default.Table_79a6a0f0c6944313b9987f23b6a09557 [id=3f3723927dc04cf99499f68d39db8038]) successfully deleted
W20250901 01:52:28.458659 858 leader_election.cc:336] T 615b2306fca84003b264da9d5f95ddab P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:28.458717 1492 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: default.Table_bbbde49530e34a76b7cb17988ac6baca [id=76928d8caa2f4134a014656cf2cc871a] alter complete (version 1)
I20250901 01:52:28.459041 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 74) took 3234 ms. Trace:
I20250901 01:52:28.459172 2118 rpcz_store.cc:276] 0901 01:52:25.224933 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.224986 (+ 53us) service_pool.cc:225] Handling call
0901 01:52:28.458931 (+3233945us) ts_tablet_manager.cc:1145] Deleting tablet 34042ea5c1b645969c1de41fe587c343
0901 01:52:28.458943 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:28.459022 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3168742}
I20250901 01:52:28.459515 922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6789505328b1409ea288e8b4eaf94935" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:28.459643 992 leader_election.cc:304] T 500e3c33598b45a4afa983da7cb0845a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:28.459724 922 raft_consensus.cc:2466] T 6789505328b1409ea288e8b4eaf94935 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:28.460163 2118 tablet_replica.cc:331] T b6daeff0070a4db08e90020752d21c29 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:28.460472 2118 raft_consensus.cc:2241] T b6daeff0070a4db08e90020752d21c29 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:28.460960 2409 raft_consensus.cc:2802] T 500e3c33598b45a4afa983da7cb0845a P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:28.461117 2409 raft_consensus.cc:695] T 500e3c33598b45a4afa983da7cb0845a P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:28.461218 2118 raft_consensus.cc:2270] T b6daeff0070a4db08e90020752d21c29 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:28.461403 2409 consensus_queue.cc:237] T 500e3c33598b45a4afa983da7cb0845a P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:28.464123 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 34042ea5c1b645969c1de41fe587c343 because the tablet was not found. No further retry: Not found: Tablet not found: 34042ea5c1b645969c1de41fe587c343
I20250901 01:52:28.462810 2118 ts_tablet_manager.cc:1905] T b6daeff0070a4db08e90020752d21c29 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:28.467069 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "615b2306fca84003b264da9d5f95ddab" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:28.468092 1492 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: default.Table_fd17a2ad225546aa821489ce94a788a2 [id=033c76e0598342c495ce2be42a0df5dd] alter complete (version 1)
I20250901 01:52:28.473577 992 leader_election.cc:304] T 6789505328b1409ea288e8b4eaf94935 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:28.478946 1491 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 8a13346dcb3d4b6ab35c261829d50ae5
I20250901 01:52:28.479374 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_265a238b1a0c48b8850b8dee8d0679aa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.480291 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_034e1393dc9f4692ab2072015a0b7bde"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.481065 1491 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_927955c1048f4e9c84b5b3a113a473f7" } new_table_name: "default.Table_8c439dfdc16046b682876df01dadd3e0" modify_external_catalogs: true
I20250901 01:52:28.482728 1150 tablet_bootstrap.cc:492] T 4fca1435d82447f289fda4035aa2ed89 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:28.482951 1150 ts_tablet_manager.cc:1397] T 4fca1435d82447f289fda4035aa2ed89 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.047s user 0.000s sys 0.004s
I20250901 01:52:28.483207 2409 raft_consensus.cc:2802] T 6789505328b1409ea288e8b4eaf94935 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:28.483314 2409 raft_consensus.cc:491] T 6789505328b1409ea288e8b4eaf94935 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:28.483505 2409 raft_consensus.cc:3058] T 6789505328b1409ea288e8b4eaf94935 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:28.484243 1485 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet c422881b40f54971a68530abfbd1945b
W20250901 01:52:28.484441 1485 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet c422881b40f54971a68530abfbd1945b: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:28.485092 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_bc6a719a855f4087a67164e059968d0a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.485854 1485 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_9d0d7a93ff15424092012d4d681b8ae0" } modify_external_catalogs: true
I20250901 01:52:28.486100 1150 raft_consensus.cc:357] T 4fca1435d82447f289fda4035aa2ed89 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:28.486351 1150 raft_consensus.cc:383] T 4fca1435d82447f289fda4035aa2ed89 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:28.486552 1150 raft_consensus.cc:738] T 4fca1435d82447f289fda4035aa2ed89 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:28.486580 2409 raft_consensus.cc:513] T 6789505328b1409ea288e8b4eaf94935 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:28.487025 1150 consensus_queue.cc:260] T 4fca1435d82447f289fda4035aa2ed89 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:28.487295 887 tablet_service.cc:1515] Processing DeleteTablet for tablet c422881b40f54971a68530abfbd1945b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:28 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:28.487327 2409 leader_election.cc:290] T 6789505328b1409ea288e8b4eaf94935 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:28.487775 859 leader_election.cc:343] T 615b2306fca84003b264da9d5f95ddab P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
W20250901 01:52:28.487799 992 leader_election.cc:336] T 6789505328b1409ea288e8b4eaf94935 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:28.487887 859 leader_election.cc:304] T 615b2306fca84003b264da9d5f95ddab P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:28.488382 2357 raft_consensus.cc:2747] T 615b2306fca84003b264da9d5f95ddab P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:28.488566 2357 consensus_queue.cc:1035] T f6a8b425968e49deaecfa3adacfd3f1f P 00c282f779c1477d920f24dc0dc16628 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:28.489380 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet c422881b40f54971a68530abfbd1945b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:28 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:28.489717 1150 ts_tablet_manager.cc:1428] T 4fca1435d82447f289fda4035aa2ed89 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250901 01:52:28.489977 1484 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: default.Table_95e1338a49a04f22b679e363e253262e [id=e24b91b4ac904ef0a8c4725f70bced2f] alter complete (version 1)
I20250901 01:52:28.490375 1490 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet b2e8b2cd10154de08ddd30c214592a4a
W20250901 01:52:28.490621 1490 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet b2e8b2cd10154de08ddd30c214592a4a: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:28.490713 1150 tablet_bootstrap.cc:492] T 7f46e9ddadaf4ff986d8c991178daa0c P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:28.491360 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_6217b0e78a444dcea14c1932763891ae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.490037 1489 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 7d7346ef9a2e4a35874605256ce41c6a
I20250901 01:52:28.492393 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_bb5ec9ae3f964a7c8e16c932696e5631"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.492723 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_97f9b1d397264891bb562bab5472c77d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.493121 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2aecebbfbbac4db0a31ed5bc30ca0a6a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.493335 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c6b87c32ae13473db432eb1767000a29"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.493861 1489 master_service.cc:948] ReplaceTablet: received request to replace tablet 8b9d79a25452410891c09fd3f7c4be6d from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:28.493820 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f1bff2a91bbd49318bae49e4d17a77dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.494510 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a38907bf63014b999e8791daf1526413"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.495251 1490 master_service.cc:948] ReplaceTablet: received request to replace tablet 1495de70f0844f82a0eea99c059f076b from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:28.496433 901 tablet_service.cc:1515] Processing DeleteTablet for tablet e25923b2daf9429683b2d44a75a56189 with delete_type TABLET_DATA_DELETED (replaced by tablet 2973b59bb1df41d18d9a5b6a29ae336f) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:28.496623 887 tablet_service.cc:1515] Processing DeleteTablet for tablet b2e8b2cd10154de08ddd30c214592a4a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:28 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:28.497288 901 tablet_service.cc:1515] Processing DeleteTablet for tablet bb0cf5a6a27045a08df6845d7473184d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:28.497790 901 tablet_service.cc:1515] Processing DeleteTablet for tablet a54af99bc10045e8b0b6868e5aed98b4 with delete_type TABLET_DATA_DELETED (replaced by tablet 50472a7aa2f04e0fbd65ac6c86c34b3e) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:28.498346 901 tablet_service.cc:1515] Processing DeleteTablet for tablet 7a2f6e2c7ee2402ea310c058e046416c with delete_type TABLET_DATA_DELETED (replaced by tablet 5f5396045979450193d461ba1334a5c9) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:28.498842 901 tablet_service.cc:1515] Processing DeleteTablet for tablet 26c8ed083cea4dc0acca80d00b11b29b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:23 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:28.499697 901 tablet_service.cc:1515] Processing DeleteTablet for tablet f98d1d8d660443cb8066a547e5d9b77f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:23 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:28.500452 901 tablet_service.cc:1515] Processing DeleteTablet for tablet 50472a7aa2f04e0fbd65ac6c86c34b3e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:20 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:28.500516 887 tablet_service.cc:1515] Processing DeleteTablet for tablet c763d252598745a088344aa0eb5fa3d3 with delete_type TABLET_DATA_DELETED (replaced by tablet a8483e7212d94daba7e5401256c33aef) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:28.502167 901 tablet_service.cc:1515] Processing DeleteTablet for tablet 20f7bfab18134e079e91c52c89468cc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:17 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:28.502528 901 tablet_service.cc:1515] Processing DeleteTablet for tablet 56f2c4d2e6ab4bb4bdd6d025843589b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:28.503252 901 tablet_service.cc:1515] Processing DeleteTablet for tablet 0dbdcb76eeba4fefa56e7835c23c0316 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:17 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:28.503697 901 tablet_service.cc:1515] Processing DeleteTablet for tablet e1c8fe96a6ba4edba4f09c00f7add75e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:28.503988 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 764bfeb2a7d14c40815b5ec1b07bdd62 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:23 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:28.504273 2118 ts_tablet_manager.cc:1918] T b6daeff0070a4db08e90020752d21c29 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:28.504374 2118 log.cc:1199] T b6daeff0070a4db08e90020752d21c29 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/b6daeff0070a4db08e90020752d21c29
I20250901 01:52:28.504855 2118 ts_tablet_manager.cc:1939] T b6daeff0070a4db08e90020752d21c29 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:28.505883 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 4fca1435d82447f289fda4035aa2ed89 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:20 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:28.506526 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 46bfbdff3f904715a8c24907b20fcab0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:28.507308 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 670d90a37b934579b353bb4e4820127b with delete_type TABLET_DATA_DELETED (replaced by tablet 75a703c8e20a4a2fa0ab6ec58d41ef92) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:28.507894 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 50e3da53433f4c14b57db8a733fa53bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:18 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:28.508472 887 tablet_service.cc:1515] Processing DeleteTablet for tablet f8a5886ddb644a86b76df44aab309057 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:14 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:28.509023 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 1286c0ab85704837a99b6e5a62184d77 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:16 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:28.509356 1150 tablet_bootstrap.cc:654] T 7f46e9ddadaf4ff986d8c991178daa0c P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:28.509919 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 822382a1781b4304bce29355ce83730e with delete_type TABLET_DATA_DELETED (replaced by tablet 0958b0a4068344abb0f9db601d7dd2ad) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:28.510481 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 579b9cdf819f4609a532e96bec093fd5 with delete_type TABLET_DATA_DELETED (replaced by tablet 56475ba491a24301a534304811f30140) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:28.511328 887 tablet_service.cc:1515] Processing DeleteTablet for tablet d77aadd7291647a0bae7c421f244c252 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:14 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:28.512116 887 tablet_service.cc:1515] Processing DeleteTablet for tablet c49d1143d54b4c319739488906b0e271 with delete_type TABLET_DATA_DELETED (replaced by tablet 69f61ec283484f9d8b3eebcea27f741b) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:28.512939 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 4855448edb4b40c184838aa7f81c112b with delete_type TABLET_DATA_DELETED (replaced by tablet 5e2472ee5d6e4d3bbf1f24027ba531ca) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:28.513379 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet b2e8b2cd10154de08ddd30c214592a4a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:28 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:28.515718 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 77) took 3290 ms. Trace:
I20250901 01:52:28.518977 919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6789505328b1409ea288e8b4eaf94935" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:28.519552 919 raft_consensus.cc:3058] T 6789505328b1409ea288e8b4eaf94935 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:28.519860 2323 raft_consensus.cc:491] T c6f7c525f59b49938b697de98d4b5835 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:28.519973 2323 raft_consensus.cc:513] T c6f7c525f59b49938b697de98d4b5835 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:28.520573 2323 leader_election.cc:290] T c6f7c525f59b49938b697de98d4b5835 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:28.528223 919 raft_consensus.cc:2466] T 6789505328b1409ea288e8b4eaf94935 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
W20250901 01:52:28.528774 858 leader_election.cc:336] T c6f7c525f59b49938b697de98d4b5835 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:28.528927 2118 rpcz_store.cc:276] 0901 01:52:25.225409 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.225582 (+ 173us) service_pool.cc:225] Handling call
0901 01:52:28.460114 (+3234532us) ts_tablet_manager.cc:1145] Deleting tablet b6daeff0070a4db08e90020752d21c29
0901 01:52:28.460125 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:28.486184 (+ 26059us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:28.504166 (+ 17982us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:28.515687 (+ 11521us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3168740}
I20250901 01:52:28.529079 1151 tablet_bootstrap.cc:492] T f9b293bf6a17453b9ea9892a6970d0b0 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:28.529479 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet b6daeff0070a4db08e90020752d21c29 (table default.Table_8022c1172d8a4df8bc1295d193bb6542 [id=f278d7b5596944238ef56dfb5de097dc]) successfully deleted
I20250901 01:52:28.529858 2118 tablet_replica.cc:331] T 121803e79ab64ec0b948b448646b9d0a P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:28.530112 2118 raft_consensus.cc:2241] T 121803e79ab64ec0b948b448646b9d0a P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:28.530332 2118 raft_consensus.cc:2270] T 121803e79ab64ec0b948b448646b9d0a P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:28.530983 2118 ts_tablet_manager.cc:1905] T 121803e79ab64ec0b948b448646b9d0a P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:28.537678 992 consensus_peers.cc:489] T ddbc15685fca4c708329420109ad0c57 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250901 01:52:28.540602 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c6f7c525f59b49938b697de98d4b5835" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:28.540927 992 leader_election.cc:336] T 6789505328b1409ea288e8b4eaf94935 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:28.543624 2459 tablet.cc:1722] T 081e2630e284498aa819f85bf8326315 P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20250901 01:52:28.549496 1484 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:54603 (request call id 182) took 1470 ms (client timeout 1000 ms). Trace:
W20250901 01:52:28.549726 1484 rpcz_store.cc:269] 0901 01:52:27.078536 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:27.079549 (+ 1013us) service_pool.cc:225] Handling call
0901 01:52:27.644912 (+565363us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:27.644921 (+ 9us) write_op.cc:454] Released schema lock
0901 01:52:28.549429 (+904508us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0901 01:52:27.623307 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:27.623432 (+ 125us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:27.623439 (+ 7us) write_op.cc:435] Acquired schema lock
0901 01:52:27.623442 (+ 3us) tablet.cc:660] Decoding operations
0901 01:52:27.623567 (+ 125us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:27.623581 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:27.623586 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0901 01:52:27.623653 (+ 67us) tablet.cc:699] Row locks acquired
0901 01:52:27.623656 (+ 3us) write_op.cc:260] PREPARE: finished
0901 01:52:27.623689 (+ 33us) write_op.cc:270] Start()
0901 01:52:27.623722 (+ 33us) write_op.cc:276] Timestamp: P: 1756691547623664 usec, L: 0
0901 01:52:27.623726 (+ 4us) op_driver.cc:348] REPLICATION: starting
0901 01:52:27.623900 (+ 174us) log.cc:844] Serialized 527 byte log entry
0901 01:52:27.627616 (+ 3716us) op_driver.cc:464] REPLICATION: finished
0901 01:52:27.627879 (+ 263us) write_op.cc:301] APPLY: starting
0901 01:52:27.627895 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:27.627937 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:27.627968 (+ 31us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:27.628023 (+ 55us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:27.628055 (+ 32us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0901 01:52:27.628062 (+ 7us) write_op.cc:312] APPLY: finished
0901 01:52:27.628131 (+ 69us) log.cc:844] Serialized 23 byte log entry
0901 01:52:27.628189 (+ 58us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:27.628207 (+ 18us) write_op.cc:454] Released schema lock
0901 01:52:27.628213 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0901 01:52:27.990083 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:27.990212 (+ 129us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:27.990216 (+ 4us) write_op.cc:435] Acquired schema lock
0901 01:52:27.990218 (+ 2us) tablet.cc:660] Decoding operations
0901 01:52:27.990288 (+ 70us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:27.990297 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:27.990301 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0901 01:52:27.990378 (+ 77us) tablet.cc:699] Row locks acquired
0901 01:52:27.990382 (+ 4us) write_op.cc:260] PREPARE: finished
0901 01:52:27.990409 (+ 27us) write_op.cc:270] Start()
0901 01:52:27.990435 (+ 26us) write_op.cc:276] Timestamp: P: 1756691547990348 usec, L: 0
0901 01:52:27.990443 (+ 8us) op_driver.cc:348] REPLICATION: starting
0901 01:52:27.990573 (+ 130us) log.cc:844] Serialized 356 byte log entry
0901 01:52:28.015842 (+ 25269us) op_driver.cc:464] REPLICATION: finished
0901 01:52:28.016171 (+ 329us) write_op.cc:301] APPLY: starting
0901 01:52:28.016274 (+ 103us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:28.016395 (+ 121us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:28.016460 (+ 65us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:28.019404 (+ 2944us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:28.019615 (+ 211us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0901 01:52:28.019829 (+ 214us) write_op.cc:312] APPLY: finished
0901 01:52:28.020468 (+ 639us) log.cc:844] Serialized 23 byte log entry
0901 01:52:28.021164 (+ 696us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:28.021357 (+ 193us) write_op.cc:454] Released schema lock
0901 01:52:28.021368 (+ 11us) write_op.cc:341] FINISH: Updating metrics
0901 01:52:28.061252 (+ 39884us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:28.061260 (+ 8us) write_op.cc:454] Released schema lock
Related trace 'op':
0901 01:52:28.193878 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:28.194404 (+ 526us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:28.194508 (+ 104us) write_op.cc:435] Acquired schema lock
0901 01:52:28.194513 (+ 5us) tablet.cc:660] Decoding operations
0901 01:52:28.195439 (+ 926us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:28.195608 (+ 169us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:28.195617 (+ 9us) tablet.cc:683] Acquiring locks for 1 operations
0901 01:52:28.199268 (+ 3651us) tablet.cc:699] Row locks acquired
0901 01:52:28.199272 (+ 4us) write_op.cc:260] PREPARE: finished
0901 01:52:28.199797 (+ 525us) write_op.cc:270] Start()
0901 01:52:28.199993 (+ 196us) write_op.cc:276] Timestamp: P: 1756691548199278 usec, L: 0
0901 01:52:28.199999 (+ 6us) op_driver.cc:348] REPLICATION: starting
0901 01:52:28.240546 (+ 40547us) log.cc:844] Serialized 356 byte log entry
0901 01:52:28.241083 (+ 537us) spinlock_profiling.cc:243] Waited 40.1 ms on lock 0x616000040c54. stack: 00007f0d64ec57d6 00007f0d64ec5590 00007f0d63eb83ad 0000000000972d94 0000000000972d5d 000000000097272a 00007f0d70921d62 00007f0d71cca2d5 00007f0d71cc8df9 00007f0d71cd1220 00007f0d71cd102d 0000000000a774dc 00007f0d64f0bb5f 00007f0d64f0ee2b 00007f0d64f0eca8 0000000000a774dc
0901 01:52:28.296617 (+ 55534us) op_driver.cc:464] REPLICATION: finished
0901 01:52:28.296821 (+ 204us) write_op.cc:301] APPLY: starting
0901 01:52:28.296834 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:28.296871 (+ 37us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:28.296874 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:28.296923 (+ 49us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:28.296953 (+ 30us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0901 01:52:28.296960 (+ 7us) write_op.cc:312] APPLY: finished
0901 01:52:28.297026 (+ 66us) log.cc:844] Serialized 23 byte log entry
0901 01:52:28.297081 (+ 55us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:28.297094 (+ 13us) write_op.cc:454] Released schema lock
0901 01:52:28.297100 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0901 01:52:28.297425 (+ 325us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:28.297429 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":37378,"reported_tablets":193,"child_traces":[["op",{"apply.queue_time_us":212,"apply.run_cpu_time_us":398,"apply.run_wall_time_us":398,"num_ops":1,"prepare.queue_time_us":1144,"prepare.run_cpu_time_us":698,"prepare.run_wall_time_us":698,"raft.queue_time_us":409,"raft.run_cpu_time_us":280,"raft.run_wall_time_us":288,"replication_time_us":3862}],["op",{"apply.queue_time_us":171,"apply.run_cpu_time_us":848,"apply.run_wall_time_us":5616,"num_ops":1,"prepare.queue_time_us":1743,"prepare.run_cpu_time_us":9655,"prepare.run_wall_time_us":71150,"raft.queue_time_us":517,"raft.run_cpu_time_us":2,"raft.run_wall_time_us":3,"replication_time_us":25365,"thread_start_us":70412,"threads_started":1}],["op",{"apply.queue_time_us":159,"apply.run_cpu_time_us":336,"apply.run_wall_time_us":594,"num_ops":1,"prepare.queue_time_us":156505,"prepare.run_cpu_time_us":1974,"prepare.run_wall_time_us":52904,"raft.queue_time_us":44,"raft.run_cpu_time_us":322,"raft.run_wall_time_us":323,"replication_time_us":96591,"spinlock_wait_cycles":92163456}]]}
I20250901 01:52:28.553136 1484 master_service.cc:948] ReplaceTablet: received request to replace tablet a6445b29591548be9f9577db1ba0bfa5 from {username='slave'} at 127.0.0.1:33478
W20250901 01:52:28.555445 858 consensus_peers.cc:489] T c58edaa76ebf47dfb1d9d5fab6de7dfa P 00c282f779c1477d920f24dc0dc16628 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:28.557282 1151 tablet_bootstrap.cc:654] T f9b293bf6a17453b9ea9892a6970d0b0 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:28.565713 992 leader_election.cc:304] T 6789505328b1409ea288e8b4eaf94935 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:28.566408 2409 raft_consensus.cc:2802] T 6789505328b1409ea288e8b4eaf94935 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:28.566628 2409 raft_consensus.cc:695] T 6789505328b1409ea288e8b4eaf94935 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:28.566929 2409 consensus_queue.cc:237] T 6789505328b1409ea288e8b4eaf94935 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:28.568662 859 leader_election.cc:343] T c6f7c525f59b49938b697de98d4b5835 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:28.568827 859 leader_election.cc:304] T c6f7c525f59b49938b697de98d4b5835 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:28.569228 2323 raft_consensus.cc:2747] T c6f7c525f59b49938b697de98d4b5835 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:28.573587 1151 tablet_bootstrap.cc:492] T f9b293bf6a17453b9ea9892a6970d0b0 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:28.574180 1151 ts_tablet_manager.cc:1397] T f9b293bf6a17453b9ea9892a6970d0b0 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.045s user 0.005s sys 0.000s
I20250901 01:52:28.577108 1761 raft_consensus.cc:491] T 7954393c8c1c4457ab8bb723c03a9c8b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:28.577235 1761 raft_consensus.cc:513] T 7954393c8c1c4457ab8bb723c03a9c8b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:28.577796 1761 leader_election.cc:290] T 7954393c8c1c4457ab8bb723c03a9c8b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:28.578423 2118 ts_tablet_manager.cc:1918] T 121803e79ab64ec0b948b448646b9d0a P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:28.578650 2118 log.cc:1199] T 121803e79ab64ec0b948b448646b9d0a P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/121803e79ab64ec0b948b448646b9d0a
I20250901 01:52:28.579437 2118 ts_tablet_manager.cc:1939] T 121803e79ab64ec0b948b448646b9d0a P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:28.579861 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 76) took 3354 ms. Trace:
I20250901 01:52:28.580016 2118 rpcz_store.cc:276] 0901 01:52:25.225252 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.225319 (+ 67us) service_pool.cc:225] Handling call
0901 01:52:28.529797 (+3304478us) ts_tablet_manager.cc:1145] Deleting tablet 121803e79ab64ec0b948b448646b9d0a
0901 01:52:28.529811 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:28.562335 (+ 32524us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:28.578252 (+ 15917us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:28.579817 (+ 1565us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3238349}
I20250901 01:52:28.579594 2113 ts_tablet_manager.cc:1918] T a54af99bc10045e8b0b6868e5aed98b4 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:28.580632 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 121803e79ab64ec0b948b448646b9d0a (table default.Table_117c6c43b552476dbb8b7b5ccdce47b7 [id=a4c947d85200437395c66bb4f7e607af]) successfully deleted
I20250901 01:52:28.581517 2463 tablet.cc:1722] T 081e2630e284498aa819f85bf8326315 P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250901 01:52:28.581084 2113 log.cc:1199] T a54af99bc10045e8b0b6868e5aed98b4 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/a54af99bc10045e8b0b6868e5aed98b4
I20250901 01:52:28.582528 2118 tablet_replica.cc:331] T 0cb3262973934c8689b851adad60afe0 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:28.584141 2118 raft_consensus.cc:2241] T 0cb3262973934c8689b851adad60afe0 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:28.584376 2118 raft_consensus.cc:2270] T 0cb3262973934c8689b851adad60afe0 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:28.586628 2118 ts_tablet_manager.cc:1905] T 0cb3262973934c8689b851adad60afe0 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:28.587515 919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7954393c8c1c4457ab8bb723c03a9c8b" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:28.587764 919 raft_consensus.cc:2466] T 7954393c8c1c4457ab8bb723c03a9c8b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
W20250901 01:52:28.589242 858 consensus_peers.cc:489] T f6a8b425968e49deaecfa3adacfd3f1f P 00c282f779c1477d920f24dc0dc16628 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:28.596684 2113 ts_tablet_manager.cc:1939] T a54af99bc10045e8b0b6868e5aed98b4 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:28.597054 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 97) took 3515 ms. Trace:
I20250901 01:52:28.597262 2113 rpcz_store.cc:276] 0901 01:52:25.081873 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.081953 (+ 80us) service_pool.cc:225] Handling call
0901 01:52:28.455068 (+3373115us) ts_tablet_manager.cc:1145] Deleting tablet a54af99bc10045e8b0b6868e5aed98b4
0901 01:52:28.455079 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:28.533169 (+ 78090us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:28.573754 (+ 40585us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:28.597037 (+ 23283us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3361609}
I20250901 01:52:28.597900 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet a54af99bc10045e8b0b6868e5aed98b4 (table default.Table_489b87a28dac48d093c241f301d5cccc [id=5fdd352d5e5e47a3a2bd8fd608a263dd]) successfully deleted
I20250901 01:52:28.598362 2113 tablet_replica.cc:331] T c95798f0daf4437db18a291c4464522e P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:28.579628 1151 raft_consensus.cc:357] T f9b293bf6a17453b9ea9892a6970d0b0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:28.602597 1151 raft_consensus.cc:383] T f9b293bf6a17453b9ea9892a6970d0b0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:28.603091 1151 raft_consensus.cc:738] T f9b293bf6a17453b9ea9892a6970d0b0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:28.604576 992 leader_election.cc:304] T 7954393c8c1c4457ab8bb723c03a9c8b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:28.604791 2113 raft_consensus.cc:2241] T c95798f0daf4437db18a291c4464522e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:28.604532 1151 consensus_queue.cc:260] T f9b293bf6a17453b9ea9892a6970d0b0 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:28.605803 2409 raft_consensus.cc:2802] T 7954393c8c1c4457ab8bb723c03a9c8b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:28.605962 2409 raft_consensus.cc:491] T 7954393c8c1c4457ab8bb723c03a9c8b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:28.606073 2409 raft_consensus.cc:3058] T 7954393c8c1c4457ab8bb723c03a9c8b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:28.606499 2118 ts_tablet_manager.cc:1918] T 0cb3262973934c8689b851adad60afe0 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:28.606676 2118 log.cc:1199] T 0cb3262973934c8689b851adad60afe0 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/0cb3262973934c8689b851adad60afe0
I20250901 01:52:28.607082 2118 ts_tablet_manager.cc:1939] T 0cb3262973934c8689b851adad60afe0 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:28.607933 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 82) took 3355 ms. Trace:
I20250901 01:52:28.608059 2118 rpcz_store.cc:276] 0901 01:52:25.252603 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.252676 (+ 73us) service_pool.cc:225] Handling call
0901 01:52:28.582439 (+3329763us) ts_tablet_manager.cc:1145] Deleting tablet 0cb3262973934c8689b851adad60afe0
0901 01:52:28.582450 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:28.593608 (+ 11158us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:28.606346 (+ 12738us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:28.607875 (+ 1529us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3289863}
I20250901 01:52:28.608561 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 0cb3262973934c8689b851adad60afe0 (table default.Table_0fc5eac5c02943af9b6801ac9cee55a3 [id=28235bf690f747b398aea70f53caa578]) successfully deleted
I20250901 01:52:28.608968 2409 raft_consensus.cc:513] T 7954393c8c1c4457ab8bb723c03a9c8b P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:28.609563 2409 leader_election.cc:290] T 7954393c8c1c4457ab8bb723c03a9c8b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:28.609993 2113 raft_consensus.cc:2270] T c95798f0daf4437db18a291c4464522e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:28.610605 2323 consensus_queue.cc:1035] T 71941b815d6840478c5acc1cbb187cc9 P 00c282f779c1477d920f24dc0dc16628 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:28.612769 1151 ts_tablet_manager.cc:1428] T f9b293bf6a17453b9ea9892a6970d0b0 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.038s user 0.003s sys 0.000s
I20250901 01:52:28.613507 2113 ts_tablet_manager.cc:1905] T c95798f0daf4437db18a291c4464522e P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:28.614979 2118 tablet_replica.cc:331] T b7b802f976f3471faaaecc964ec87e48 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:28.615211 2118 raft_consensus.cc:2241] T b7b802f976f3471faaaecc964ec87e48 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:28.615427 2118 raft_consensus.cc:2270] T b7b802f976f3471faaaecc964ec87e48 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:28.616031 2118 ts_tablet_manager.cc:1905] T b7b802f976f3471faaaecc964ec87e48 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:28.617216 1151 tablet_bootstrap.cc:492] T afc002ae2486476b9208e1cf8215157c P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:28.620931 1150 tablet_bootstrap.cc:492] T 7f46e9ddadaf4ff986d8c991178daa0c P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:28.621109 1150 ts_tablet_manager.cc:1397] T 7f46e9ddadaf4ff986d8c991178daa0c P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.131s user 0.007s sys 0.011s
I20250901 01:52:28.621941 1150 raft_consensus.cc:357] T 7f46e9ddadaf4ff986d8c991178daa0c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:28.622125 1150 raft_consensus.cc:383] T 7f46e9ddadaf4ff986d8c991178daa0c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:28.622224 1150 raft_consensus.cc:738] T 7f46e9ddadaf4ff986d8c991178daa0c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:28.622439 1150 consensus_queue.cc:260] T 7f46e9ddadaf4ff986d8c991178daa0c P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:28.623005 1150 ts_tablet_manager.cc:1428] T 7f46e9ddadaf4ff986d8c991178daa0c P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250901 01:52:28.623312 1150 tablet_bootstrap.cc:492] T 45f6e8ba99b74898a2922b36768c6c36 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:28.631017 1150 tablet_bootstrap.cc:654] T 45f6e8ba99b74898a2922b36768c6c36 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
E20250901 01:52:28.632721 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae4844183a3c4a04a775e6fd878391a3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_aee74198b51d4f089538c4f2bf3ce4dc'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:28.632925 1696 catalog_manager.cc:6205] Error selecting replicas for tablet be8dfbc466564842a2fdb4953eee939b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_75ba3ba70ca24a94920e7bce96190ff2'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:28.633071 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 7317eb7848ee44698038a4b404d4bf21, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b0e916c76a7a4766aa0887cc8691488f'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:28.633189 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9745582cebfe47b58c21d0191ab4cb9b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2f09538b6fbe401f909105decb56e1cb'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:28.633320 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 4abf9725be5d42f3893f07f0a9c7614c, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_8445fcd774874cadaf4acc3581eee417'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:28.633428 1696 catalog_manager.cc:6205] Error selecting replicas for tablet e2ed32b3292141278af28026528e96c2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_50ae3908a7944f70949c54be8fbf1564'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:28.633550 1696 catalog_manager.cc:6205] Error selecting replicas for tablet dcc60bb000f440f1aca9133928b5ce20, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2fbca8463d6546c9b8f5b018fc60251d'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:28.633662 1696 catalog_manager.cc:6205] Error selecting replicas for tablet a2ca69fabadd4ad69e1670cae7a0c503, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_32c4716e3aa24a87a6b1472984b255c4'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:28.633795 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 76ba642c16734cdf969dbd3130639fda, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_faf7ef12324f454396e5ae64d64ebe51'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:28.633919 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f8f923ddcd2740bcb7e521f4f57c4034, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_5e65c5f068b84995a2d7a76fa65dda0f'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:28.633849 2118 ts_tablet_manager.cc:1918] T b7b802f976f3471faaaecc964ec87e48 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
E20250901 01:52:28.634091 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 0813cc68d4b0488ba60a523e4479971b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d1302673d68f424ebb54b220911ce385'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:28.634208 1696 catalog_manager.cc:6205] Error selecting replicas for tablet b1ba0e19dc0b462e80c537bf2d183f5b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0e2cd82b00f1466eaef7f287402e644d'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:28.634334 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5f0e054f62de4cf6bc6f69ad6664b192, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ef6832d0845044ed9f165ca3541b4c3f'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:28.634279 2118 log.cc:1199] T b7b802f976f3471faaaecc964ec87e48 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/b7b802f976f3471faaaecc964ec87e48
E20250901 01:52:28.634584 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 787be682c0e44248a80d7037fb61dc6c, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7c12d66048a6470e8c227dc1d9f45fa4'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:28.634730 1696 catalog_manager.cc:6205] Error selecting replicas for tablet cc3b8736b9094d7e892c80da4d5b20f6, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_93afd37289414655980b22934695bd55'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:28.634876 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae9a4ad4fdfb4771ba23a91f7007447e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_5e84ae9a9c2a4084a1f9c5ffb4594de2'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:28.635012 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9074543dea9949e7a085717b40913348, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_53e30205c5fd488eb3b48c685a159d04'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:28.635129 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 18498c9cb09141d08c1c5c0854b63ebd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77f4810bc83c4d948531da2a45a3792e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:28.635277 1696 catalog_manager.cc:6205] Error selecting replicas for tablet a6b539e160194da3a849d9e2f1ec9859, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_03fb48eaab9b42858ab7cefb722e43a8'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:28.635452 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 6ce9521b925b4f97bb4a810ce77e147e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_9e04f976a93f4d30baccf90fa3622dfe'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:28.635699 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2f61edb50d7a42198bfd70639c7bf4de, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_a5c783cbdb694af69047b31d99ff8def'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:28.636044 2118 ts_tablet_manager.cc:1939] T b7b802f976f3471faaaecc964ec87e48 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:28.638566 919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7954393c8c1c4457ab8bb723c03a9c8b" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
W20250901 01:52:28.639497 992 leader_election.cc:336] T 7954393c8c1c4457ab8bb723c03a9c8b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:28.639719 919 raft_consensus.cc:3058] T 7954393c8c1c4457ab8bb723c03a9c8b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
W20250901 01:52:28.640470 992 consensus_peers.cc:489] T bb0cf5a6a27045a08df6845d7473184d P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250901 01:52:28.645115 1151 tablet_bootstrap.cc:654] T afc002ae2486476b9208e1cf8215157c P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:28.649868 919 raft_consensus.cc:2466] T 7954393c8c1c4457ab8bb723c03a9c8b P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:28.651603 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 83) took 3398 ms. Trace:
I20250901 01:52:28.652175 2118 rpcz_store.cc:276] 0901 01:52:25.252754 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.252815 (+ 61us) service_pool.cc:225] Handling call
0901 01:52:28.614880 (+3362065us) ts_tablet_manager.cc:1145] Deleting tablet b7b802f976f3471faaaecc964ec87e48
0901 01:52:28.614892 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:28.620671 (+ 5779us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:28.633317 (+ 12646us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:28.651436 (+ 18119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3322205}
I20250901 01:52:28.653347 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet b7b802f976f3471faaaecc964ec87e48 (table default.Table_3552b4561f10466b9b77f797ca003e53 [id=ec380a4600f34080bc1e343130537377]) successfully deleted
I20250901 01:52:28.654274 2118 tablet_replica.cc:331] T 2240302d9178445d8700e35f5ea274f7 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:28.655592 2118 raft_consensus.cc:2241] T 2240302d9178445d8700e35f5ea274f7 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:28.656324 2118 raft_consensus.cc:2270] T 2240302d9178445d8700e35f5ea274f7 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:28.658985 2118 ts_tablet_manager.cc:1905] T 2240302d9178445d8700e35f5ea274f7 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:28.659417 2409 raft_consensus.cc:491] T aec951f2cc84499b9162b9fa24d5440e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:28.659633 2409 raft_consensus.cc:513] T aec951f2cc84499b9162b9fa24d5440e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:28.660322 2409 leader_election.cc:290] T aec951f2cc84499b9162b9fa24d5440e P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:28.665215 2323 raft_consensus.cc:491] T 858218c4427c41e980b2b5c969a5af40 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:28.665429 2323 raft_consensus.cc:513] T 858218c4427c41e980b2b5c969a5af40 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:28.666082 2323 leader_election.cc:290] T 858218c4427c41e980b2b5c969a5af40 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:28.667109 992 leader_election.cc:336] T 7954393c8c1c4457ab8bb723c03a9c8b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:28.673822 858 leader_election.cc:336] T 858218c4427c41e980b2b5c969a5af40 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:28.680083 919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aec951f2cc84499b9162b9fa24d5440e" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:28.680327 919 raft_consensus.cc:2466] T aec951f2cc84499b9162b9fa24d5440e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
W20250901 01:52:28.681614 1474 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet b2e8b2cd10154de08ddd30c214592a4a on TS 0cf599f74322478d91475df234e58965 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:28.682862 2323 raft_consensus.cc:491] T 7f46e9ddadaf4ff986d8c991178daa0c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:28.683018 2323 raft_consensus.cc:513] T 7f46e9ddadaf4ff986d8c991178daa0c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:28.683673 2323 leader_election.cc:290] T 7f46e9ddadaf4ff986d8c991178daa0c P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:28.684993 858 leader_election.cc:336] T 7f46e9ddadaf4ff986d8c991178daa0c P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:28.685390 1151 tablet_bootstrap.cc:492] T afc002ae2486476b9208e1cf8215157c P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:28.685593 1151 ts_tablet_manager.cc:1397] T afc002ae2486476b9208e1cf8215157c P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.072s user 0.003s sys 0.000s
I20250901 01:52:28.686369 1151 raft_consensus.cc:357] T afc002ae2486476b9208e1cf8215157c P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:28.686637 1151 raft_consensus.cc:383] T afc002ae2486476b9208e1cf8215157c P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:28.686846 1151 raft_consensus.cc:738] T afc002ae2486476b9208e1cf8215157c P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:28.687340 1151 consensus_queue.cc:260] T afc002ae2486476b9208e1cf8215157c P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:28.688995 1151 ts_tablet_manager.cc:1428] T afc002ae2486476b9208e1cf8215157c P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250901 01:52:28.693845 1151 tablet_bootstrap.cc:492] T 14b466b2a5a249aa841f1b8b06301edc P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:28.696619 1051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "858218c4427c41e980b2b5c969a5af40" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:28.697057 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f46e9ddadaf4ff986d8c991178daa0c" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:28.698959 992 leader_election.cc:304] T 7954393c8c1c4457ab8bb723c03a9c8b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:28.701292 2409 raft_consensus.cc:2802] T 7954393c8c1c4457ab8bb723c03a9c8b P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:28.701579 2409 raft_consensus.cc:695] T 7954393c8c1c4457ab8bb723c03a9c8b P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:28.701882 2409 consensus_queue.cc:237] T 7954393c8c1c4457ab8bb723c03a9c8b P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:28.704869 858 consensus_peers.cc:489] T 2599f98abfdc4871b573edfc2ad97d22 P 00c282f779c1477d920f24dc0dc16628 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:28.705448 1761 raft_consensus.cc:491] T 482a9a2cb81f448bb2823892e35920d4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:28.705564 1761 raft_consensus.cc:513] T 482a9a2cb81f448bb2823892e35920d4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:28.706063 1761 leader_election.cc:290] T 482a9a2cb81f448bb2823892e35920d4 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:28.712378 1761 consensus_queue.cc:1035] T 8ed6616ce54548ba83ba299c4ecf3c38 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:28.713652 2118 ts_tablet_manager.cc:1918] T 2240302d9178445d8700e35f5ea274f7 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:28.714483 2118 log.cc:1199] T 2240302d9178445d8700e35f5ea274f7 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/2240302d9178445d8700e35f5ea274f7
I20250901 01:52:28.714666 992 leader_election.cc:304] T aec951f2cc84499b9162b9fa24d5440e P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:28.716777 2118 ts_tablet_manager.cc:1939] T 2240302d9178445d8700e35f5ea274f7 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:28.717365 1761 raft_consensus.cc:2802] T aec951f2cc84499b9162b9fa24d5440e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:28.717445 1761 raft_consensus.cc:491] T aec951f2cc84499b9162b9fa24d5440e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:28.717510 1761 raft_consensus.cc:3058] T aec951f2cc84499b9162b9fa24d5440e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:28.718760 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 84) took 3465 ms. Trace:
I20250901 01:52:28.720067 1761 raft_consensus.cc:513] T aec951f2cc84499b9162b9fa24d5440e P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:28.720847 2459 tablet.cc:1722] T f4fd117c395e41c2ab679c684908c5b7 P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:28.721562 1761 leader_election.cc:290] T aec951f2cc84499b9162b9fa24d5440e P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:28.719156 2118 rpcz_store.cc:276] 0901 01:52:25.252870 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.252954 (+ 84us) service_pool.cc:225] Handling call
0901 01:52:28.653996 (+3401042us) ts_tablet_manager.cc:1145] Deleting tablet 2240302d9178445d8700e35f5ea274f7
0901 01:52:28.654010 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:28.687662 (+ 33652us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:28.712908 (+ 25246us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:28.718584 (+ 5676us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3360995}
I20250901 01:52:28.722113 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 2240302d9178445d8700e35f5ea274f7 (table default.Table_36f5f8854ed14d1ea48eb18865d00659 [id=bdf8a102ae7d43228173d7b82507d80d]) successfully deleted
I20250901 01:52:28.722968 2118 tablet_replica.cc:331] T af2c4bcbb00741f789edee109252d8cb P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:28.724325 2118 raft_consensus.cc:2241] T af2c4bcbb00741f789edee109252d8cb P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:28.725184 2118 raft_consensus.cc:2270] T af2c4bcbb00741f789edee109252d8cb P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250901 01:52:28.724326 992 consensus_peers.cc:489] T 579b9cdf819f4609a532e96bec093fd5 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Couldn't send request to peer 00c282f779c1477d920f24dc0dc16628. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 579b9cdf819f4609a532e96bec093fd5. This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:28.729839 1150 tablet_bootstrap.cc:492] T 45f6e8ba99b74898a2922b36768c6c36 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:28.730114 1150 ts_tablet_manager.cc:1397] T 45f6e8ba99b74898a2922b36768c6c36 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.107s user 0.002s sys 0.021s
I20250901 01:52:28.733732 1485 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_9d0d7a93ff15424092012d4d681b8ae0" } modify_external_catalogs: true
I20250901 01:52:28.733978 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b9cd35f0d24d45b08c76fc1d925a18dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.734911 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c70fe93923ac4850b9922d1a4dec6926"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.735466 2118 ts_tablet_manager.cc:1905] T af2c4bcbb00741f789edee109252d8cb P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:28.735634 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ee043e9b987f441db48cdf67cb8eb1ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.736255 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ef609f6fcde24cbd875ce38b7a357334"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.739562 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_87806a58074b4728b6732d62fffbf08b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.740198 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_bd6d6b058f4e4fd2bd4d3cedbd5e8b56"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.740897 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5c94089b83c9416abab07d888f0fd18c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.741519 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f1bd376cfa854ec48cb80608275545e6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.742163 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4816d9bee7444a298f8fc6ff889e7511"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.742360 2118 ts_tablet_manager.cc:1918] T af2c4bcbb00741f789edee109252d8cb P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:28.742470 2118 log.cc:1199] T af2c4bcbb00741f789edee109252d8cb P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/af2c4bcbb00741f789edee109252d8cb
I20250901 01:52:28.742808 2118 ts_tablet_manager.cc:1939] T af2c4bcbb00741f789edee109252d8cb P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:28.742765 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_aed2f4206c6c41ad9ec84c1de63a9683"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.743108 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 88) took 3391 ms. Trace:
I20250901 01:52:28.743171 2118 rpcz_store.cc:276] 0901 01:52:25.351630 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.351706 (+ 76us) service_pool.cc:225] Handling call
0901 01:52:28.722754 (+3371048us) ts_tablet_manager.cc:1145] Deleting tablet af2c4bcbb00741f789edee109252d8cb
0901 01:52:28.722768 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:28.739516 (+ 16748us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:28.742273 (+ 2757us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:28.743091 (+ 818us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3370551}
I20250901 01:52:28.743335 1488 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_202b4c5ccc56474a8b2e7b9d7c4a28d6" } new_table_name: "default.Table_7790af96855e45c0a316ba3a4a87c9ed" modify_external_catalogs: true
I20250901 01:52:28.743655 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet af2c4bcbb00741f789edee109252d8cb (table default.Table_6f470109ee524f10a48d1080ec15a199 [id=dc57478deb384d52ae97315fcdce074a]) successfully deleted
I20250901 01:52:28.749289 1483 master_service.cc:948] ReplaceTablet: received request to replace tablet e2aa32450a584395ae07b49f76a7b26d from {username='slave'} at 127.0.0.1:33478
W20250901 01:52:28.749754 859 leader_election.cc:343] T 858218c4427c41e980b2b5c969a5af40 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:28.749897 859 leader_election.cc:304] T 858218c4427c41e980b2b5c969a5af40 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:28.750275 2323 raft_consensus.cc:2747] T 858218c4427c41e980b2b5c969a5af40 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:28.750808 859 leader_election.cc:343] T 7f46e9ddadaf4ff986d8c991178daa0c P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:28.750905 859 leader_election.cc:304] T 7f46e9ddadaf4ff986d8c991178daa0c P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:28.751261 2323 raft_consensus.cc:2747] T 7f46e9ddadaf4ff986d8c991178daa0c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:28.751410 1150 raft_consensus.cc:357] T 45f6e8ba99b74898a2922b36768c6c36 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:28.752264 1150 raft_consensus.cc:383] T 45f6e8ba99b74898a2922b36768c6c36 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:28.752487 1150 raft_consensus.cc:738] T 45f6e8ba99b74898a2922b36768c6c36 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:28.752962 1150 consensus_queue.cc:260] T 45f6e8ba99b74898a2922b36768c6c36 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:28.754444 1483 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_2f09538b6fbe401f909105decb56e1cb" } new_table_name: "default.Table_77728af9eee44f29a1e2131d5b75b7ca" modify_external_catalogs: true
I20250901 01:52:28.757063 2113 ts_tablet_manager.cc:1918] T c95798f0daf4437db18a291c4464522e P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:28.757175 2113 log.cc:1199] T c95798f0daf4437db18a291c4464522e P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/c95798f0daf4437db18a291c4464522e
I20250901 01:52:28.757659 1151 tablet_bootstrap.cc:654] T 14b466b2a5a249aa841f1b8b06301edc P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:28.759322 2113 ts_tablet_manager.cc:1939] T c95798f0daf4437db18a291c4464522e P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:28.764603 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 90) took 3412 ms. Trace:
I20250901 01:52:28.764679 1150 ts_tablet_manager.cc:1428] T 45f6e8ba99b74898a2922b36768c6c36 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.034s user 0.001s sys 0.001s
I20250901 01:52:28.766588 919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "482a9a2cb81f448bb2823892e35920d4" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:28.766139 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 98) took 3677 ms. Trace:
I20250901 01:52:28.767036 919 raft_consensus.cc:2466] T 482a9a2cb81f448bb2823892e35920d4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:28.765769 2118 rpcz_store.cc:276] 0901 01:52:25.351923 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.351981 (+ 58us) service_pool.cc:225] Handling call
0901 01:52:28.763475 (+3411494us) ts_tablet_manager.cc:1145] Deleting tablet ded33e0950b544509af11739353c620d
0901 01:52:28.763589 (+ 114us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:28.764352 (+ 763us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3411073}
W20250901 01:52:28.769562 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet ded33e0950b544509af11739353c620d because the tablet was not found. No further retry: Not found: Tablet not found: ded33e0950b544509af11739353c620d
I20250901 01:52:28.766979 2113 rpcz_store.cc:276] 0901 01:52:25.089171 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.089232 (+ 61us) service_pool.cc:225] Handling call
0901 01:52:28.598290 (+3509058us) ts_tablet_manager.cc:1145] Deleting tablet c95798f0daf4437db18a291c4464522e
0901 01:52:28.598301 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:28.669083 (+ 70782us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:28.756947 (+ 87864us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:28.766114 (+ 9167us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5496,"tablet-delete.queue_time_us":3504775}
I20250901 01:52:28.770327 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet c95798f0daf4437db18a291c4464522e (table default.Table_a89cc8e48563460baabc34c821ceae35 [id=9c3930e9fcfe4b49ba4bea67a2b0ad07]) successfully deleted
I20250901 01:52:28.771638 2408 raft_consensus.cc:491] T d10c193452954c198bba686169b8025b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:28.771785 2408 raft_consensus.cc:513] T d10c193452954c198bba686169b8025b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:28.771994 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 99) took 3682 ms. Trace:
I20250901 01:52:28.772400 2408 leader_election.cc:290] T d10c193452954c198bba686169b8025b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:28.772662 2113 rpcz_store.cc:276] 0901 01:52:25.089625 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.089679 (+ 54us) service_pool.cc:225] Handling call
0901 01:52:28.771164 (+3681485us) ts_tablet_manager.cc:1145] Deleting tablet f06a77ecc841462396a8d5026c7327f4
0901 01:52:28.771356 (+ 192us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:28.771795 (+ 439us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3677453}
W20250901 01:52:28.773401 992 leader_election.cc:336] T aec951f2cc84499b9162b9fa24d5440e P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:28.774732 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet f06a77ecc841462396a8d5026c7327f4 because the tablet was not found. No further retry: Not found: Tablet not found: f06a77ecc841462396a8d5026c7327f4
W20250901 01:52:28.774849 858 leader_election.cc:336] T d10c193452954c198bba686169b8025b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:28.776499 2408 raft_consensus.cc:491] T 45f6e8ba99b74898a2922b36768c6c36 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:28.776659 2408 raft_consensus.cc:513] T 45f6e8ba99b74898a2922b36768c6c36 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:28.777386 2408 leader_election.cc:290] T 45f6e8ba99b74898a2922b36768c6c36 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:28.778630 858 leader_election.cc:336] T 45f6e8ba99b74898a2922b36768c6c36 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:28.779568 2408 raft_consensus.cc:491] T a9b7f2e6548043a7865e25387cdd06c9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:28.779290 2118 tablet_replica.cc:331] T 7f64ebd080944190be2e64d02d70a8a7 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:28.779723 2408 raft_consensus.cc:513] T a9b7f2e6548043a7865e25387cdd06c9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:28.780293 2408 leader_election.cc:290] T a9b7f2e6548043a7865e25387cdd06c9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:28.780824 1150 tablet_bootstrap.cc:492] T 2dacb608897d47d28512df87dc320927 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
W20250901 01:52:28.781528 858 leader_election.cc:336] T a9b7f2e6548043a7865e25387cdd06c9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:28.782282 2463 tablet.cc:1722] T f4fd117c395e41c2ab679c684908c5b7 P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:28.784353 919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aec951f2cc84499b9162b9fa24d5440e" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:28.784745 919 raft_consensus.cc:3058] T aec951f2cc84499b9162b9fa24d5440e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:28.786406 2113 tablet_replica.cc:331] T 438cd35ff3f74c49b908d072560077db P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:28.786834 2113 raft_consensus.cc:2241] T 438cd35ff3f74c49b908d072560077db P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:28.787106 2113 raft_consensus.cc:2270] T 438cd35ff3f74c49b908d072560077db P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:28.787631 919 raft_consensus.cc:2466] T aec951f2cc84499b9162b9fa24d5440e P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:28.788391 2113 ts_tablet_manager.cc:1905] T 438cd35ff3f74c49b908d072560077db P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:28.792861 2409 raft_consensus.cc:491] T 05630a72ddae4de0b890dece2feacc31 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:28.792992 2409 raft_consensus.cc:513] T 05630a72ddae4de0b890dece2feacc31 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:28.784150 2118 raft_consensus.cc:2241] T 7f64ebd080944190be2e64d02d70a8a7 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:28.793874 2409 leader_election.cc:290] T 05630a72ddae4de0b890dece2feacc31 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:28.800978 858 consensus_peers.cc:489] T 71941b815d6840478c5acc1cbb187cc9 P 00c282f779c1477d920f24dc0dc16628 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:28.803818 1150 tablet_bootstrap.cc:654] T 2dacb608897d47d28512df87dc320927 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:28.803458 992 leader_election.cc:336] T aec951f2cc84499b9162b9fa24d5440e P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:28.804598 2118 raft_consensus.cc:2270] T 7f64ebd080944190be2e64d02d70a8a7 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250901 01:52:28.806568 992 leader_election.cc:336] T 482a9a2cb81f448bb2823892e35920d4 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:28.813091 992 consensus_peers.cc:489] T 606fac5c6efe48d1877104b01c23ba5b P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250901 01:52:28.822759 1150 tablet_bootstrap.cc:492] T 2dacb608897d47d28512df87dc320927 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:28.823022 1150 ts_tablet_manager.cc:1397] T 2dacb608897d47d28512df87dc320927 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.057s user 0.003s sys 0.002s
I20250901 01:52:28.823882 1150 raft_consensus.cc:357] T 2dacb608897d47d28512df87dc320927 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:28.824546 1150 raft_consensus.cc:383] T 2dacb608897d47d28512df87dc320927 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:28.824750 1150 raft_consensus.cc:738] T 2dacb608897d47d28512df87dc320927 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:28.825117 1150 consensus_queue.cc:260] T 2dacb608897d47d28512df87dc320927 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:28.830938 1150 ts_tablet_manager.cc:1428] T 2dacb608897d47d28512df87dc320927 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250901 01:52:28.832926 1150 tablet_bootstrap.cc:492] T fecf7d538b4c4c619182d5eccb44dd8b P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:28.835913 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d10c193452954c198bba686169b8025b" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:28.840330 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a9b7f2e6548043a7865e25387cdd06c9" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:28.842437 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45f6e8ba99b74898a2922b36768c6c36" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:28.842859 2118 ts_tablet_manager.cc:1905] T 7f64ebd080944190be2e64d02d70a8a7 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:28.848651 1150 tablet_bootstrap.cc:654] T fecf7d538b4c4c619182d5eccb44dd8b P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:28.852540 1151 tablet_bootstrap.cc:492] T 14b466b2a5a249aa841f1b8b06301edc P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:28.852722 1151 ts_tablet_manager.cc:1397] T 14b466b2a5a249aa841f1b8b06301edc P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.160s user 0.001s sys 0.013s
I20250901 01:52:28.852764 2323 raft_consensus.cc:491] T da5f69c5b0ca4257ad2ff7918ee2af2e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:28.852910 2323 raft_consensus.cc:513] T da5f69c5b0ca4257ad2ff7918ee2af2e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:28.853377 1151 raft_consensus.cc:357] T 14b466b2a5a249aa841f1b8b06301edc P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:28.853499 2323 leader_election.cc:290] T da5f69c5b0ca4257ad2ff7918ee2af2e P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:28.853549 1151 raft_consensus.cc:383] T 14b466b2a5a249aa841f1b8b06301edc P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:28.853595 1151 raft_consensus.cc:738] T 14b466b2a5a249aa841f1b8b06301edc P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:28.854053 1151 consensus_queue.cc:260] T 14b466b2a5a249aa841f1b8b06301edc P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:28.855011 1151 ts_tablet_manager.cc:1428] T 14b466b2a5a249aa841f1b8b06301edc P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250901 01:52:28.855558 1151 tablet_bootstrap.cc:492] T 615b2306fca84003b264da9d5f95ddab P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
W20250901 01:52:28.857187 858 leader_election.cc:336] T da5f69c5b0ca4257ad2ff7918ee2af2e P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:28.859064 1484 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet a6445b29591548be9f9577db1ba0bfa5
W20250901 01:52:28.859386 1484 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet a6445b29591548be9f9577db1ba0bfa5: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:28.860239 887 tablet_service.cc:1515] Processing DeleteTablet for tablet a6445b29591548be9f9577db1ba0bfa5 with delete_type TABLET_DATA_DELETED (replaced by tablet 3a62034e688348ed9e2ac9de009ed3a2) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:28.861024 1033 tablet_service.cc:1515] Processing DeleteTablet for tablet a6445b29591548be9f9577db1ba0bfa5 with delete_type TABLET_DATA_DELETED (replaced by tablet 3a62034e688348ed9e2ac9de009ed3a2) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:28.861325 1484 catalog_manager.cc:6652] ReplaceTablet: tablet a6445b29591548be9f9577db1ba0bfa5 deleted and replaced by tablet 3a62034e688348ed9e2ac9de009ed3a2
I20250901 01:52:28.859489 1490 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 1495de70f0844f82a0eea99c059f076b
W20250901 01:52:28.863152 1490 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 1495de70f0844f82a0eea99c059f076b: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:28.863914 1033 tablet_service.cc:1515] Processing DeleteTablet for tablet 1495de70f0844f82a0eea99c059f076b with delete_type TABLET_DATA_DELETED (replaced by tablet f5e3316adeae47cd848833dd84b972bb) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:28.864670 1490 catalog_manager.cc:6652] ReplaceTablet: tablet 1495de70f0844f82a0eea99c059f076b deleted and replaced by tablet f5e3316adeae47cd848833dd84b972bb
I20250901 01:52:28.864835 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 1495de70f0844f82a0eea99c059f076b with delete_type TABLET_DATA_DELETED (replaced by tablet f5e3316adeae47cd848833dd84b972bb) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:28.865149 2118 ts_tablet_manager.cc:1918] T 7f64ebd080944190be2e64d02d70a8a7 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:28.865706 1489 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 8b9d79a25452410891c09fd3f7c4be6d
W20250901 01:52:28.865892 1489 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 8b9d79a25452410891c09fd3f7c4be6d: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:28.866165 1489 catalog_manager.cc:6652] ReplaceTablet: tablet 8b9d79a25452410891c09fd3f7c4be6d deleted and replaced by tablet 0ee3f35392954817b350e5f48653be78
I20250901 01:52:28.867216 1033 tablet_service.cc:1515] Processing DeleteTablet for tablet 8b9d79a25452410891c09fd3f7c4be6d with delete_type TABLET_DATA_DELETED (replaced by tablet 0ee3f35392954817b350e5f48653be78) from {username='slave'} at 127.0.0.1:35946
W20250901 01:52:28.858347 992 consensus_peers.cc:489] T 3c1fd0820d8a4995abe027f685da7f7b P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250901 01:52:28.876225 1489 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_1e59568bdcd14662ae9d5e98293d74bd" } modify_external_catalogs: true
I20250901 01:52:28.876456 1489 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_1e59568bdcd14662ae9d5e98293d74bd" } modify_external_catalogs: true
I20250901 01:52:28.876470 1490 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_b0e916c76a7a4766aa0887cc8691488f" } modify_external_catalogs: true
I20250901 01:52:28.877811 1490 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_b0e916c76a7a4766aa0887cc8691488f" } modify_external_catalogs: true
I20250901 01:52:28.877952 1484 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_a4ebd8ab87c842f082da5cd04d1bddac" } modify_external_catalogs: true
I20250901 01:52:28.878123 1484 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_a4ebd8ab87c842f082da5cd04d1bddac" } modify_external_catalogs: true
I20250901 01:52:28.867189 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 8b9d79a25452410891c09fd3f7c4be6d with delete_type TABLET_DATA_DELETED (replaced by tablet 0ee3f35392954817b350e5f48653be78) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:28.880096 2118 log.cc:1199] T 7f64ebd080944190be2e64d02d70a8a7 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/7f64ebd080944190be2e64d02d70a8a7
W20250901 01:52:28.880316 859 leader_election.cc:343] T d10c193452954c198bba686169b8025b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:28.880487 859 leader_election.cc:304] T d10c193452954c198bba686169b8025b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:28.880616 2118 ts_tablet_manager.cc:1939] T 7f64ebd080944190be2e64d02d70a8a7 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
W20250901 01:52:28.880945 859 leader_election.cc:343] T a9b7f2e6548043a7865e25387cdd06c9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:28.881042 859 leader_election.cc:304] T a9b7f2e6548043a7865e25387cdd06c9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:28.881381 859 leader_election.cc:343] T 45f6e8ba99b74898a2922b36768c6c36 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:28.881503 859 leader_election.cc:304] T 45f6e8ba99b74898a2922b36768c6c36 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:28.881922 2323 raft_consensus.cc:2747] T d10c193452954c198bba686169b8025b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:28.882091 2323 raft_consensus.cc:2747] T a9b7f2e6548043a7865e25387cdd06c9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:28.882226 2323 raft_consensus.cc:2747] T 45f6e8ba99b74898a2922b36768c6c36 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:28.883486 992 leader_election.cc:304] T 482a9a2cb81f448bb2823892e35920d4 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:28.883847 1051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da5f69c5b0ca4257ad2ff7918ee2af2e" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:28.884088 1150 tablet_bootstrap.cc:492] T fecf7d538b4c4c619182d5eccb44dd8b P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:28.881924 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 89) took 3530 ms. Trace:
I20250901 01:52:28.884358 1150 ts_tablet_manager.cc:1397] T fecf7d538b4c4c619182d5eccb44dd8b P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.052s user 0.002s sys 0.004s
I20250901 01:52:28.884279 2118 rpcz_store.cc:276] 0901 01:52:25.351810 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.352080 (+ 270us) service_pool.cc:225] Handling call
0901 01:52:28.770633 (+3418553us) ts_tablet_manager.cc:1145] Deleting tablet 7f64ebd080944190be2e64d02d70a8a7
0901 01:52:28.770762 (+ 129us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:28.862034 (+ 91272us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:28.865048 (+ 3014us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:28.881908 (+ 16860us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3418056}
I20250901 01:52:28.884882 1151 tablet_bootstrap.cc:654] T 615b2306fca84003b264da9d5f95ddab P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:28.884925 2118 tablet_replica.cc:331] T 25ce0fe8c9544f828694184d19d13df9 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:28.884898 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 7f64ebd080944190be2e64d02d70a8a7 (table default.Table_f94fe76831a54c6e8c2b13c81c251b14 [id=8b4cf5984d174d71a736959f5db32144]) successfully deleted
I20250901 01:52:28.885191 2118 raft_consensus.cc:2241] T 25ce0fe8c9544f828694184d19d13df9 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:28.885412 2118 raft_consensus.cc:2270] T 25ce0fe8c9544f828694184d19d13df9 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:28.886431 2357 raft_consensus.cc:491] T dabebc77f72e4bb19e264be2f496202a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:28.886565 2357 raft_consensus.cc:513] T dabebc77f72e4bb19e264be2f496202a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:28.886924 2118 ts_tablet_manager.cc:1905] T 25ce0fe8c9544f828694184d19d13df9 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:28.887198 2357 leader_election.cc:290] T dabebc77f72e4bb19e264be2f496202a P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:28.887203 2113 ts_tablet_manager.cc:1918] T 438cd35ff3f74c49b908d072560077db P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:28.887311 2113 log.cc:1199] T 438cd35ff3f74c49b908d072560077db P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/438cd35ff3f74c49b908d072560077db
I20250901 01:52:28.887713 2113 ts_tablet_manager.cc:1939] T 438cd35ff3f74c49b908d072560077db P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:28.888175 1150 raft_consensus.cc:357] T fecf7d538b4c4c619182d5eccb44dd8b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:28.888556 2412 raft_consensus.cc:2802] T 482a9a2cb81f448bb2823892e35920d4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:28.888538 1150 raft_consensus.cc:383] T fecf7d538b4c4c619182d5eccb44dd8b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:28.888640 2412 raft_consensus.cc:491] T 482a9a2cb81f448bb2823892e35920d4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:28.888720 2412 raft_consensus.cc:3058] T 482a9a2cb81f448bb2823892e35920d4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
W20250901 01:52:28.888687 858 leader_election.cc:336] T dabebc77f72e4bb19e264be2f496202a P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:28.888671 1150 raft_consensus.cc:738] T fecf7d538b4c4c619182d5eccb44dd8b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:28.889163 1150 consensus_queue.cc:260] T fecf7d538b4c4c619182d5eccb44dd8b P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:28.891335 1150 ts_tablet_manager.cc:1428] T fecf7d538b4c4c619182d5eccb44dd8b P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250901 01:52:28.891289 2412 raft_consensus.cc:513] T 482a9a2cb81f448bb2823892e35920d4 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:28.891891 2412 leader_election.cc:290] T 482a9a2cb81f448bb2823892e35920d4 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:28.892254 1150 tablet_bootstrap.cc:492] T 0e4009963cfe42d6ba164fc828897f09 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:28.893431 992 leader_election.cc:304] T aec951f2cc84499b9162b9fa24d5440e P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:28.893702 2412 raft_consensus.cc:2802] T aec951f2cc84499b9162b9fa24d5440e P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:28.893958 2412 raft_consensus.cc:695] T aec951f2cc84499b9162b9fa24d5440e P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:28.894244 2412 consensus_queue.cc:237] T aec951f2cc84499b9162b9fa24d5440e P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:28.898667 1150 tablet_bootstrap.cc:654] T 0e4009963cfe42d6ba164fc828897f09 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:28.902974 992 leader_election.cc:343] T 05630a72ddae4de0b890dece2feacc31 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 05630a72ddae4de0b890dece2feacc31
I20250901 01:52:28.905367 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 93) took 3827 ms. Trace:
I20250901 01:52:28.906288 2409 consensus_queue.cc:1035] T f13b972e25214f168421439fe3943e8a P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:28.906924 2113 rpcz_store.cc:276] 0901 01:52:25.078385 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.080045 (+ 1660us) service_pool.cc:225] Handling call
0901 01:52:28.775675 (+3695630us) ts_tablet_manager.cc:1145] Deleting tablet 438cd35ff3f74c49b908d072560077db
0901 01:52:28.786323 (+ 10648us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:28.841346 (+ 55023us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:28.887100 (+ 45754us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:28.905329 (+ 18229us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15186,"tablet-delete.queue_time_us":3681847}
I20250901 01:52:28.908180 2113 tablet_replica.cc:331] T b73e362321284df1b18ef755dfe930a5 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:28.908301 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 438cd35ff3f74c49b908d072560077db (table default.Table_62ececf8293e4e22bad9e9568f496bbe [id=5a33cfdf60684867bb5f36d84ee80468]) successfully deleted
I20250901 01:52:28.909097 2113 raft_consensus.cc:2241] T b73e362321284df1b18ef755dfe930a5 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:28.911420 2118 ts_tablet_manager.cc:1918] T 25ce0fe8c9544f828694184d19d13df9 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:28.921626 2113 raft_consensus.cc:2270] T b73e362321284df1b18ef755dfe930a5 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:28.923269 2113 ts_tablet_manager.cc:1905] T b73e362321284df1b18ef755dfe930a5 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:28.935191 2113 ts_tablet_manager.cc:1918] T b73e362321284df1b18ef755dfe930a5 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:28.933784 1150 tablet_bootstrap.cc:492] T 0e4009963cfe42d6ba164fc828897f09 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:28.936512 2118 log.cc:1199] T 25ce0fe8c9544f828694184d19d13df9 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/25ce0fe8c9544f828694184d19d13df9
I20250901 01:52:28.936733 1150 ts_tablet_manager.cc:1397] T 0e4009963cfe42d6ba164fc828897f09 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.045s user 0.006s sys 0.002s
I20250901 01:52:28.936900 2118 ts_tablet_manager.cc:1939] T 25ce0fe8c9544f828694184d19d13df9 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:28.937242 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 91) took 3585 ms. Trace:
I20250901 01:52:28.937328 2118 rpcz_store.cc:276] 0901 01:52:25.352029 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.352079 (+ 50us) service_pool.cc:225] Handling call
0901 01:52:28.884877 (+3532798us) ts_tablet_manager.cc:1145] Deleting tablet 25ce0fe8c9544f828694184d19d13df9
0901 01:52:28.884888 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:28.907876 (+ 22988us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:28.911308 (+ 3432us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:28.937224 (+ 25916us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":757,"tablet-delete.queue_time_us":3532419}
I20250901 01:52:28.937929 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 25ce0fe8c9544f828694184d19d13df9 (table default.Table_dcacb44eed4241d3acbae153ebaad663 [id=0241aa3ab1984557a151461005c6b97c]) successfully deleted
I20250901 01:52:28.938369 1150 raft_consensus.cc:357] T 0e4009963cfe42d6ba164fc828897f09 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:28.938746 1150 raft_consensus.cc:383] T 0e4009963cfe42d6ba164fc828897f09 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:28.938887 1150 raft_consensus.cc:738] T 0e4009963cfe42d6ba164fc828897f09 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:28.939322 1150 consensus_queue.cc:260] T 0e4009963cfe42d6ba164fc828897f09 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:28.940811 1150 ts_tablet_manager.cc:1428] T 0e4009963cfe42d6ba164fc828897f09 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250901 01:52:28.938889 2118 tablet_replica.cc:331] T eaa521c52a2f4408b59770bd2a05907c P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:28.941056 1486 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet ed1b41340ce742a88c3e4fd85cd25ca6 (table default.Table_139f6d174d794a6e9a618e90587d0c0d [id=0baa4993b72e4a8d94ee3059c9d2db6c]). Expected version 1 got 0
I20250901 01:52:28.941365 1486 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 14b466b2a5a249aa841f1b8b06301edc (table default.Table_927955c1048f4e9c84b5b3a113a473f7 [id=78cd71321e9a480ca2d7d88327f157ee]). Expected version 1 got 0
I20250901 01:52:28.941639 1486 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet e87d1281f35648b0b115b67c95e244e7 (table default.Table_f840789a58984799bd8057128f6ea439 [id=4453959e273c473fadd737f56f7db3dd]). Expected version 1 got 0
W20250901 01:52:28.907513 992 consensus_peers.cc:489] T d77aadd7291647a0bae7c421f244c252 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250901 01:52:28.941869 992 leader_election.cc:336] T 05630a72ddae4de0b890dece2feacc31 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:28.941938 1486 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 7954393c8c1c4457ab8bb723c03a9c8b (table default.Table_7050639ebd0147959ac4a251f534eb96 [id=543f2074598d4c29972e564be1123fca]). Expected version 5 got 0
I20250901 01:52:28.941998 992 leader_election.cc:304] T 05630a72ddae4de0b890dece2feacc31 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:28.942152 1486 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 6c0abbfc8d904dce989ce51961e4237d (table default.Table_3f9422b08b0349b9a5466218f0fe1b06 [id=f0dfb3811e744bd4ba6d59195cebde6f]). Expected version 2 got 0
I20250901 01:52:28.942355 1486 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet aec951f2cc84499b9162b9fa24d5440e (table default.Table_156d1feafd5246ff8ff1bb7b9ae444a5 [id=55c31cd42fac43e6b8eddc048f565d0a]). Expected version 1 got 0
I20250901 01:52:28.942585 1486 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet c6f7c525f59b49938b697de98d4b5835 (table default.Table_db61e055dbd7450898e82c4c0c4763be [id=aabca67f6f754820977ac9c4c20f3e4f]). Expected version 1 got 0
W20250901 01:52:28.942914 859 leader_election.cc:343] T da5f69c5b0ca4257ad2ff7918ee2af2e P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:28.943030 859 leader_election.cc:304] T da5f69c5b0ca4257ad2ff7918ee2af2e P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:28.942832 1486 catalog_manager.cc:5582] T f13b972e25214f168421439fe3943e8a P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: UNKNOWN } } }
I20250901 01:52:28.943454 2408 consensus_queue.cc:1035] T 2599f98abfdc4871b573edfc2ad97d22 P 00c282f779c1477d920f24dc0dc16628 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:28.943367 1486 catalog_manager.cc:5582] T ea846f88b3534beebd0001b7f17b5ca9 P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: UNKNOWN } } }
I20250901 01:52:28.943833 2408 raft_consensus.cc:2747] T da5f69c5b0ca4257ad2ff7918ee2af2e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:28.943850 2412 raft_consensus.cc:2747] T 05630a72ddae4de0b890dece2feacc31 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:28.943950 1486 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 081e2630e284498aa819f85bf8326315 (table default.Table_b3e694397be14321a668e46c09c51a71 [id=8199118617cb4707bc3ae9b1363750f7]). Expected version 3 got 0
I20250901 01:52:28.944777 919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "482a9a2cb81f448bb2823892e35920d4" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:28.944911 919 raft_consensus.cc:3058] T 482a9a2cb81f448bb2823892e35920d4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:28.946128 1486 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet f80f13db501c42c7950499df8fcb6fc8 (table default.Table_ce86974fd8974de9a5a737954fac2ad7 [id=ff772d20b287476f94716379b86d2a71]). Expected version 1 got 0
I20250901 01:52:28.946293 1486 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 33f637f6a8134215bb1995ec38440073 (table default.Table_3d5afc22c4e14d6990aea31564dfde20 [id=557afe29976d40f6bfb6ae4ca348f887]). Expected version 2 got 0
I20250901 01:52:28.946480 1486 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet d0c3c363dda444519a17dca63ad9caa7 (table default.Table_42422c44ca254d8ca989134c2433391f [id=8c1b263824ec4c15a5ec424364c78895]). Expected version 3 got 0
I20250901 01:52:28.946802 1486 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet e9f1f892658b4736859ff507536920a1 (table default.Table_8825dd28afbe464193eb1b668cc9326a [id=9aa62cb634314a0a849612c54c1220d9]). Expected version 1 got 0
I20250901 01:52:28.947016 1486 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 44af4764d4f64e44aec84ccb84be6f5f (table default.Table_9896b980158948339a0b91f5882ed56b [id=a541b55be0dc420a8353ef4793b925b5]). Expected version 1 got 0
I20250901 01:52:28.947326 1486 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 5f5396045979450193d461ba1334a5c9 (table default.Table_a89cc8e48563460baabc34c821ceae35 [id=9c3930e9fcfe4b49ba4bea67a2b0ad07]). Expected version 1 got 0
I20250901 01:52:28.947631 919 raft_consensus.cc:2466] T 482a9a2cb81f448bb2823892e35920d4 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:28.947870 2412 raft_consensus.cc:491] T c9a4c80bec1d41458974a0c449aecaf7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:28.948004 2412 raft_consensus.cc:513] T c9a4c80bec1d41458974a0c449aecaf7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:28.948136 1486 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet b164b4a67a9143ceaff69b0ce49d63fe (table default.Table_da2c70a8f2fe49a0a8184e88c0a9e0ea [id=c1b1e9e3876244f68d31c63ce7bdebf1]). Expected version 1 got 0
I20250901 01:52:28.948150 1150 tablet_bootstrap.cc:492] T 2973b59bb1df41d18d9a5b6a29ae336f P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:28.948258 1486 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet ae21b9ae2e3b472eb67fb5f96388a50f (table default.Table_046289f393ff432aae0b033bdec1fb17 [id=db57b18a9ce24e068b362f5fbe940531]). Expected version 1 got 0
I20250901 01:52:28.948683 1486 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 45f6e8ba99b74898a2922b36768c6c36 (table default.Table_938421db3cf74cf8b878089bac931818 [id=9ce26dcf307149b595a43b6ac4562fba]). Expected version 1 got 0
I20250901 01:52:28.948922 2412 leader_election.cc:290] T c9a4c80bec1d41458974a0c449aecaf7 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:28.922770 1151 tablet_bootstrap.cc:492] T 615b2306fca84003b264da9d5f95ddab P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:28.949155 1151 ts_tablet_manager.cc:1397] T 615b2306fca84003b264da9d5f95ddab P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.094s user 0.001s sys 0.005s
I20250901 01:52:28.949155 2118 raft_consensus.cc:2241] T eaa521c52a2f4408b59770bd2a05907c P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:28.949822 1151 raft_consensus.cc:357] T 615b2306fca84003b264da9d5f95ddab P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:28.949954 1151 raft_consensus.cc:383] T 615b2306fca84003b264da9d5f95ddab P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:28.949996 1151 raft_consensus.cc:738] T 615b2306fca84003b264da9d5f95ddab P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:28.949923 2118 raft_consensus.cc:2270] T eaa521c52a2f4408b59770bd2a05907c P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:28.950166 1151 consensus_queue.cc:260] T 615b2306fca84003b264da9d5f95ddab P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:28.950675 1151 ts_tablet_manager.cc:1428] T 615b2306fca84003b264da9d5f95ddab P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250901 01:52:28.950690 1486 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 0e4009963cfe42d6ba164fc828897f09 (table default.Table_1adc70cccd2f4cb3a79df96861cb4181 [id=24fffc7ebf5a47bdb37211910301acf7]). Expected version 1 got 0
I20250901 01:52:28.950950 1151 tablet_bootstrap.cc:492] T ff013ea889d44cba8797834e4611b489 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:28.950945 1486 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 93bcac1fd95d44dc927abfca81692921 (table default.Table_b8f8ee74648c406eaf589d3ab80934db [id=b8f69313439543be9c11eea88dfa69c5]). Expected version 1 got 0
I20250901 01:52:28.951170 1486 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 508406601c994e42a5c5b1f665e74957 (table default.Table_345b7f9fe8214aa6878f20a166125364 [id=4aabc328f6374ec5b1dea26f778208b2]). Expected version 1 got 0
I20250901 01:52:28.951412 1486 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 5d68ea97c71942c7b55b967ba870e0d7 (table default.Table_f610cb75906d4c5cb3b57cfdb80ecefc [id=36915a3e843340ba8ef3578ac35a9af4]). Expected version 1 got 0
I20250901 01:52:28.951563 1486 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet d000ba05e12843b4925e622ab55471c0 (table default.Table_e0902dfbdd984f85b3fdac8d283731f7 [id=b43a92ed46164d199b445287ef039bc6]). Expected version 1 got 0
I20250901 01:52:28.952639 1486 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 3618a3eb4f174c43805ce7dc0e5315cb (table default.Table_83f2c0acedbc4c6982caf5313646e90b [id=31a2e50c32644321825da40176127015]). Expected version 1 got 0
I20250901 01:52:28.952862 1486 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet d1f2d98712ff4ea6aec665512de068bd (table default.Table_82d91a1ba2eb4265896824028dec8d3a [id=3f7624d2901445e39155c12fa68e34e9]). Expected version 1 got 0
I20250901 01:52:28.953263 1486 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 615b2306fca84003b264da9d5f95ddab (table default.Table_302f6e7cf97d46b9a621eb852e6aefe0 [id=cecdd8d6ca6d445c9fd82decd2a95366]). Expected version 2 got 0
W20250901 01:52:28.953845 992 consensus_peers.cc:489] T 89d40f0331114490b51de3ba6d773571 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:28.953935 1150 tablet_bootstrap.cc:654] T 2973b59bb1df41d18d9a5b6a29ae336f P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:28.956105 1051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dabebc77f72e4bb19e264be2f496202a" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:28.956357 2118 ts_tablet_manager.cc:1905] T eaa521c52a2f4408b59770bd2a05907c P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:28.961006 1486 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet ad45c90ba0da43c9860745e534664a98 (table default.Table_1cd3f3fb6b244b3f8796bfca9f7622c8 [id=6fd739b0e725424ca844dbb652ea8028]). Expected version 2 got 0
W20250901 01:52:28.961812 992 consensus_peers.cc:489] T 1286c0ab85704837a99b6e5a62184d77 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Couldn't send request to peer 00c282f779c1477d920f24dc0dc16628. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1286c0ab85704837a99b6e5a62184d77. This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:28.961961 992 leader_election.cc:304] T 482a9a2cb81f448bb2823892e35920d4 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:28.962538 2409 raft_consensus.cc:2802] T 482a9a2cb81f448bb2823892e35920d4 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:28.962625 2409 raft_consensus.cc:695] T 482a9a2cb81f448bb2823892e35920d4 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:28.962821 2409 consensus_queue.cc:237] T 482a9a2cb81f448bb2823892e35920d4 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:28.964403 1486 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet fecf7d538b4c4c619182d5eccb44dd8b (table default.Table_1b5729d9768546fe99eaafc4e2604f42 [id=02d0056abe2146ac9af81ebf4447b99d]). Expected version 1 got 0
I20250901 01:52:28.965209 1486 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 8ca5cf4e99014c0d84a3acad4ef713af (table default.Table_ed6fd284be6f4ae3a008e21e7ce17930 [id=303e76c918bc4c39aa47a4d24849a358]). Expected version 1 got 0
I20250901 01:52:28.966004 1486 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 61ddcf86c9f34726939cc116434f0822 (table default.Table_63d0f2ead9d946d39ce40162ebd769d6 [id=f9dcf4937b1143248bfa9d88053fce17]). Expected version 3 got 0
I20250901 01:52:28.966789 1486 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 858218c4427c41e980b2b5c969a5af40 (table default.Table_6d47266af0ed433eb1f2abbc81d18e69 [id=264cc469a73841e79538843479a165ab]). Expected version 1 got 0
I20250901 01:52:28.967597 1486 catalog_manager.cc:5582] T 89d40f0331114490b51de3ba6d773571 P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: UNKNOWN } } }
I20250901 01:52:28.968456 2357 raft_consensus.cc:491] T 1495de70f0844f82a0eea99c059f076b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:28.968504 1486 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 89d40f0331114490b51de3ba6d773571 (table default.Table_4dacb3b1d2e94e42ac8937eb6eb58375 [id=8ee0aa1ba24042dd8ded004644913b9c]). Expected version 1 got 0
I20250901 01:52:28.968595 2357 raft_consensus.cc:513] T 1495de70f0844f82a0eea99c059f076b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:28.969162 2357 leader_election.cc:290] T 1495de70f0844f82a0eea99c059f076b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:28.970386 858 leader_election.cc:336] T 1495de70f0844f82a0eea99c059f076b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:28.971483 1486 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet f4fd117c395e41c2ab679c684908c5b7 (table default.Table_97526c9a6c9f4acbb6dedd78475571e0 [id=305490ecfddb4d46927f7bb128c7f878]). Expected version 1 got 0
I20250901 01:52:28.971946 1486 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 81ce448b148447ddbc41875d863e0518 (table default.Table_6e461d9cb0ae4dd095975bbc21533794 [id=49673fa73f8c436e9b8026efc4f3cbbc]). Expected version 1 got 0
W20250901 01:52:28.972815 1488 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2973b59bb1df41d18d9a5b6a29ae336f consensus state has no leader
I20250901 01:52:28.972887 1489 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 7304937048ca48a7bd153f3cdcfaeff0
W20250901 01:52:28.973065 1489 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 7304937048ca48a7bd153f3cdcfaeff0: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:28.973318 1488 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_faf7ef12324f454396e5ae64d64ebe51" } modify_external_catalogs: true
I20250901 01:52:28.973579 1488 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_faf7ef12324f454396e5ae64d64ebe51" } modify_external_catalogs: true
I20250901 01:52:28.973758 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_09a698ad886c4c3685fe3b2df03c462c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.974713 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_bb9417ac39f34d498fe4e4426a7cbb23"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:28.975016 1491 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 14b466b2a5a249aa841f1b8b06301edc consensus state has no leader
I20250901 01:52:28.975409 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_63c0e9b48b3347bb9398a32a76ae224c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.975507 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_6481cc8f024c4832b5f1f1f6abb2a2bf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.976212 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_85ab40f0d10c4d7ab70c604c73f4cb3b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.976258 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9ed4ba499b974c01a3f33eb6253dbee7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.977233 2409 consensus_queue.cc:1035] T ea846f88b3534beebd0001b7f17b5ca9 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:28.978389 1150 tablet_bootstrap.cc:492] T 2973b59bb1df41d18d9a5b6a29ae336f P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:28.978681 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 7304937048ca48a7bd153f3cdcfaeff0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:28 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:28.978665 1150 ts_tablet_manager.cc:1397] T 2973b59bb1df41d18d9a5b6a29ae336f P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.031s user 0.006s sys 0.000s
W20250901 01:52:28.979429 992 leader_election.cc:343] T c9a4c80bec1d41458974a0c449aecaf7 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: c9a4c80bec1d41458974a0c449aecaf7
I20250901 01:52:28.981156 1150 raft_consensus.cc:357] T 2973b59bb1df41d18d9a5b6a29ae336f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:28.981690 2323 raft_consensus.cc:491] T fecf7d538b4c4c619182d5eccb44dd8b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:28.981839 2323 raft_consensus.cc:513] T fecf7d538b4c4c619182d5eccb44dd8b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:28.982353 1151 tablet_bootstrap.cc:654] T ff013ea889d44cba8797834e4611b489 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:28.982457 2323 leader_election.cc:290] T fecf7d538b4c4c619182d5eccb44dd8b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:28.982772 1033 tablet_service.cc:1515] Processing DeleteTablet for tablet 7304937048ca48a7bd153f3cdcfaeff0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:28 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:28.981575 1150 raft_consensus.cc:383] T 2973b59bb1df41d18d9a5b6a29ae336f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:28.983595 1150 raft_consensus.cc:738] T 2973b59bb1df41d18d9a5b6a29ae336f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:28.984244 1150 consensus_queue.cc:260] T 2973b59bb1df41d18d9a5b6a29ae336f P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
W20250901 01:52:28.986227 858 leader_election.cc:336] T fecf7d538b4c4c619182d5eccb44dd8b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:28.986388 858 consensus_peers.cc:489] T e87d1281f35648b0b115b67c95e244e7 P 00c282f779c1477d920f24dc0dc16628 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:28.987186 1150 ts_tablet_manager.cc:1428] T 2973b59bb1df41d18d9a5b6a29ae336f P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
W20250901 01:52:28.988610 859 leader_election.cc:343] T dabebc77f72e4bb19e264be2f496202a P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
W20250901 01:52:28.988656 992 leader_election.cc:336] T 482a9a2cb81f448bb2823892e35920d4 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:28.988734 859 leader_election.cc:304] T dabebc77f72e4bb19e264be2f496202a P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:28.988879 992 consensus_peers.cc:489] T 8ed6616ce54548ba83ba299c4ecf3c38 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250901 01:52:28.988965 992 consensus_peers.cc:489] T f13b972e25214f168421439fe3943e8a P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:28.988952 2357 raft_consensus.cc:2747] T dabebc77f72e4bb19e264be2f496202a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:28.989792 992 consensus_peers.cc:489] T 5d794471dbed4477b9d52fd411fa4d10 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250901 01:52:28.989830 1150 tablet_bootstrap.cc:492] T 5f5396045979450193d461ba1334a5c9 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:28.991104 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_381ba4eba5234d4ab9307df159af2a86"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.992084 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_32ceffa2e2354da9b2cb8d091bc9a7c6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.993471 1488 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 76ba642c16734cdf969dbd3130639fda
I20250901 01:52:28.977061 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f3713c8f61f04d67a928a4e4307f891b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.995568 1491 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_2b74c001cb4f4d82ab314557f4f09422" } new_table_name: "default.Table_dd92add170f94aaf966f77d81e2c3883" modify_external_catalogs: true
I20250901 01:52:28.995810 2118 ts_tablet_manager.cc:1918] T eaa521c52a2f4408b59770bd2a05907c P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:28.996037 1488 master_service.cc:948] ReplaceTablet: received request to replace tablet e6c10426f60f43e8bde69d9b1b9c8eb6 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:28.996443 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e80c059bd16c4890a568ce5f9e4d8139"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:28.996433 2118 log.cc:1199] T eaa521c52a2f4408b59770bd2a05907c P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/eaa521c52a2f4408b59770bd2a05907c
I20250901 01:52:28.997488 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1495de70f0844f82a0eea99c059f076b" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:28.997566 1489 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_99855394cd424528b7f54ee20375788e" } new_table_name: "default.Table_a07fd8be1b7540a280aca916e23b221e" modify_external_catalogs: true
I20250901 01:52:28.998382 1051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fecf7d538b4c4c619182d5eccb44dd8b" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:29.000123 2118 ts_tablet_manager.cc:1939] T eaa521c52a2f4408b59770bd2a05907c P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:29.001504 1150 tablet_bootstrap.cc:654] T 5f5396045979450193d461ba1334a5c9 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:29.002271 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 92) took 3649 ms. Trace:
I20250901 01:52:29.002593 2118 rpcz_store.cc:276] 0901 01:52:25.352309 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.352371 (+ 62us) service_pool.cc:225] Handling call
0901 01:52:28.938839 (+3586468us) ts_tablet_manager.cc:1145] Deleting tablet eaa521c52a2f4408b59770bd2a05907c
0901 01:52:28.938852 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:28.968176 (+ 29324us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:28.995210 (+ 27034us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:29.002096 (+ 6886us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3586190}
I20250901 01:52:29.004169 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet eaa521c52a2f4408b59770bd2a05907c (table default.Table_14cf61059c5145b0a02159aae2a3e457 [id=a33b9b981bd64d7981829d663b0b48b5]) successfully deleted
I20250901 01:52:29.004941 2118 tablet_replica.cc:331] T c58edaa76ebf47dfb1d9d5fab6de7dfa P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:29.005157 2113 log.cc:1199] T b73e362321284df1b18ef755dfe930a5 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/b73e362321284df1b18ef755dfe930a5
I20250901 01:52:29.005635 2113 ts_tablet_manager.cc:1939] T b73e362321284df1b18ef755dfe930a5 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
W20250901 01:52:29.011677 1489 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e5e250e6c84d4248b857bf1478d47703 consensus state has no leader
I20250901 01:52:29.009197 1151 tablet_bootstrap.cc:492] T ff013ea889d44cba8797834e4611b489 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:29.012130 1151 ts_tablet_manager.cc:1397] T ff013ea889d44cba8797834e4611b489 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.061s user 0.005s sys 0.000s
I20250901 01:52:29.012127 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_fb69b5383c814ec6993aedb3f8e78023"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.012929 1151 raft_consensus.cc:357] T ff013ea889d44cba8797834e4611b489 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:29.013118 1151 raft_consensus.cc:383] T ff013ea889d44cba8797834e4611b489 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:29.013185 1151 raft_consensus.cc:738] T ff013ea889d44cba8797834e4611b489 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:29.013219 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7b97bfa563674ef18beab7e9212296c7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.013376 1151 consensus_queue.cc:260] T ff013ea889d44cba8797834e4611b489 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:29.013912 1151 ts_tablet_manager.cc:1428] T ff013ea889d44cba8797834e4611b489 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250901 01:52:29.014175 1151 tablet_bootstrap.cc:492] T ad1a6be9fcc045d3bb4e71a2f6bd5ccd P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:29.014380 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a3fe02b3dd074634ac6b1a63891b8ef3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.015193 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2fa48d516b8e4a6a98c03a409473e679"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.015877 1151 tablet_bootstrap.cc:654] T ad1a6be9fcc045d3bb4e71a2f6bd5ccd P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:29.016242 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_acc2d3c3e7be434ab0c774adbf18aecf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:29.016790 1491 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ea846f88b3534beebd0001b7f17b5ca9 consensus state has no leader
I20250901 01:52:29.016957 2357 raft_consensus.cc:491] T 617cc33d0d23439cb5f1e0c31bf11281 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250901 01:52:29.017079 1483 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9745582cebfe47b58c21d0191ab4cb9b has no consensus state
I20250901 01:52:29.017103 2357 raft_consensus.cc:513] T 617cc33d0d23439cb5f1e0c31bf11281 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:29.017258 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2c886ea7936f4b85b1c342aeea472a82"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.017360 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a1991c4b0034461bbbe5acf4a89f6174"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.017699 2357 leader_election.cc:290] T 617cc33d0d23439cb5f1e0c31bf11281 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:29.017905 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a77902bb18ca47dfabe6383c4eea319e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.018625 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1c5331059d444bcb9748b1a513c76392"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:29.019110 858 leader_election.cc:336] T 617cc33d0d23439cb5f1e0c31bf11281 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:29.017881 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d3bd08aa1f414b42a1b73bb6f628dbb5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.020882 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_058bc89c06444d968235f9f3a516c2e8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.021550 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d155c4c88ba04671a5cb16556b7a880b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.022215 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c7b2d118e6ec4d89af7993dcc8c761c5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.022670 1485 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 186ec810d02e4748aad15228b5866ac1
W20250901 01:52:29.022897 1485 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 186ec810d02e4748aad15228b5866ac1: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:29.022966 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8580902bdb2040c887322ed34fb4376f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.023502 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_adfdea1f8d3e4aebb8965391d7282020"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.023605 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a555f999d4ed431ea7cabd4e75769dd7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.025219 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2e9c35cf3b6c4b5badb0224aeee2c88a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.026041 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_51182fd3efa44c2e8982079857173780"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.026302 1033 tablet_service.cc:1515] Processing DeleteTablet for tablet 186ec810d02e4748aad15228b5866ac1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:28 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:29.026738 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3169efecb226407eb3cdbfad34b91e90"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.027365 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 186ec810d02e4748aad15228b5866ac1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:28 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:29.027586 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d3d099d09df14d7592af5928f3e0d15a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.028201 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d7b26c5211c44a8bb61a88aab63e077c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.028414 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_bed309546d75408d8f69c080e5dd8f81"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:29.010432 992 leader_election.cc:336] T c9a4c80bec1d41458974a0c449aecaf7 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:29.029383 992 leader_election.cc:304] T c9a4c80bec1d41458974a0c449aecaf7 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:29.029520 1490 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 7317eb7848ee44698038a4b404d4bf21
I20250901 01:52:29.029697 1993 raft_consensus.cc:2747] T c9a4c80bec1d41458974a0c449aecaf7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:29.030702 859 leader_election.cc:343] T 1495de70f0844f82a0eea99c059f076b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:29.030824 859 leader_election.cc:304] T 1495de70f0844f82a0eea99c059f076b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:29.031045 2323 raft_consensus.cc:2747] T 1495de70f0844f82a0eea99c059f076b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:29.031214 859 leader_election.cc:343] T fecf7d538b4c4c619182d5eccb44dd8b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:29.031303 859 leader_election.cc:304] T fecf7d538b4c4c619182d5eccb44dd8b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:29.031543 2323 raft_consensus.cc:2747] T fecf7d538b4c4c619182d5eccb44dd8b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:29.031795 2323 consensus_queue.cc:1035] T e87d1281f35648b0b115b67c95e244e7 P 00c282f779c1477d920f24dc0dc16628 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:29.031955 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 100) took 3933 ms. Trace:
I20250901 01:52:29.032047 2113 rpcz_store.cc:276] 0901 01:52:25.098450 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.098581 (+ 131us) service_pool.cc:225] Handling call
0901 01:52:28.908105 (+3809524us) ts_tablet_manager.cc:1145] Deleting tablet b73e362321284df1b18ef755dfe930a5
0901 01:52:28.908117 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:28.932506 (+ 24389us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:28.935111 (+ 2605us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:29.031933 (+ 96822us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3809295}
I20250901 01:52:29.032559 2113 tablet_replica.cc:331] T c763d252598745a088344aa0eb5fa3d3 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:29.032791 2113 raft_consensus.cc:2241] T c763d252598745a088344aa0eb5fa3d3 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:29.033171 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet b73e362321284df1b18ef755dfe930a5 (table default.Table_81b532507ef2460d9ff52022c060c07a [id=8ef63407ade74b5688d236eed2d1561c]) successfully deleted
I20250901 01:52:29.034585 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_32fc2267665446c083133da3002fcb0c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.037197 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ce7a56653d864963a845f4bc38eaaf6e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.038322 2118 raft_consensus.cc:2241] T c58edaa76ebf47dfb1d9d5fab6de7dfa P 00c282f779c1477d920f24dc0dc16628 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:29.038586 2118 pending_rounds.cc:70] T c58edaa76ebf47dfb1d9d5fab6de7dfa P 00c282f779c1477d920f24dc0dc16628: Trying to abort 1 pending ops.
I20250901 01:52:29.038641 2118 pending_rounds.cc:77] T c58edaa76ebf47dfb1d9d5fab6de7dfa P 00c282f779c1477d920f24dc0dc16628: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7195408579681144832 op_type: NO_OP noop_request { }
I20250901 01:52:29.038758 2118 raft_consensus.cc:2887] T c58edaa76ebf47dfb1d9d5fab6de7dfa P 00c282f779c1477d920f24dc0dc16628 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250901 01:52:29.038848 2118 raft_consensus.cc:2270] T c58edaa76ebf47dfb1d9d5fab6de7dfa P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:29.039469 2118 ts_tablet_manager.cc:1905] T c58edaa76ebf47dfb1d9d5fab6de7dfa P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:29.044744 2412 consensus_queue.cc:1035] T 89d40f0331114490b51de3ba6d773571 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:29.045871 1150 tablet_bootstrap.cc:492] T 5f5396045979450193d461ba1334a5c9 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:29.046005 1150 ts_tablet_manager.cc:1397] T 5f5396045979450193d461ba1334a5c9 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.058s user 0.004s sys 0.001s
I20250901 01:52:29.046664 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "617cc33d0d23439cb5f1e0c31bf11281" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:29.046725 1150 raft_consensus.cc:357] T 5f5396045979450193d461ba1334a5c9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:29.046864 1150 raft_consensus.cc:383] T 5f5396045979450193d461ba1334a5c9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:29.046914 1150 raft_consensus.cc:738] T 5f5396045979450193d461ba1334a5c9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:29.046983 2113 raft_consensus.cc:2270] T c763d252598745a088344aa0eb5fa3d3 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:29.047101 1150 consensus_queue.cc:260] T 5f5396045979450193d461ba1334a5c9 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:29.047611 1150 ts_tablet_manager.cc:1428] T 5f5396045979450193d461ba1334a5c9 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250901 01:52:29.047896 1150 tablet_bootstrap.cc:492] T f98d1d8d660443cb8066a547e5d9b77f P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:29.047624 2113 ts_tablet_manager.cc:1905] T c763d252598745a088344aa0eb5fa3d3 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:29.051326 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e0ab4e8ae7854d3584be120fb3afa625"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.052471 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_439203cf9bf647769f0e5f9a6f52af8d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.052425 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3313e6fec47644219b0dc4a28bd2b08f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.053200 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4e669d3dac0f43b4a343ad7dcca9a7a7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.053171 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f4769e3cfc85418f90d5ad515a73d96e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.055981 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_528d1a29dcd14c7dbeb0907c32afccd1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.056638 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_0f81d32a5b894963a875858828bbb4de"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.057219 1490 master_service.cc:948] ReplaceTablet: received request to replace tablet 5f52698fb56d472d9e91fb10cb10c69a from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:29.053371 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e9c66a5d5bba44e78a598170918f81a7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.057888 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1f8056d11c0247bd802eeeb2b225fc88"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.058490 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5611d52a2b8749b98fd63aeb5711d5e4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.059090 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2eebaed5dc3f4022aeb02788e330fee6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.060143 1150 tablet_bootstrap.cc:654] T f98d1d8d660443cb8066a547e5d9b77f P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:29.061307 858 consensus_peers.cc:489] T 3d90c6ef06d1436d8e2bb1333e7d9985 P 00c282f779c1477d920f24dc0dc16628 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:29.062045 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c7dda8c6cebe452aa325c8d9294ed6bb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.062772 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_cfb9bf251c8d464eac2014eb0f06c5cd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.063479 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5253a7882a044edcbd2261a58b7049a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:29.063773 859 leader_election.cc:343] T 617cc33d0d23439cb5f1e0c31bf11281 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:29.063896 859 leader_election.cc:304] T 617cc33d0d23439cb5f1e0c31bf11281 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:29.063979 992 consensus_peers.cc:489] T ea846f88b3534beebd0001b7f17b5ca9 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:29.064116 2323 raft_consensus.cc:2747] T 617cc33d0d23439cb5f1e0c31bf11281 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:29.066212 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_44d2ec0b490a4d1a8d7d5c5a6f4e8b0d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.066874 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_64a39f63878646c28abe53db57a625a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.067513 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_89f59add66c04c668d0d6db4a962196d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.053548 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4ec4bf82118143b6b5769dee37a0294f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.069732 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2bca516097244a199ce7623a7e063574"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.071051 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_81793075ec7b46bd96a0bf05cff9c025"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.072739 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_62478c26e4f7483f81ebb424a68c4959"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.072827 2323 raft_consensus.cc:491] T 4b4aa063ffe948e69e47f03102581899 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:29.072966 2323 raft_consensus.cc:513] T 4b4aa063ffe948e69e47f03102581899 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:29.073601 2323 leader_election.cc:290] T 4b4aa063ffe948e69e47f03102581899 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:29.073701 2412 raft_consensus.cc:491] T afc002ae2486476b9208e1cf8215157c P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:29.073869 2412 raft_consensus.cc:513] T afc002ae2486476b9208e1cf8215157c P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:29.074405 2412 leader_election.cc:290] T afc002ae2486476b9208e1cf8215157c P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:29.074774 992 consensus_peers.cc:489] T 670d90a37b934579b353bb4e4820127b P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250901 01:52:29.053714 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_37519275572a46328a9884acf90cac8c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:29.075565 858 leader_election.cc:336] T 4b4aa063ffe948e69e47f03102581899 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:29.075857 2412 raft_consensus.cc:491] T a59ac57aebe24860aa02f53ce0485910 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:29.075999 2412 raft_consensus.cc:513] T a59ac57aebe24860aa02f53ce0485910 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:29.076117 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8e2ecae266044f5e893f6d2b75016360"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.077090 2412 leader_election.cc:290] T a59ac57aebe24860aa02f53ce0485910 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:29.077986 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2d42d51607ec43888c14de52351a99e9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.078366 1151 tablet_bootstrap.cc:492] T ad1a6be9fcc045d3bb4e71a2f6bd5ccd P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:29.078514 1151 ts_tablet_manager.cc:1397] T ad1a6be9fcc045d3bb4e71a2f6bd5ccd P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.064s user 0.002s sys 0.004s
I20250901 01:52:29.079362 1151 raft_consensus.cc:357] T ad1a6be9fcc045d3bb4e71a2f6bd5ccd P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:29.079563 1151 raft_consensus.cc:383] T ad1a6be9fcc045d3bb4e71a2f6bd5ccd P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:29.079627 1151 raft_consensus.cc:738] T ad1a6be9fcc045d3bb4e71a2f6bd5ccd P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:29.076864 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c4bd0d6957944ac1a82669abae81f992"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.079852 1151 consensus_queue.cc:260] T ad1a6be9fcc045d3bb4e71a2f6bd5ccd P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:29.080509 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e913070af4484dde9f8944d135c2760b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.080667 1151 ts_tablet_manager.cc:1428] T ad1a6be9fcc045d3bb4e71a2f6bd5ccd P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250901 01:52:29.080885 2118 ts_tablet_manager.cc:1918] T c58edaa76ebf47dfb1d9d5fab6de7dfa P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:29.081208 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_648e00d4d3814b99a5e6c0beb32e14a5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.081640 2118 log.cc:1199] T c58edaa76ebf47dfb1d9d5fab6de7dfa P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/c58edaa76ebf47dfb1d9d5fab6de7dfa
I20250901 01:52:29.083525 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_af49e94f00d04db6b915c0ff054dcb21"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.083999 969 heartbeater.cc:499] Master 127.0.104.60:44231 was elected leader, sending a full tablet report...
I20250901 01:52:29.084136 2118 ts_tablet_manager.cc:1939] T c58edaa76ebf47dfb1d9d5fab6de7dfa P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:29.086262 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_20b8040c6fd44025972979e5bd994c58"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.087795 1151 tablet_bootstrap.cc:492] T d10c193452954c198bba686169b8025b P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:29.087435 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 93) took 3734 ms. Trace:
I20250901 01:52:29.088090 2118 rpcz_store.cc:276] 0901 01:52:25.352557 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.352618 (+ 61us) service_pool.cc:225] Handling call
0901 01:52:29.004728 (+3652110us) ts_tablet_manager.cc:1145] Deleting tablet c58edaa76ebf47dfb1d9d5fab6de7dfa
0901 01:52:29.004741 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:29.042590 (+ 37849us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:29.080197 (+ 37607us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:29.087381 (+ 7184us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":32594,"tablet-delete.queue_time_us":3651700}
I20250901 01:52:29.089960 1151 tablet_bootstrap.cc:654] T d10c193452954c198bba686169b8025b P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:29.090344 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet c58edaa76ebf47dfb1d9d5fab6de7dfa (table default.Table_3df3e67f179f46a8be496e3bfe3cb89f [id=3fa0578ede344c0a8ff5d00b3703ad74]) successfully deleted
I20250901 01:52:29.091979 2118 tablet_replica.cc:331] T e249474dcbf642189560c91ec3a99fdf P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:29.093914 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_aed1156f451e4261a6fecf0f70c35c52"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:29.094805 992 leader_election.cc:343] T afc002ae2486476b9208e1cf8215157c P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: afc002ae2486476b9208e1cf8215157c
I20250901 01:52:29.094990 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_0c189284a02d4689b357d97763d2e5ee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.095615 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4b4aa063ffe948e69e47f03102581899" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:29.094626 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7a62ff98800c449ab8f61aeae37e2c24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.094833 2118 raft_consensus.cc:2241] T e249474dcbf642189560c91ec3a99fdf P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:29.096297 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_aaea09a009974fc295defce005b03ee2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.097460 2118 raft_consensus.cc:2270] T e249474dcbf642189560c91ec3a99fdf P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:29.098057 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_bf8cb520670a417c93e9ee252b2a1857"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.098513 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f1c25b0a92ce41abac08d0af353ed00f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.101452 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_bbadd930135c4eb798dd1f309d9d524d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.100775 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7ac5e948265c4939ab765cd89558adec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.101900 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f8bf2b14d94c49fb98dda8be8f3ead18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.102550 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d262d55c20c64464b3de2f9ddb65d8a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:29.106882 859 leader_election.cc:343] T 4b4aa063ffe948e69e47f03102581899 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:29.107016 859 leader_election.cc:304] T 4b4aa063ffe948e69e47f03102581899 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:29.107434 2408 raft_consensus.cc:2747] T 4b4aa063ffe948e69e47f03102581899 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:29.112753 2118 ts_tablet_manager.cc:1905] T e249474dcbf642189560c91ec3a99fdf P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:29.123211 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_75a7fe43a15e42e3842b833187d29bc9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.123559 1150 tablet_bootstrap.cc:492] T f98d1d8d660443cb8066a547e5d9b77f P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:29.123703 1150 ts_tablet_manager.cc:1397] T f98d1d8d660443cb8066a547e5d9b77f P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.076s user 0.005s sys 0.005s
I20250901 01:52:29.124259 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f468b0f5417646799a880bdef828f80a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.125171 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a1ad1326557343739f2ceb91e46d54a4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.125360 1150 raft_consensus.cc:357] T f98d1d8d660443cb8066a547e5d9b77f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:29.125548 1150 raft_consensus.cc:383] T f98d1d8d660443cb8066a547e5d9b77f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:29.125604 1150 raft_consensus.cc:738] T f98d1d8d660443cb8066a547e5d9b77f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:29.125810 1150 consensus_queue.cc:260] T f98d1d8d660443cb8066a547e5d9b77f P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:29.126314 1150 ts_tablet_manager.cc:1428] T f98d1d8d660443cb8066a547e5d9b77f P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
W20250901 01:52:29.126674 992 leader_election.cc:343] T a59ac57aebe24860aa02f53ce0485910 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: a59ac57aebe24860aa02f53ce0485910
I20250901 01:52:29.131227 1151 tablet_bootstrap.cc:492] T d10c193452954c198bba686169b8025b P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:29.131409 1151 ts_tablet_manager.cc:1397] T d10c193452954c198bba686169b8025b P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.044s user 0.001s sys 0.002s
I20250901 01:52:29.131660 2113 ts_tablet_manager.cc:1918] T c763d252598745a088344aa0eb5fa3d3 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:29.135736 1150 tablet_bootstrap.cc:492] T 93bcac1fd95d44dc927abfca81692921 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:29.138671 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a1eb7798195543eaad1bcf9bdbfad3f2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.139171 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f4cdcc2f55af47009c9e8d52aedf2cd6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.139865 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b848815b41874a1c99c26df61cdfc1ac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.140689 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_08ff932c4d4a47c19b8f17a980290822"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.141394 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_028c3cb202fc4c7c87bf12144c6b8366"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.142220 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_53531751ee8d402cb1112d86fa6c2dac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.142818 2113 log.cc:1199] T c763d252598745a088344aa0eb5fa3d3 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/c763d252598745a088344aa0eb5fa3d3
I20250901 01:52:29.143208 2113 ts_tablet_manager.cc:1939] T c763d252598745a088344aa0eb5fa3d3 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
W20250901 01:52:29.133335 992 leader_election.cc:336] T a59ac57aebe24860aa02f53ce0485910 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:29.143479 992 leader_election.cc:304] T a59ac57aebe24860aa02f53ce0485910 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:29.143711 2412 raft_consensus.cc:2747] T a59ac57aebe24860aa02f53ce0485910 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:29.143993 992 leader_election.cc:336] T afc002ae2486476b9208e1cf8215157c P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:29.144083 992 leader_election.cc:304] T afc002ae2486476b9208e1cf8215157c P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:29.144268 2412 raft_consensus.cc:2747] T afc002ae2486476b9208e1cf8215157c P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:29.146144 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5693afb69a894015a3894df6abfbb189"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.147004 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_29a8b955cc5241128a69d6c9cc893c15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.141175 1151 raft_consensus.cc:357] T d10c193452954c198bba686169b8025b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:29.149524 1151 raft_consensus.cc:383] T d10c193452954c198bba686169b8025b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:29.149597 1151 raft_consensus.cc:738] T d10c193452954c198bba686169b8025b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:29.149787 1151 consensus_queue.cc:260] T d10c193452954c198bba686169b8025b P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:29.150347 1484 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 3a8231590ea643b0a1f83eec4d9a8191
W20250901 01:52:29.150537 1484 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 3a8231590ea643b0a1f83eec4d9a8191: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:29.150076 2357 raft_consensus.cc:491] T 4cde5f4f02044b7bbd7d43b49aac4ae8 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:29.152876 2357 raft_consensus.cc:513] T 4cde5f4f02044b7bbd7d43b49aac4ae8 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:29.153548 2357 leader_election.cc:290] T 4cde5f4f02044b7bbd7d43b49aac4ae8 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:29.157691 1150 tablet_bootstrap.cc:654] T 93bcac1fd95d44dc927abfca81692921 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:29.159644 1492 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 7f46e9ddadaf4ff986d8c991178daa0c (table default.Table_9978497447394e5694ae928412634ecb [id=ea928501fe954cd0b81d38add4ef7128], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet 7f46e9ddadaf4ff986d8c991178daa0c consensus state has no leader
I20250901 01:52:29.162892 1033 tablet_service.cc:1515] Processing DeleteTablet for tablet 3a8231590ea643b0a1f83eec4d9a8191 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:28 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:29.163189 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 438cd35ff3f74c49b908d072560077db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:29.163337 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 49d30da95b14400b845270122487a2a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:29.163468 1034 tablet_service.cc:1515] Processing DeleteTablet for tablet a774cc6ae3924d299127fd9ffb01ca57 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:16 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:29.163635 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet a7ff996482164776b468cf305989866b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:13 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:29.163769 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet a54af99bc10045e8b0b6868e5aed98b4 with delete_type TABLET_DATA_DELETED (replaced by tablet 50472a7aa2f04e0fbd65ac6c86c34b3e) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:29.163926 1036 tablet_service.cc:1515] Processing DeleteTablet for tablet c95798f0daf4437db18a291c4464522e with delete_type TABLET_DATA_DELETED (replaced by tablet 7a2f6e2c7ee2402ea310c058e046416c) from {username='slave'} at 127.0.0.1:35946
W20250901 01:52:29.164290 858 leader_election.cc:336] T 4cde5f4f02044b7bbd7d43b49aac4ae8 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:29.164500 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 3a8231590ea643b0a1f83eec4d9a8191 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:28 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:29.169394 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e8f51a9cd0754c0c8bca651f8b044f49"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.170136 2118 ts_tablet_manager.cc:1918] T e249474dcbf642189560c91ec3a99fdf P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:29.170257 2118 log.cc:1199] T e249474dcbf642189560c91ec3a99fdf P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/e249474dcbf642189560c91ec3a99fdf
I20250901 01:52:29.170284 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_63741cee7c114c14b863f7e9cf76a992"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.170696 2118 ts_tablet_manager.cc:1939] T e249474dcbf642189560c91ec3a99fdf P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:29.170967 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1f2a8450900a4021a5f60ef68278c491"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.173972 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7c1ab77f20774ca1aa27ed52d5aca8ef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.174666 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_814b336b9cf84f838d1cb022341229b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.175275 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_224118aa430841358aba5f18a615a744"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.175872 1485 master_service.cc:948] ReplaceTablet: received request to replace tablet 4fd0e4f9d86c4362af96ce9964b72328 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:29.179381 1488 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet e6c10426f60f43e8bde69d9b1b9c8eb6
W20250901 01:52:29.179574 1488 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet e6c10426f60f43e8bde69d9b1b9c8eb6: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:29.179718 1150 tablet_bootstrap.cc:492] T 93bcac1fd95d44dc927abfca81692921 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:29.179841 1150 ts_tablet_manager.cc:1397] T 93bcac1fd95d44dc927abfca81692921 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.053s user 0.003s sys 0.001s
I20250901 01:52:29.179870 1488 catalog_manager.cc:6652] ReplaceTablet: tablet e6c10426f60f43e8bde69d9b1b9c8eb6 deleted and replaced by tablet 494b6488aa0d4355b4caa46004267465
I20250901 01:52:29.180014 1490 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 5f52698fb56d472d9e91fb10cb10c69a
W20250901 01:52:29.180193 1490 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 5f52698fb56d472d9e91fb10cb10c69a: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:29.180238 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_07e86182d45d4663b37fb04f672714a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.180585 1490 catalog_manager.cc:6652] ReplaceTablet: tablet 5f52698fb56d472d9e91fb10cb10c69a deleted and replaced by tablet 8e10a01e77d14cf28c2c5e0205e595b9
I20250901 01:52:29.180637 1150 raft_consensus.cc:357] T 93bcac1fd95d44dc927abfca81692921 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:29.180790 1150 raft_consensus.cc:383] T 93bcac1fd95d44dc927abfca81692921 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:29.180847 1150 raft_consensus.cc:738] T 93bcac1fd95d44dc927abfca81692921 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:29.180944 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e53e29f873e34ec98096448050646852"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.181027 1150 consensus_queue.cc:260] T 93bcac1fd95d44dc927abfca81692921 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:29.181141 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_205db1ac16c34c1a8d061afeb7542db2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.181247 2323 raft_consensus.cc:491] T f98d1d8d660443cb8066a547e5d9b77f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:29.181349 2323 raft_consensus.cc:513] T f98d1d8d660443cb8066a547e5d9b77f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:29.181649 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_636ca788a1a14486adf37400aa672a12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.181936 2323 leader_election.cc:290] T f98d1d8d660443cb8066a547e5d9b77f P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:29.182337 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_64deb14ddb9545dc9fae91e8b2879857"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.182986 901 tablet_service.cc:1515] Processing DeleteTablet for tablet 5f52698fb56d472d9e91fb10cb10c69a with delete_type TABLET_DATA_DELETED (replaced by tablet 8e10a01e77d14cf28c2c5e0205e595b9) from {username='slave'} at 127.0.0.1:34472
W20250901 01:52:29.183980 858 leader_election.cc:336] T f98d1d8d660443cb8066a547e5d9b77f P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:29.184026 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_72545cce79bd4ead89bbc913b9c3a3b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:29.184170 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae4844183a3c4a04a775e6fd878391a3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_aee74198b51d4f089538c4f2bf3ce4dc'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:29.184175 1150 ts_tablet_manager.cc:1428] T 93bcac1fd95d44dc927abfca81692921 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
E20250901 01:52:29.184303 1696 catalog_manager.cc:6205] Error selecting replicas for tablet be8dfbc466564842a2fdb4953eee939b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_75ba3ba70ca24a94920e7bce96190ff2'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:29.184679 887 tablet_service.cc:1515] Processing DeleteTablet for tablet e6c10426f60f43e8bde69d9b1b9c8eb6 with delete_type TABLET_DATA_DELETED (replaced by tablet 494b6488aa0d4355b4caa46004267465) from {username='slave'} at 127.0.0.1:34472
E20250901 01:52:29.186098 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9745582cebfe47b58c21d0191ab4cb9b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77728af9eee44f29a1e2131d5b75b7ca'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:29.186780 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 4abf9725be5d42f3893f07f0a9c7614c, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_8445fcd774874cadaf4acc3581eee417'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:29.186883 1696 catalog_manager.cc:6205] Error selecting replicas for tablet e2ed32b3292141278af28026528e96c2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_50ae3908a7944f70949c54be8fbf1564'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:29.187000 1696 catalog_manager.cc:6205] Error selecting replicas for tablet dcc60bb000f440f1aca9133928b5ce20, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2fbca8463d6546c9b8f5b018fc60251d'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:29.187105 1696 catalog_manager.cc:6205] Error selecting replicas for tablet a2ca69fabadd4ad69e1670cae7a0c503, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_32c4716e3aa24a87a6b1472984b255c4'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:29.187206 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f8f923ddcd2740bcb7e521f4f57c4034, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_5e65c5f068b84995a2d7a76fa65dda0f'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:29.187306 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 0813cc68d4b0488ba60a523e4479971b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d1302673d68f424ebb54b220911ce385'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:29.187299 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7b94f8d43c6647109908290529d1f2cf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:29.187466 1696 catalog_manager.cc:6205] Error selecting replicas for tablet b1ba0e19dc0b462e80c537bf2d183f5b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0e2cd82b00f1466eaef7f287402e644d'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:29.187551 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5f0e054f62de4cf6bc6f69ad6664b192, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ef6832d0845044ed9f165ca3541b4c3f'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:29.187651 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 787be682c0e44248a80d7037fb61dc6c, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7c12d66048a6470e8c227dc1d9f45fa4'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:29.187762 1696 catalog_manager.cc:6205] Error selecting replicas for tablet cc3b8736b9094d7e892c80da4d5b20f6, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_93afd37289414655980b22934695bd55'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:29.187873 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae9a4ad4fdfb4771ba23a91f7007447e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_5e84ae9a9c2a4084a1f9c5ffb4594de2'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:29.187971 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9074543dea9949e7a085717b40913348, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_53e30205c5fd488eb3b48c685a159d04'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:29.188076 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 18498c9cb09141d08c1c5c0854b63ebd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77f4810bc83c4d948531da2a45a3792e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:29.188179 1696 catalog_manager.cc:6205] Error selecting replicas for tablet a6b539e160194da3a849d9e2f1ec9859, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_03fb48eaab9b42858ab7cefb722e43a8'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:29.188156 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_dbbc78f325b1486e98b5910bc10645a5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:29.188275 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 6ce9521b925b4f97bb4a810ce77e147e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_9e04f976a93f4d30baccf90fa3622dfe'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:29.188356 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2f61edb50d7a42198bfd70639c7bf4de, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_a5c783cbdb694af69047b31d99ff8def'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:29.189946 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_26a9c7d19bc9445fbc70b7fc74a969a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.190601 1490 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_139f6d174d794a6e9a618e90587d0c0d" } new_table_name: "default.Table_ce2f3b515c8f4c1bb557c5ed84b48dba" modify_external_catalogs: true
I20250901 01:52:29.190976 1491 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_122585be3bdc45fb9498d02564fb8567" } modify_external_catalogs: true
I20250901 01:52:29.191145 1491 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_122585be3bdc45fb9498d02564fb8567" } modify_external_catalogs: true
I20250901 01:52:29.192791 1483 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_0451d0afa26f4d5f8d4214a946eea04f" } new_table_name: "default.Table_0f4b4094d2974b7591875375eda6bed0" modify_external_catalogs: true
I20250901 01:52:29.193660 1482 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: default.Table_0451d0afa26f4d5f8d4214a946eea04f [id=5ee13d46e06a4fa59b1070547573e523] alter complete (version 1)
I20250901 01:52:29.195565 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7ef8c647ee574071b37bd2194a3712b2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.196346 1484 master_service.cc:948] ReplaceTablet: received request to replace tablet 606fac5c6efe48d1877104b01c23ba5b from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:29.197356 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b905ad7f42ae4332a44c820e740f8135"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.198308 1151 ts_tablet_manager.cc:1428] T d10c193452954c198bba686169b8025b P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.067s user 0.003s sys 0.000s
I20250901 01:52:29.199100 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1defe6f720fc44a3aa8cc4495366ca6e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:29.200093 992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:40561, user_credentials={real_user=slave}} blocked reactor thread for 50081.5us
I20250901 01:52:29.202921 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_81054658920244cd977bc16a17ee7b47"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.209751 970 heartbeater.cc:344] Connected to a master server at 127.0.104.62:43871
I20250901 01:52:29.209797 1053 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f98d1d8d660443cb8066a547e5d9b77f" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:29.212324 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4cde5f4f02044b7bbd7d43b49aac4ae8" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:29.213565 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 102) took 4113 ms. Trace:
I20250901 01:52:29.213734 2113 rpcz_store.cc:276] 0901 01:52:25.100245 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.100386 (+ 141us) service_pool.cc:225] Handling call
0901 01:52:29.032511 (+3932125us) ts_tablet_manager.cc:1145] Deleting tablet c763d252598745a088344aa0eb5fa3d3
0901 01:52:29.032522 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:29.051473 (+ 18951us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:29.131547 (+ 80074us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:29.213545 (+ 81998us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13819,"tablet-delete.queue_time_us":3931905}
I20250901 01:52:29.214275 2113 tablet_replica.cc:331] T 579b9cdf819f4609a532e96bec093fd5 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:29.214366 1151 tablet_bootstrap.cc:492] T ed1b41340ce742a88c3e4fd85cd25ca6 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:29.214560 2113 raft_consensus.cc:2241] T 579b9cdf819f4609a532e96bec093fd5 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:29.214884 2113 raft_consensus.cc:2270] T 579b9cdf819f4609a532e96bec093fd5 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:29.215412 2113 ts_tablet_manager.cc:1905] T 579b9cdf819f4609a532e96bec093fd5 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:29.215786 2357 raft_consensus.cc:491] T 14b466b2a5a249aa841f1b8b06301edc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:29.215915 2357 raft_consensus.cc:513] T 14b466b2a5a249aa841f1b8b06301edc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:29.216030 1151 tablet_bootstrap.cc:654] T ed1b41340ce742a88c3e4fd85cd25ca6 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:29.216714 2357 leader_election.cc:290] T 14b466b2a5a249aa841f1b8b06301edc P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:29.217283 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_80d1e6a5926d4b2b9d2105c9223753cb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:29.218014 858 leader_election.cc:336] T 14b466b2a5a249aa841f1b8b06301edc P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:29.218137 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_876b559bd7964cd39804ff22619b5676"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.218962 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1d88c6aebb174170b73a068bfab2520f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.219723 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_6ceda9e4c3d6439fb397d096d783233a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.220716 1488 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_a07fd8be1b7540a280aca916e23b221e" } modify_external_catalogs: true
I20250901 01:52:29.222879 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4d07e0af95e24408ae0decdad0caaa35"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:29.227516 1492 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.3:51295 (request call id 182) took 1977 ms (client timeout 1000 ms). Trace:
I20250901 01:52:29.233206 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 94) took 3880 ms. Trace:
I20250901 01:52:29.233274 2357 raft_consensus.cc:491] T 93bcac1fd95d44dc927abfca81692921 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:29.233399 2357 raft_consensus.cc:513] T 93bcac1fd95d44dc927abfca81692921 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:29.233920 1150 tablet_bootstrap.cc:492] T d0b7c84ec9394a20ad77add3808e555a P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:29.234623 2357 leader_election.cc:290] T 93bcac1fd95d44dc927abfca81692921 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:29.236231 1150 tablet_bootstrap.cc:654] T d0b7c84ec9394a20ad77add3808e555a P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:29.233310 2118 rpcz_store.cc:276] 0901 01:52:25.352759 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.352889 (+ 130us) service_pool.cc:225] Handling call
0901 01:52:29.091193 (+3738304us) ts_tablet_manager.cc:1145] Deleting tablet e249474dcbf642189560c91ec3a99fdf
0901 01:52:29.091561 (+ 368us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:29.137489 (+ 45928us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:29.162078 (+ 24589us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:29.233182 (+ 71104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3738104}
I20250901 01:52:29.237833 2118 tablet_replica.cc:331] T ad1a6be9fcc045d3bb4e71a2f6bd5ccd P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:29.238034 2118 raft_consensus.cc:2241] T ad1a6be9fcc045d3bb4e71a2f6bd5ccd P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:29.238178 2118 raft_consensus.cc:2270] T ad1a6be9fcc045d3bb4e71a2f6bd5ccd P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:29.223999 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_312688cf99cc44f198be8d01690d0793"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:29.238602 2221 log.cc:927] Time spent T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Append to log took a long time: real 0.126s user 0.004s sys 0.000s
I20250901 01:52:29.238664 2118 ts_tablet_manager.cc:1905] T ad1a6be9fcc045d3bb4e71a2f6bd5ccd P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:29.241230 858 leader_election.cc:336] T 93bcac1fd95d44dc927abfca81692921 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:29.243346 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ee193c808375400dab3798481599f1b2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.244308 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5cb33e527e644da69c5192923f889333"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.246500 2118 ts_tablet_manager.cc:1918] T ad1a6be9fcc045d3bb4e71a2f6bd5ccd P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:29.248391 2118 log.cc:1199] T ad1a6be9fcc045d3bb4e71a2f6bd5ccd P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/ad1a6be9fcc045d3bb4e71a2f6bd5ccd
I20250901 01:52:29.248998 2118 ts_tablet_manager.cc:1939] T ad1a6be9fcc045d3bb4e71a2f6bd5ccd P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:29.249258 1489 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_a89cc8e48563460baabc34c821ceae35" } new_table_name: "default.Table_2b946a63bd574ac0971fa3d3b9c9a6a4" modify_external_catalogs: true
I20250901 01:52:29.250315 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 95) took 3897 ms. Trace:
I20250901 01:52:29.253530 2118 rpcz_store.cc:276] 0901 01:52:25.352889 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.352959 (+ 70us) service_pool.cc:225] Handling call
0901 01:52:29.237777 (+3884818us) ts_tablet_manager.cc:1145] Deleting tablet ad1a6be9fcc045d3bb4e71a2f6bd5ccd
0901 01:52:29.237790 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:29.243171 (+ 5381us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:29.246399 (+ 3228us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:29.250292 (+ 3893us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3884654}
I20250901 01:52:29.254952 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet ad1a6be9fcc045d3bb4e71a2f6bd5ccd (table default.Table_5e84ae9a9c2a4084a1f9c5ffb4594de2 [id=9bebb2e486ab48888e6b523281ac1e88]) successfully deleted
W20250901 01:52:29.255067 1490 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ed1b41340ce742a88c3e4fd85cd25ca6 consensus state has no leader
I20250901 01:52:29.255224 1491 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 3a2bc5ad1cd243daa3c92fceca8ba414
I20250901 01:52:29.255053 2118 tablet_replica.cc:331] T 6cf09e8f299548dba2cce16d7b565bad P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:29.255465 1490 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_50ae3908a7944f70949c54be8fbf1564" } new_table_name: "default.Table_b5077f55951d48be92a67407ffe26a08" modify_external_catalogs: true
W20250901 01:52:29.255559 1491 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 3a2bc5ad1cd243daa3c92fceca8ba414: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:29.256295 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 3a2bc5ad1cd243daa3c92fceca8ba414 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:29 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:29.256624 2118 raft_consensus.cc:2241] T 6cf09e8f299548dba2cce16d7b565bad P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:29.257287 1491 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_c931e9dfbef04fd3b49aa4f43db3d981" } new_table_name: "default.Table_d53a456dbca64daf92aeb5fb4517b417" modify_external_catalogs: true
I20250901 01:52:29.257506 1483 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_97622024147a400ea7db77c74af5f71e" } modify_external_catalogs: true
I20250901 01:52:29.257715 1483 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_97622024147a400ea7db77c74af5f71e" } modify_external_catalogs: true
I20250901 01:52:29.258546 1488 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_a07fd8be1b7540a280aca916e23b221e" } modify_external_catalogs: true
I20250901 01:52:29.257791 2118 raft_consensus.cc:2270] T 6cf09e8f299548dba2cce16d7b565bad P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:29.259454 2118 ts_tablet_manager.cc:1905] T 6cf09e8f299548dba2cce16d7b565bad P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:29.259441 1485 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 4fd0e4f9d86c4362af96ce9964b72328
W20250901 01:52:29.259820 1485 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 4fd0e4f9d86c4362af96ce9964b72328: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:29.260591 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 4fd0e4f9d86c4362af96ce9964b72328 with delete_type TABLET_DATA_DELETED (replaced by tablet a656adbf1b29431687e0989f0db3459c) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:29.260916 1485 catalog_manager.cc:6652] ReplaceTablet: tablet 4fd0e4f9d86c4362af96ce9964b72328 deleted and replaced by tablet a656adbf1b29431687e0989f0db3459c
I20250901 01:52:29.261603 887 tablet_service.cc:1515] Processing DeleteTablet for tablet ad1a6be9fcc045d3bb4e71a2f6bd5ccd with delete_type TABLET_DATA_DELETED (replaced by tablet ae9a4ad4fdfb4771ba23a91f7007447e) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:29.262730 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 121803e79ab64ec0b948b448646b9d0a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:14 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:29.263343 885 tablet_service.cc:1515] Processing DeleteTablet for tablet b7b802f976f3471faaaecc964ec87e48 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:16 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:29.263558 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 2240302d9178445d8700e35f5ea274f7 with delete_type TABLET_DATA_DELETED (replaced by tablet b13afacd37474781877b9f2cd20b313a) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:29.263597 901 tablet_service.cc:1515] Processing DeleteTablet for tablet 0cb3262973934c8689b851adad60afe0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:16 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:29.263724 898 tablet_service.cc:1515] Processing DeleteTablet for tablet af2c4bcbb00741f789edee109252d8cb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:18 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:29.263764 887 tablet_service.cc:1515] Processing DeleteTablet for tablet b6daeff0070a4db08e90020752d21c29 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:14 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:29.264328 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 7f64ebd080944190be2e64d02d70a8a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:15 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:29.264846 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 25ce0fe8c9544f828694184d19d13df9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:16 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:29.265051 898 tablet_service.cc:1515] Processing DeleteTablet for tablet c58edaa76ebf47dfb1d9d5fab6de7dfa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:17 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:29.265118 887 tablet_service.cc:1515] Processing DeleteTablet for tablet eaa521c52a2f4408b59770bd2a05907c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:29.265836 885 tablet_service.cc:1515] Processing DeleteTablet for tablet e249474dcbf642189560c91ec3a99fdf with delete_type TABLET_DATA_DELETED (replaced by tablet 7f46e9ddadaf4ff986d8c991178daa0c) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:29.270375 1151 tablet_bootstrap.cc:492] T ed1b41340ce742a88c3e4fd85cd25ca6 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:29.270529 1151 ts_tablet_manager.cc:1397] T ed1b41340ce742a88c3e4fd85cd25ca6 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.056s user 0.007s sys 0.002s
I20250901 01:52:29.271276 1151 raft_consensus.cc:357] T ed1b41340ce742a88c3e4fd85cd25ca6 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:29.271442 1151 raft_consensus.cc:383] T ed1b41340ce742a88c3e4fd85cd25ca6 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:29.271499 1151 raft_consensus.cc:738] T ed1b41340ce742a88c3e4fd85cd25ca6 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:29.271711 1151 consensus_queue.cc:260] T ed1b41340ce742a88c3e4fd85cd25ca6 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:29.272233 1151 ts_tablet_manager.cc:1428] T ed1b41340ce742a88c3e4fd85cd25ca6 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250901 01:52:29.272864 1058 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "93bcac1fd95d44dc927abfca81692921" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:29.272730 1150 tablet_bootstrap.cc:492] T d0b7c84ec9394a20ad77add3808e555a P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:29.273397 1150 ts_tablet_manager.cc:1397] T d0b7c84ec9394a20ad77add3808e555a P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.040s user 0.009s sys 0.000s
I20250901 01:52:29.274222 1150 raft_consensus.cc:357] T d0b7c84ec9394a20ad77add3808e555a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:29.272821 859 leader_election.cc:343] T f98d1d8d660443cb8066a547e5d9b77f P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:29.274474 1150 raft_consensus.cc:383] T d0b7c84ec9394a20ad77add3808e555a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:29.274724 1150 raft_consensus.cc:738] T d0b7c84ec9394a20ad77add3808e555a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:29.275032 1150 consensus_queue.cc:260] T d0b7c84ec9394a20ad77add3808e555a P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:29.275602 1150 ts_tablet_manager.cc:1428] T d0b7c84ec9394a20ad77add3808e555a P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250901 01:52:29.274608 859 leader_election.cc:304] T f98d1d8d660443cb8066a547e5d9b77f P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:29.275956 1150 tablet_bootstrap.cc:492] T f195b040a6ae4b7b83a8308ab0a12b37 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
W20250901 01:52:29.276185 859 leader_election.cc:343] T 4cde5f4f02044b7bbd7d43b49aac4ae8 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:29.276306 859 leader_election.cc:304] T 4cde5f4f02044b7bbd7d43b49aac4ae8 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:29.276882 2323 raft_consensus.cc:2747] T f98d1d8d660443cb8066a547e5d9b77f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:29.277073 2323 raft_consensus.cc:2747] T 4cde5f4f02044b7bbd7d43b49aac4ae8 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:29.241293 1492 rpcz_store.cc:269] 0901 01:52:27.249612 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:27.249756 (+ 144us) service_pool.cc:225] Handling call
0901 01:52:29.227455 (+1977699us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0901 01:52:27.988479 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:27.988631 (+ 152us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:27.988640 (+ 9us) write_op.cc:435] Acquired schema lock
0901 01:52:27.988645 (+ 5us) tablet.cc:660] Decoding operations
0901 01:52:27.988912 (+ 267us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:27.988926 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:27.988930 (+ 4us) tablet.cc:683] Acquiring locks for 7 operations
0901 01:52:27.989069 (+ 139us) tablet.cc:699] Row locks acquired
0901 01:52:27.989072 (+ 3us) write_op.cc:260] PREPARE: finished
0901 01:52:27.989108 (+ 36us) write_op.cc:270] Start()
0901 01:52:27.989143 (+ 35us) write_op.cc:276] Timestamp: P: 1756691547989045 usec, L: 0
0901 01:52:27.989146 (+ 3us) op_driver.cc:348] REPLICATION: starting
0901 01:52:27.989362 (+ 216us) log.cc:844] Serialized 2772 byte log entry
0901 01:52:28.002071 (+ 12709us) op_driver.cc:464] REPLICATION: finished
0901 01:52:28.002416 (+ 345us) write_op.cc:301] APPLY: starting
0901 01:52:28.002436 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:28.003212 (+ 776us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:28.003253 (+ 41us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:28.003451 (+ 198us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:28.003507 (+ 56us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=7
0901 01:52:28.003515 (+ 8us) write_op.cc:312] APPLY: finished
0901 01:52:28.003599 (+ 84us) log.cc:844] Serialized 59 byte log entry
0901 01:52:28.003663 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:28.003684 (+ 21us) write_op.cc:454] Released schema lock
0901 01:52:28.003691 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0901 01:52:28.003771 (+ 80us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:28.003775 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0901 01:52:28.160801 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:28.160892 (+ 91us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:28.160897 (+ 5us) write_op.cc:435] Acquired schema lock
0901 01:52:28.160899 (+ 2us) tablet.cc:660] Decoding operations
0901 01:52:28.160970 (+ 71us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:28.160978 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:28.161258 (+ 280us) tablet.cc:683] Acquiring locks for 1 operations
0901 01:52:28.161313 (+ 55us) tablet.cc:699] Row locks acquired
0901 01:52:28.161317 (+ 4us) write_op.cc:260] PREPARE: finished
0901 01:52:28.161346 (+ 29us) write_op.cc:270] Start()
0901 01:52:28.161373 (+ 27us) write_op.cc:276] Timestamp: P: 1756691548161313 usec, L: 0
0901 01:52:28.161379 (+ 6us) op_driver.cc:348] REPLICATION: starting
0901 01:52:28.161514 (+ 135us) log.cc:844] Serialized 356 byte log entry
0901 01:52:28.232292 (+ 70778us) op_driver.cc:464] REPLICATION: finished
0901 01:52:28.239085 (+ 6793us) write_op.cc:301] APPLY: starting
0901 01:52:28.240781 (+ 1696us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:28.240827 (+ 46us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:28.240895 (+ 68us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:28.240989 (+ 94us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:28.241037 (+ 48us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0901 01:52:28.241046 (+ 9us) write_op.cc:312] APPLY: finished
0901 01:52:28.241175 (+ 129us) log.cc:844] Serialized 23 byte log entry
0901 01:52:28.241499 (+ 324us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:28.241522 (+ 23us) write_op.cc:454] Released schema lock
0901 01:52:28.241532 (+ 10us) write_op.cc:341] FINISH: Updating metrics
0901 01:52:28.241696 (+ 164us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:28.241701 (+ 5us) write_op.cc:454] Released schema lock
Related trace 'op':
0901 01:52:28.459919 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:28.460026 (+ 107us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:28.460031 (+ 5us) write_op.cc:435] Acquired schema lock
0901 01:52:28.460034 (+ 3us) tablet.cc:660] Decoding operations
0901 01:52:28.460123 (+ 89us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:28.460135 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:28.460139 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0901 01:52:28.460196 (+ 57us) tablet.cc:699] Row locks acquired
0901 01:52:28.460199 (+ 3us) write_op.cc:260] PREPARE: finished
0901 01:52:28.460234 (+ 35us) write_op.cc:270] Start()
0901 01:52:28.460266 (+ 32us) write_op.cc:276] Timestamp: P: 1756691548460171 usec, L: 0
0901 01:52:28.460269 (+ 3us) op_driver.cc:348] REPLICATION: starting
0901 01:52:28.461087 (+ 818us) log.cc:844] Serialized 356 byte log entry
0901 01:52:28.465168 (+ 4081us) op_driver.cc:464] REPLICATION: finished
0901 01:52:28.465281 (+ 113us) write_op.cc:301] APPLY: starting
0901 01:52:28.465296 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:28.465364 (+ 68us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:28.465409 (+ 45us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:28.465465 (+ 56us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:28.465500 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0901 01:52:28.465509 (+ 9us) write_op.cc:312] APPLY: finished
0901 01:52:28.465587 (+ 78us) log.cc:844] Serialized 23 byte log entry
0901 01:52:28.465647 (+ 60us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:28.465667 (+ 20us) write_op.cc:454] Released schema lock
0901 01:52:28.465673 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0901 01:52:28.465754 (+ 81us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:28.465759 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":222,"child_traces":[["op",{"apply.queue_time_us":287,"apply.run_cpu_time_us":1347,"apply.run_wall_time_us":1347,"num_ops":7,"prepare.queue_time_us":2266,"prepare.run_cpu_time_us":987,"prepare.run_wall_time_us":987,"raft.queue_time_us":1336,"raft.run_cpu_time_us":294,"raft.run_wall_time_us":295,"replication_time_us":12894}],["op",{"apply.queue_time_us":6708,"apply.run_cpu_time_us":661,"apply.run_wall_time_us":2610,"num_ops":1,"prepare.queue_time_us":124653,"prepare.run_cpu_time_us":629,"prepare.run_wall_time_us":32604,"raft.queue_time_us":174,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":172,"replication_time_us":70879}],["op",{"apply.queue_time_us":67,"apply.run_cpu_time_us":450,"apply.run_wall_time_us":449,"num_ops":1,"prepare.queue_time_us":47,"prepare.run_cpu_time_us":720,"prepare.run_wall_time_us":1274,"raft.queue_time_us":389,"raft.run_cpu_time_us":242,"raft.run_wall_time_us":461,"replication_time_us":4870}]]}
I20250901 01:52:29.277851 2323 raft_consensus.cc:491] T 61ddcf86c9f34726939cc116434f0822 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:29.277993 2323 raft_consensus.cc:513] T 61ddcf86c9f34726939cc116434f0822 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:29.278225 1150 tablet_bootstrap.cc:654] T f195b040a6ae4b7b83a8308ab0a12b37 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:29.278616 2323 leader_election.cc:290] T 61ddcf86c9f34726939cc116434f0822 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:29.279811 1489 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5f5396045979450193d461ba1334a5c9 consensus state has no leader
I20250901 01:52:29.280130 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_05a9fc41f9c54ba7a95256f61907ad6b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:29.280292 858 leader_election.cc:336] T 61ddcf86c9f34726939cc116434f0822 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:29.282603 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_99404881b1de4a11b83d742f56a5a733"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:29.285566 1490 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e2ed32b3292141278af28026528e96c2 has no consensus state
I20250901 01:52:29.285962 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2429f2670f6a40ed8d1d5da7a32ec94f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:29.287370 859 leader_election.cc:343] T 93bcac1fd95d44dc927abfca81692921 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:29.287508 859 leader_election.cc:304] T 93bcac1fd95d44dc927abfca81692921 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:29.287873 1483 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet e4fdbc8d4bff4173a20db2c182acfd9e
I20250901 01:52:29.288033 2357 raft_consensus.cc:2747] T 93bcac1fd95d44dc927abfca81692921 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:29.288121 1483 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet e4fdbc8d4bff4173a20db2c182acfd9e: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:29.288242 2357 consensus_queue.cc:1035] T 3d90c6ef06d1436d8e2bb1333e7d9985 P 00c282f779c1477d920f24dc0dc16628 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250901 01:52:29.294677 1491 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4cde5f4f02044b7bbd7d43b49aac4ae8 consensus state has no leader
I20250901 01:52:29.296214 1489 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_8445fcd774874cadaf4acc3581eee417" } modify_external_catalogs: true
I20250901 01:52:29.296515 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_6db53bd9d52f4723804622ce01534ff6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.297546 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_473fa7b0bd4442b99936d6a27ecf95ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.298272 885 tablet_service.cc:1515] Processing DeleteTablet for tablet e4fdbc8d4bff4173a20db2c182acfd9e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:29 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:29.298913 1489 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_8445fcd774874cadaf4acc3581eee417" } modify_external_catalogs: true
I20250901 01:52:29.299584 1488 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet e5e250e6c84d4248b857bf1478d47703
I20250901 01:52:29.299728 1058 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "61ddcf86c9f34726939cc116434f0822" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:29.299820 1488 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet e5e250e6c84d4248b857bf1478d47703: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:29.301654 2118 ts_tablet_manager.cc:1918] T 6cf09e8f299548dba2cce16d7b565bad P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:29.303936 2118 log.cc:1199] T 6cf09e8f299548dba2cce16d7b565bad P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/6cf09e8f299548dba2cce16d7b565bad
I20250901 01:52:29.304379 2118 ts_tablet_manager.cc:1939] T 6cf09e8f299548dba2cce16d7b565bad P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:29.304522 1151 tablet_bootstrap.cc:492] T 0dbdcb76eeba4fefa56e7835c23c0316 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:29.303778 1150 tablet_bootstrap.cc:492] T f195b040a6ae4b7b83a8308ab0a12b37 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:29.305361 1150 ts_tablet_manager.cc:1397] T f195b040a6ae4b7b83a8308ab0a12b37 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.000s
I20250901 01:52:29.305843 885 tablet_service.cc:1515] Processing DeleteTablet for tablet e5e250e6c84d4248b857bf1478d47703 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:29 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:29.305999 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14b466b2a5a249aa841f1b8b06301edc" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:29.306318 1150 raft_consensus.cc:357] T f195b040a6ae4b7b83a8308ab0a12b37 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:29.306581 1150 raft_consensus.cc:383] T f195b040a6ae4b7b83a8308ab0a12b37 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:29.306715 1150 raft_consensus.cc:738] T f195b040a6ae4b7b83a8308ab0a12b37 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:29.306995 1150 consensus_queue.cc:260] T f195b040a6ae4b7b83a8308ab0a12b37 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:29.307077 2412 consensus_queue.cc:1035] T 500e3c33598b45a4afa983da7cb0845a P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20250901 01:52:29.307708 1150 ts_tablet_manager.cc:1428] T f195b040a6ae4b7b83a8308ab0a12b37 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250901 01:52:29.308043 1150 tablet_bootstrap.cc:492] T 764bfeb2a7d14c40815b5ec1b07bdd62 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:29.309785 1150 tablet_bootstrap.cc:654] T 764bfeb2a7d14c40815b5ec1b07bdd62 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:29.310681 1057 raft_consensus.cc:2466] T 14b466b2a5a249aa841f1b8b06301edc P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00c282f779c1477d920f24dc0dc16628 in term 0.
I20250901 01:52:29.310623 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 87) took 4029 ms. Trace:
I20250901 01:52:29.318441 1151 tablet_bootstrap.cc:654] T 0dbdcb76eeba4fefa56e7835c23c0316 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:29.318583 2118 rpcz_store.cc:276] 0901 01:52:25.281036 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.281081 (+ 45us) service_pool.cc:225] Handling call
0901 01:52:29.254985 (+3973904us) ts_tablet_manager.cc:1145] Deleting tablet 6cf09e8f299548dba2cce16d7b565bad
0901 01:52:29.254998 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:29.284091 (+ 29093us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:29.301491 (+ 17400us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:29.310585 (+ 9094us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3874283}
I20250901 01:52:29.319255 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 6cf09e8f299548dba2cce16d7b565bad (table default.Table_1adc70cccd2f4cb3a79df96861cb4181 [id=24fffc7ebf5a47bdb37211910301acf7]) successfully deleted
I20250901 01:52:29.319657 2118 tablet_replica.cc:331] T d58f69fad01f403eab74a47d3695ed83 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:29.319957 2118 raft_consensus.cc:2241] T d58f69fad01f403eab74a47d3695ed83 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:29.320207 2118 raft_consensus.cc:2270] T d58f69fad01f403eab74a47d3695ed83 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:29.320904 2118 ts_tablet_manager.cc:1905] T d58f69fad01f403eab74a47d3695ed83 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:29.321322 859 leader_election.cc:343] T 61ddcf86c9f34726939cc116434f0822 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:29.326318 859 leader_election.cc:304] T 61ddcf86c9f34726939cc116434f0822 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:29.327278 2357 raft_consensus.cc:2747] T 61ddcf86c9f34726939cc116434f0822 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:29.332782 2526 tablet.cc:1722] T e87d1281f35648b0b115b67c95e244e7 P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:29.332954 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_82ff953c3c8f420d93a89f4bce84f71d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.334733 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e492fa33803f40cc917147d4fddf3665"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.335004 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e2850caebc4d4a3e9b8e5933781cbad3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.337016 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_0832f6047fe64631aa973f258845a53b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.338569 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a181c832e6354a56aac9eb8bde90f4a2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:29.339665 1482 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:54603 (request call id 183) took 1262 ms (client timeout 1000 ms). Trace:
I20250901 01:52:29.340251 1488 master_service.cc:948] ReplaceTablet: received request to replace tablet 69f61ec283484f9d8b3eebcea27f741b from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:29.341270 2118 ts_tablet_manager.cc:1918] T d58f69fad01f403eab74a47d3695ed83 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:29.341562 2118 log.cc:1199] T d58f69fad01f403eab74a47d3695ed83 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/d58f69fad01f403eab74a47d3695ed83
I20250901 01:52:29.342358 2118 ts_tablet_manager.cc:1939] T d58f69fad01f403eab74a47d3695ed83 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:29.342419 1103 heartbeater.cc:499] Master 127.0.104.60:44231 was elected leader, sending a full tablet report...
I20250901 01:52:29.344547 1150 tablet_bootstrap.cc:492] T 764bfeb2a7d14c40815b5ec1b07bdd62 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:29.345659 859 leader_election.cc:304] T 14b466b2a5a249aa841f1b8b06301edc P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:29.346760 2357 raft_consensus.cc:2802] T 14b466b2a5a249aa841f1b8b06301edc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:29.346884 2357 raft_consensus.cc:491] T 14b466b2a5a249aa841f1b8b06301edc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:29.347081 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 73) took 4122 ms. Trace:
I20250901 01:52:29.346820 1150 ts_tablet_manager.cc:1397] T 764bfeb2a7d14c40815b5ec1b07bdd62 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.038s user 0.005s sys 0.000s
I20250901 01:52:29.347321 2323 raft_consensus.cc:491] T d0b7c84ec9394a20ad77add3808e555a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:29.347215 2357 raft_consensus.cc:3058] T 14b466b2a5a249aa841f1b8b06301edc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:29.347517 2323 raft_consensus.cc:513] T d0b7c84ec9394a20ad77add3808e555a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:29.348196 2323 leader_election.cc:290] T d0b7c84ec9394a20ad77add3808e555a P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:29.347206 2118 rpcz_store.cc:276] 0901 01:52:25.224822 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.224869 (+ 47us) service_pool.cc:225] Handling call
0901 01:52:29.319594 (+4094725us) ts_tablet_manager.cc:1145] Deleting tablet d58f69fad01f403eab74a47d3695ed83
0901 01:52:29.319603 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:29.332618 (+ 13015us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:29.341119 (+ 8501us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:29.347016 (+ 5897us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3938756}
W20250901 01:52:29.349798 992 consensus_peers.cc:489] T 9127f9c5239e445ba86012951b7c75d9 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250901 01:52:29.349992 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet d58f69fad01f403eab74a47d3695ed83 (table default.Table_95884295301d476fb3e03054727f9cf1 [id=1aa18a2513db4e99a72572846527663c]) successfully deleted
I20250901 01:52:29.350332 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 80) took 4123 ms. Trace:
I20250901 01:52:29.350474 2118 rpcz_store.cc:276] 0901 01:52:25.225928 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.226104 (+ 176us) service_pool.cc:225] Handling call
0901 01:52:29.348914 (+4122810us) ts_tablet_manager.cc:1145] Deleting tablet eac50cbae691416e903cb2cbc907f96d
0901 01:52:29.348925 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:29.348998 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3967887}
I20250901 01:52:29.350883 2118 tablet_replica.cc:331] T 6c05eff658924ca4a9a1e7a37f917d67 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:29.351142 2118 raft_consensus.cc:2241] T 6c05eff658924ca4a9a1e7a37f917d67 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:29.351367 2118 raft_consensus.cc:2270] T 6c05eff658924ca4a9a1e7a37f917d67 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:29.352002 2118 ts_tablet_manager.cc:1905] T 6c05eff658924ca4a9a1e7a37f917d67 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:29.339831 1482 rpcz_store.cc:269] 0901 01:52:28.077677 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.081619 (+ 3942us) service_pool.cc:225] Handling call
0901 01:52:28.551224 (+469605us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:28.551233 (+ 9us) write_op.cc:454] Released schema lock
0901 01:52:29.339630 (+788397us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0901 01:52:28.527726 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:28.527925 (+ 199us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:28.527936 (+ 11us) write_op.cc:435] Acquired schema lock
0901 01:52:28.527939 (+ 3us) tablet.cc:660] Decoding operations
0901 01:52:28.528113 (+ 174us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:28.528127 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:28.528132 (+ 5us) tablet.cc:683] Acquiring locks for 3 operations
0901 01:52:28.528226 (+ 94us) tablet.cc:699] Row locks acquired
0901 01:52:28.528228 (+ 2us) write_op.cc:260] PREPARE: finished
0901 01:52:28.528268 (+ 40us) write_op.cc:270] Start()
0901 01:52:28.528305 (+ 37us) write_op.cc:276] Timestamp: P: 1756691548528195 usec, L: 0
0901 01:52:28.528308 (+ 3us) op_driver.cc:348] REPLICATION: starting
0901 01:52:28.528614 (+ 306us) log.cc:844] Serialized 1275 byte log entry
0901 01:52:28.532318 (+ 3704us) op_driver.cc:464] REPLICATION: finished
0901 01:52:28.532428 (+ 110us) write_op.cc:301] APPLY: starting
0901 01:52:28.532443 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:28.532514 (+ 71us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:28.532566 (+ 52us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:28.532648 (+ 82us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:28.532686 (+ 38us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0901 01:52:28.532696 (+ 10us) write_op.cc:312] APPLY: finished
0901 01:52:28.532788 (+ 92us) log.cc:844] Serialized 35 byte log entry
0901 01:52:28.532855 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:28.532892 (+ 37us) write_op.cc:454] Released schema lock
0901 01:52:28.532901 (+ 9us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0901 01:52:29.167127 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:29.167209 (+ 82us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:29.167214 (+ 5us) write_op.cc:435] Acquired schema lock
0901 01:52:29.167216 (+ 2us) tablet.cc:660] Decoding operations
0901 01:52:29.167287 (+ 71us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:29.167294 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:29.167299 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0901 01:52:29.167340 (+ 41us) tablet.cc:699] Row locks acquired
0901 01:52:29.167342 (+ 2us) write_op.cc:260] PREPARE: finished
0901 01:52:29.167365 (+ 23us) write_op.cc:270] Start()
0901 01:52:29.167387 (+ 22us) write_op.cc:276] Timestamp: P: 1756691549167339 usec, L: 0
0901 01:52:29.167390 (+ 3us) op_driver.cc:348] REPLICATION: starting
0901 01:52:29.167565 (+ 175us) log.cc:844] Serialized 356 byte log entry
0901 01:52:29.178590 (+ 11025us) op_driver.cc:464] REPLICATION: finished
0901 01:52:29.179166 (+ 576us) write_op.cc:301] APPLY: starting
0901 01:52:29.179177 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:29.179201 (+ 24us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:29.179222 (+ 21us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:29.179260 (+ 38us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:29.179282 (+ 22us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0901 01:52:29.179288 (+ 6us) write_op.cc:312] APPLY: finished
0901 01:52:29.179341 (+ 53us) log.cc:844] Serialized 23 byte log entry
0901 01:52:29.179384 (+ 43us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:29.179397 (+ 13us) write_op.cc:454] Released schema lock
0901 01:52:29.179402 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0901 01:52:29.179458 (+ 56us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:29.179462 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":180,"child_traces":[["op",{"apply.queue_time_us":51,"apply.run_cpu_time_us":529,"apply.run_wall_time_us":528,"num_ops":3,"prepare.queue_time_us":282,"prepare.run_cpu_time_us":986,"prepare.run_wall_time_us":995,"raft.queue_time_us":72,"raft.run_cpu_time_us":243,"raft.run_wall_time_us":243,"replication_time_us":3983}],["op",{"apply.queue_time_us":536,"apply.run_cpu_time_us":286,"apply.run_wall_time_us":286,"num_ops":1,"prepare.queue_time_us":11880,"prepare.run_cpu_time_us":492,"prepare.run_wall_time_us":491,"replication_time_us":11185}]]}
I20250901 01:52:29.356431 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_6603f72117ef4a0ead395fcfc3d1c6ee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:29.356609 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet eac50cbae691416e903cb2cbc907f96d because the tablet was not found. No further retry: Not found: Tablet not found: eac50cbae691416e903cb2cbc907f96d
W20250901 01:52:29.357542 858 leader_election.cc:336] T d0b7c84ec9394a20ad77add3808e555a P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:29.360074 2357 raft_consensus.cc:513] T 14b466b2a5a249aa841f1b8b06301edc P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:29.362380 2357 leader_election.cc:290] T 14b466b2a5a249aa841f1b8b06301edc P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:29.364519 858 leader_election.cc:336] T 14b466b2a5a249aa841f1b8b06301edc P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:29.357566 1150 raft_consensus.cc:357] T 764bfeb2a7d14c40815b5ec1b07bdd62 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:29.368654 1150 raft_consensus.cc:383] T 764bfeb2a7d14c40815b5ec1b07bdd62 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:29.368816 1150 raft_consensus.cc:738] T 764bfeb2a7d14c40815b5ec1b07bdd62 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:29.369378 2118 ts_tablet_manager.cc:1918] T 6c05eff658924ca4a9a1e7a37f917d67 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:29.370648 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1b8195ee047346869b506328b129181d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.372875 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_233231a5fb244de4bed9184f47e2aecb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:29.358714 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 494b6488aa0d4355b4caa46004267465, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_eecfab7239cc40c5a2f1fb6ac6d35dc9'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:29.374207 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 0ee3f35392954817b350e5f48653be78, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_442e11b9211c4b209c68f653d7bfc313'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:29.369554 1150 consensus_queue.cc:260] T 764bfeb2a7d14c40815b5ec1b07bdd62 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
E20250901 01:52:29.374347 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae4844183a3c4a04a775e6fd878391a3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_aee74198b51d4f089538c4f2bf3ce4dc'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:29.374444 1696 catalog_manager.cc:6205] Error selecting replicas for tablet be8dfbc466564842a2fdb4953eee939b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_75ba3ba70ca24a94920e7bce96190ff2'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:29.374569 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9745582cebfe47b58c21d0191ab4cb9b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77728af9eee44f29a1e2131d5b75b7ca'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:29.374687 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 4abf9725be5d42f3893f07f0a9c7614c, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_8445fcd774874cadaf4acc3581eee417'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:29.374861 1696 catalog_manager.cc:6205] Error selecting replicas for tablet e2ed32b3292141278af28026528e96c2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b5077f55951d48be92a67407ffe26a08'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:29.374792 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_79e0c3abee854631a361d97ec14e8563"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:29.374966 1696 catalog_manager.cc:6205] Error selecting replicas for tablet dcc60bb000f440f1aca9133928b5ce20, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2fbca8463d6546c9b8f5b018fc60251d'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:29.375099 1696 catalog_manager.cc:6205] Error selecting replicas for tablet a2ca69fabadd4ad69e1670cae7a0c503, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_32c4716e3aa24a87a6b1472984b255c4'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:29.375207 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f8f923ddcd2740bcb7e521f4f57c4034, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_5e65c5f068b84995a2d7a76fa65dda0f'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:29.375341 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 0813cc68d4b0488ba60a523e4479971b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d1302673d68f424ebb54b220911ce385'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:29.375453 1696 catalog_manager.cc:6205] Error selecting replicas for tablet b1ba0e19dc0b462e80c537bf2d183f5b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0e2cd82b00f1466eaef7f287402e644d'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:29.375584 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 8e10a01e77d14cf28c2c5e0205e595b9, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_83cf2d86a1284544ba2e7aa9ebb66bb7'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:29.375690 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5f0e054f62de4cf6bc6f69ad6664b192, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ef6832d0845044ed9f165ca3541b4c3f'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:29.375836 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 787be682c0e44248a80d7037fb61dc6c, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7c12d66048a6470e8c227dc1d9f45fa4'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:29.375952 1696 catalog_manager.cc:6205] Error selecting replicas for tablet cc3b8736b9094d7e892c80da4d5b20f6, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_93afd37289414655980b22934695bd55'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:29.376075 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae9a4ad4fdfb4771ba23a91f7007447e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_5e84ae9a9c2a4084a1f9c5ffb4594de2'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:29.376175 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9074543dea9949e7a085717b40913348, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_53e30205c5fd488eb3b48c685a159d04'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:29.376291 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 3a62034e688348ed9e2ac9de009ed3a2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_409dad98d7eb455a8e16edfb4acedb34'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:29.376595 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8c5fed27285748f38933c5ead3a8ff8b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.378275 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f79f0ea253cb424f90bf7130e31a391c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.374262 2118 log.cc:1199] T 6c05eff658924ca4a9a1e7a37f917d67 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/6c05eff658924ca4a9a1e7a37f917d67
I20250901 01:52:29.379499 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4f38b2f845204f2791b20698a61d8ec8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.379753 2357 raft_consensus.cc:491] T 4fca1435d82447f289fda4035aa2ed89 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:29.379891 2118 ts_tablet_manager.cc:1939] T 6c05eff658924ca4a9a1e7a37f917d67 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:29.379904 2357 raft_consensus.cc:513] T 4fca1435d82447f289fda4035aa2ed89 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:29.380259 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 96) took 3673 ms. Trace:
I20250901 01:52:29.380348 2118 rpcz_store.cc:276] 0901 01:52:25.706518 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.706624 (+ 106us) service_pool.cc:225] Handling call
0901 01:52:29.350790 (+3644166us) ts_tablet_manager.cc:1145] Deleting tablet 6c05eff658924ca4a9a1e7a37f917d67
0901 01:52:29.350799 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:29.354993 (+ 4194us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:29.369231 (+ 14238us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:29.380243 (+ 11012us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3643943}
I20250901 01:52:29.381711 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3358011270c34ab598861d1ab966a439"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.384310 1993 raft_consensus.cc:491] T 12c04e0c1f804179ae23509cdb24d395 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:29.380658 2357 leader_election.cc:290] T 4fca1435d82447f289fda4035aa2ed89 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:29.382980 858 leader_election.cc:336] T 4fca1435d82447f289fda4035aa2ed89 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:29.384678 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 97) took 3678 ms. Trace:
I20250901 01:52:29.385830 2118 rpcz_store.cc:276] 0901 01:52:25.706710 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.706760 (+ 50us) service_pool.cc:225] Handling call
0901 01:52:29.384553 (+3677793us) ts_tablet_manager.cc:1145] Deleting tablet eee34044c39d4013baf78d3705489626
0901 01:52:29.384565 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:29.384660 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3677643}
I20250901 01:52:29.386394 2118 tablet_replica.cc:331] T 7cc1efcb6bff4aba8015d2d3b7f4c923 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:29.386651 2118 raft_consensus.cc:2241] T 7cc1efcb6bff4aba8015d2d3b7f4c923 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:29.386863 2118 raft_consensus.cc:2270] T 7cc1efcb6bff4aba8015d2d3b7f4c923 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:29.388437 1482 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_f0e56dba1e8a4ed49a760e204c1ff96e" } modify_external_catalogs: true
I20250901 01:52:29.388731 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 6c05eff658924ca4a9a1e7a37f917d67 (table default.Table_c1959bda12f14bc9995d832476abf670 [id=17355a61629849dd9d8228a469ab744f]) successfully deleted
I20250901 01:52:29.389122 1482 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_f0e56dba1e8a4ed49a760e204c1ff96e" } modify_external_catalogs: true
I20250901 01:52:29.392232 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_519967b17fd243bab8d031220d85ae9a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.394556 2118 ts_tablet_manager.cc:1905] T 7cc1efcb6bff4aba8015d2d3b7f4c923 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:29.392733 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ddcf8f4d243b4f10a22ad9c1655aed91"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.393215 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_fb4863d15585482b9b79277cec3c3d4f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:29.389118 1472 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet eee34044c39d4013baf78d3705489626 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eee34044c39d4013baf78d3705489626 already in progress: creating tablet
I20250901 01:52:29.403050 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f6fa65bb79aa44c6a2df1be8974c3a99"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.404980 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_bfd1c13833eb413793b255843cb6314a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.403927 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f977275d2c554d7699717d99f70f8e64"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.407254 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_018728839f3b4f019ea0ff03eddc8f5a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.409896 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_67ab4fbd0c144c6eb1d457c60c91e159"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.412050 2118 ts_tablet_manager.cc:1918] T 7cc1efcb6bff4aba8015d2d3b7f4c923 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 01:52:29.412045 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_09a4afab37d14129bb66cd9b99b828c7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.415378 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_995ca84e8109434fae79cbeb0b25bd92"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.414659 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_799ea94daa674da2add8fe1d915c399d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.417485 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ac459d99d69a4887964be93931294f1b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.417197 2357 raft_consensus.cc:491] T abf6e3be60f74872b2308dc95bf9cd37 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:29.427706 2118 log.cc:1199] T 7cc1efcb6bff4aba8015d2d3b7f4c923 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/7cc1efcb6bff4aba8015d2d3b7f4c923
I20250901 01:52:29.427722 2357 raft_consensus.cc:513] T abf6e3be60f74872b2308dc95bf9cd37 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:29.428206 2118 ts_tablet_manager.cc:1939] T 7cc1efcb6bff4aba8015d2d3b7f4c923 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:29.428462 2357 leader_election.cc:290] T abf6e3be60f74872b2308dc95bf9cd37 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:29.428619 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 98) took 3690 ms. Trace:
I20250901 01:52:29.428727 2118 rpcz_store.cc:276] 0901 01:52:25.738344 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.738407 (+ 63us) service_pool.cc:225] Handling call
0901 01:52:29.386339 (+3647932us) ts_tablet_manager.cc:1145] Deleting tablet 7cc1efcb6bff4aba8015d2d3b7f4c923
0901 01:52:29.386349 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:29.404210 (+ 17861us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:29.411939 (+ 7729us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:29.428601 (+ 16662us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3647691}
I20250901 01:52:29.429148 2118 tablet_replica.cc:331] T 06d5eeb70e664f73bf643922d445bc7a P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:29.429482 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 7cc1efcb6bff4aba8015d2d3b7f4c923 (table default.Table_5610feff8a344dc9be4bc1f0ad8ddd51 [id=78b00856a9b747c19f390b4fdae8ddcf]) successfully deleted
I20250901 01:52:29.430655 2118 raft_consensus.cc:2241] T 06d5eeb70e664f73bf643922d445bc7a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:29.430894 2118 raft_consensus.cc:2270] T 06d5eeb70e664f73bf643922d445bc7a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250901 01:52:29.431115 858 leader_election.cc:336] T abf6e3be60f74872b2308dc95bf9cd37 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:29.431591 2118 ts_tablet_manager.cc:1905] T 06d5eeb70e664f73bf643922d445bc7a P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
E20250901 01:52:29.436834 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 18498c9cb09141d08c1c5c0854b63ebd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77f4810bc83c4d948531da2a45a3792e'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:29.430387 1150 ts_tablet_manager.cc:1428] T 764bfeb2a7d14c40815b5ec1b07bdd62 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.083s user 0.015s sys 0.000s
I20250901 01:52:29.440840 1150 tablet_bootstrap.cc:492] T d000ba05e12843b4925e622ab55471c0 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:29.442395 1150 tablet_bootstrap.cc:654] T d000ba05e12843b4925e622ab55471c0 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:29.444815 1993 raft_consensus.cc:513] T 12c04e0c1f804179ae23509cdb24d395 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:29.445508 1993 leader_election.cc:290] T 12c04e0c1f804179ae23509cdb24d395 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:29.447577 2409 raft_consensus.cc:491] T ff013ea889d44cba8797834e4611b489 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:29.436928 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet c763d252598745a088344aa0eb5fa3d3 (table default.Table_142ccc8cc7414621bc0c31326d5aa0e9 [id=6a01b6c6977546be9c6128ca5f200a92]) successfully deleted
I20250901 01:52:29.449476 2113 ts_tablet_manager.cc:1918] T 579b9cdf819f4609a532e96bec093fd5 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:29.449610 2113 log.cc:1199] T 579b9cdf819f4609a532e96bec093fd5 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/579b9cdf819f4609a532e96bec093fd5
I20250901 01:52:29.450031 2113 ts_tablet_manager.cc:1939] T 579b9cdf819f4609a532e96bec093fd5 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:29.450351 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 103) took 4349 ms. Trace:
I20250901 01:52:29.450474 2113 rpcz_store.cc:276] 0901 01:52:25.100769 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.100929 (+ 160us) service_pool.cc:225] Handling call
0901 01:52:29.214223 (+4113294us) ts_tablet_manager.cc:1145] Deleting tablet 579b9cdf819f4609a532e96bec093fd5
0901 01:52:29.214232 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:29.317215 (+102983us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:29.449338 (+132123us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:29.450320 (+ 982us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4113096}
I20250901 01:52:29.450829 2113 tablet_replica.cc:331] T 822382a1781b4304bce29355ce83730e P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:29.451035 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f80ccecb28484c059624b81a20a61057"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.451771 2113 raft_consensus.cc:2241] T 822382a1781b4304bce29355ce83730e P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
E20250901 01:52:29.455329 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f5e3316adeae47cd848833dd84b972bb, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_af5338b09bbc4c879c7f4f573c32fd8d'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:29.460474 2409 raft_consensus.cc:513] T ff013ea889d44cba8797834e4611b489 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:29.462249 2118 ts_tablet_manager.cc:1918] T 06d5eeb70e664f73bf643922d445bc7a P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:29.463286 2118 log.cc:1199] T 06d5eeb70e664f73bf643922d445bc7a P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/06d5eeb70e664f73bf643922d445bc7a
I20250901 01:52:29.463816 2118 ts_tablet_manager.cc:1939] T 06d5eeb70e664f73bf643922d445bc7a P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:29.464639 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 99) took 3695 ms. Trace:
I20250901 01:52:29.468052 2113 raft_consensus.cc:2270] T 822382a1781b4304bce29355ce83730e P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:29.468346 2118 rpcz_store.cc:276] 0901 01:52:25.768721 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.768784 (+ 63us) service_pool.cc:225] Handling call
0901 01:52:29.429090 (+3660306us) ts_tablet_manager.cc:1145] Deleting tablet 06d5eeb70e664f73bf643922d445bc7a
0901 01:52:29.429102 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:29.437721 (+ 8619us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:29.462126 (+ 24405us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:29.464622 (+ 2496us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3660107}
I20250901 01:52:29.469422 2118 tablet_replica.cc:331] T 410d44e5f1974ed69e5dee65e98856a4 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:29.469667 2409 leader_election.cc:290] T ff013ea889d44cba8797834e4611b489 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:29.469982 2118 raft_consensus.cc:2241] T 410d44e5f1974ed69e5dee65e98856a4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:29.470376 2118 raft_consensus.cc:2270] T 410d44e5f1974ed69e5dee65e98856a4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:29.471134 2118 ts_tablet_manager.cc:1905] T 410d44e5f1974ed69e5dee65e98856a4 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:29.473050 2113 ts_tablet_manager.cc:1905] T 822382a1781b4304bce29355ce83730e P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:29.476158 2528 tablet.cc:1722] T e87d1281f35648b0b115b67c95e244e7 P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
E20250901 01:52:29.478511 1696 catalog_manager.cc:6205] Error selecting replicas for tablet a6b539e160194da3a849d9e2f1ec9859, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_03fb48eaab9b42858ab7cefb722e43a8'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:29.468133 1492 master_service.cc:948] ReplaceTablet: received request to replace tablet e2aa32450a584395ae07b49f76a7b26d from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:29.469144 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 06d5eeb70e664f73bf643922d445bc7a (table default.Table_33c84edd575e4969b264a684accb22e2 [id=86e2bc0201e04503a1f60452605eaf27]) successfully deleted
I20250901 01:52:29.473140 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9a9f23aa1cc44d528acc2a71b480afee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:29.482769 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 6ce9521b925b4f97bb4a810ce77e147e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_9e04f976a93f4d30baccf90fa3622dfe'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:29.482977 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2f61edb50d7a42198bfd70639c7bf4de, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_a5c783cbdb694af69047b31d99ff8def'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:29.466485 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_18703ad7691f4c92ad6317a50e7229a2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.485764 1150 tablet_bootstrap.cc:492] T d000ba05e12843b4925e622ab55471c0 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:29.485908 1150 ts_tablet_manager.cc:1397] T d000ba05e12843b4925e622ab55471c0 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.045s user 0.007s sys 0.000s
I20250901 01:52:29.486061 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3d1f0345beca40f6855d8f9eac9c4f84"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.486642 1150 raft_consensus.cc:357] T d000ba05e12843b4925e622ab55471c0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:29.486793 1150 raft_consensus.cc:383] T d000ba05e12843b4925e622ab55471c0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:29.487402 1150 raft_consensus.cc:738] T d000ba05e12843b4925e622ab55471c0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:29.487702 2221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Bootstrap replayed 1/1 log segments. Stats: ops{read=633 overwritten=0 applied=632 ignored=0} inserts{seen=625 ignored=0} mutations{seen=746 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250901 01:52:29.487730 1150 consensus_queue.cc:260] T d000ba05e12843b4925e622ab55471c0 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:29.487835 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_dd3f4460528a49cba4d89a67287895ae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.488391 1150 ts_tablet_manager.cc:1428] T d000ba05e12843b4925e622ab55471c0 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250901 01:52:29.488615 2221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: Bootstrap complete.
I20250901 01:52:29.488745 1150 tablet_bootstrap.cc:492] T 435b1cca5a7d4d49959a310cb250b2a0 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:29.489394 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5aaf27d3099740b4bbcd0b7b0e459772"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.490097 1150 tablet_bootstrap.cc:654] T 435b1cca5a7d4d49959a310cb250b2a0 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:29.490213 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet e6c10426f60f43e8bde69d9b1b9c8eb6 with delete_type TABLET_DATA_DELETED (replaced by tablet 494b6488aa0d4355b4caa46004267465) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:29.490506 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet b73e362321284df1b18ef755dfe930a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:15 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:29.490811 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet b11e8f77243e4764b96548d1132b8e5a with delete_type TABLET_DATA_DELETED (replaced by tablet 96824e68bbf24b8287dbb683038ffaba) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:29.491691 1035 tablet_service.cc:1515] Processing DeleteTablet for tablet e4fdbc8d4bff4173a20db2c182acfd9e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:29 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:29.491894 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet e5e250e6c84d4248b857bf1478d47703 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:29 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:29.492731 1036 tablet_service.cc:1515] Processing DeleteTablet for tablet 5f52698fb56d472d9e91fb10cb10c69a with delete_type TABLET_DATA_DELETED (replaced by tablet 8e10a01e77d14cf28c2c5e0205e595b9) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:29.493935 2118 ts_tablet_manager.cc:1918] T 410d44e5f1974ed69e5dee65e98856a4 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:29.497071 2113 ts_tablet_manager.cc:1918] T 822382a1781b4304bce29355ce83730e P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:29.498590 2118 log.cc:1199] T 410d44e5f1974ed69e5dee65e98856a4 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/410d44e5f1974ed69e5dee65e98856a4
I20250901 01:52:29.498914 2118 ts_tablet_manager.cc:1939] T 410d44e5f1974ed69e5dee65e98856a4 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:29.499236 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 101) took 3678 ms. Trace:
I20250901 01:52:29.499306 2118 rpcz_store.cc:276] 0901 01:52:25.820765 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.824998 (+ 4233us) service_pool.cc:225] Handling call
0901 01:52:29.469371 (+3644373us) ts_tablet_manager.cc:1145] Deleting tablet 410d44e5f1974ed69e5dee65e98856a4
0901 01:52:29.469382 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:29.475198 (+ 5816us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:29.493836 (+ 18638us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:29.499217 (+ 5381us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3644193}
I20250901 01:52:29.502060 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 102) took 3668 ms. Trace:
I20250901 01:52:29.502302 2118 rpcz_store.cc:276] 0901 01:52:25.833882 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.833934 (+ 52us) service_pool.cc:225] Handling call
0901 01:52:29.501934 (+3668000us) ts_tablet_manager.cc:1145] Deleting tablet d4c9aa723aee4dc4b6237cb5c3722b52
0901 01:52:29.501944 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:29.502043 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3667779}
I20250901 01:52:29.502825 2118 tablet_replica.cc:331] T fa7fb88ded21432a8757f37ec3db5e71 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:29.503120 2118 raft_consensus.cc:2241] T fa7fb88ded21432a8757f37ec3db5e71 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:29.503332 2118 raft_consensus.cc:2270] T fa7fb88ded21432a8757f37ec3db5e71 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:29.503908 2118 ts_tablet_manager.cc:1905] T fa7fb88ded21432a8757f37ec3db5e71 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:29.505823 1150 tablet_bootstrap.cc:492] T 435b1cca5a7d4d49959a310cb250b2a0 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:29.505967 1150 ts_tablet_manager.cc:1397] T 435b1cca5a7d4d49959a310cb250b2a0 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20250901 01:52:29.506918 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 4fd0e4f9d86c4362af96ce9964b72328 with delete_type TABLET_DATA_DELETED (replaced by tablet a656adbf1b29431687e0989f0db3459c) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:29.506768 1150 raft_consensus.cc:357] T 435b1cca5a7d4d49959a310cb250b2a0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:29.507386 1151 tablet_bootstrap.cc:492] T 0dbdcb76eeba4fefa56e7835c23c0316 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:29.507614 1150 raft_consensus.cc:383] T 435b1cca5a7d4d49959a310cb250b2a0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:29.507689 1151 ts_tablet_manager.cc:1397] T 0dbdcb76eeba4fefa56e7835c23c0316 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.231s user 0.007s sys 0.007s
I20250901 01:52:29.507818 1150 raft_consensus.cc:738] T 435b1cca5a7d4d49959a310cb250b2a0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:29.508222 1150 consensus_queue.cc:260] T 435b1cca5a7d4d49959a310cb250b2a0 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:29.509356 1150 ts_tablet_manager.cc:1428] T 435b1cca5a7d4d49959a310cb250b2a0 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250901 01:52:29.507403 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 3a2bc5ad1cd243daa3c92fceca8ba414 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:29 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:29.510792 2113 log.cc:1199] T 822382a1781b4304bce29355ce83730e P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/822382a1781b4304bce29355ce83730e
I20250901 01:52:29.511358 2113 ts_tablet_manager.cc:1939] T 822382a1781b4304bce29355ce83730e P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:29.511799 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 104) took 4410 ms. Trace:
I20250901 01:52:29.511902 2113 rpcz_store.cc:276] 0901 01:52:25.101338 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.101535 (+ 197us) service_pool.cc:225] Handling call
0901 01:52:29.450767 (+4349232us) ts_tablet_manager.cc:1145] Deleting tablet 822382a1781b4304bce29355ce83730e
0901 01:52:29.450777 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:29.481896 (+ 31119us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:29.496929 (+ 15033us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:29.511726 (+ 14797us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3032,"tablet-delete.queue_time_us":4349089}
I20250901 01:52:29.512369 2113 tablet_replica.cc:331] T a59ac57aebe24860aa02f53ce0485910 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:29.512656 2113 raft_consensus.cc:2241] T a59ac57aebe24860aa02f53ce0485910 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:29.512856 2113 raft_consensus.cc:2270] T a59ac57aebe24860aa02f53ce0485910 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:29.510277 1150 tablet_bootstrap.cc:492] T e1c8fe96a6ba4edba4f09c00f7add75e P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:29.517971 1150 tablet_bootstrap.cc:654] T e1c8fe96a6ba4edba4f09c00f7add75e P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:29.520028 1151 raft_consensus.cc:357] T 0dbdcb76eeba4fefa56e7835c23c0316 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:29.523382 1058 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abf6e3be60f74872b2308dc95bf9cd37" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:29.528360 917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff013ea889d44cba8797834e4611b489" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:29.528559 917 raft_consensus.cc:2466] T ff013ea889d44cba8797834e4611b489 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:29.526014 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14b466b2a5a249aa841f1b8b06301edc" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58"
I20250901 01:52:29.526245 1053 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4fca1435d82447f289fda4035aa2ed89" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:29.528693 1151 raft_consensus.cc:383] T 0dbdcb76eeba4fefa56e7835c23c0316 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:29.526852 1051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0b7c84ec9394a20ad77add3808e555a" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:29.528748 1057 raft_consensus.cc:3058] T 14b466b2a5a249aa841f1b8b06301edc P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:29.531464 1057 raft_consensus.cc:2466] T 14b466b2a5a249aa841f1b8b06301edc P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c282f779c1477d920f24dc0dc16628 in term 1.
I20250901 01:52:29.534818 2118 ts_tablet_manager.cc:1918] T fa7fb88ded21432a8757f37ec3db5e71 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:29.537206 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 410d44e5f1974ed69e5dee65e98856a4 (table default.Table_bec114a616504798bede9a9066e57192 [id=ae79f2c40fd7483ea400a36cf3c48888]) successfully deleted
I20250901 01:52:29.538312 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_73129447c4dd45699b2d5c200d18953b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:29.538511 1472 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet d4c9aa723aee4dc4b6237cb5c3722b52 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d4c9aa723aee4dc4b6237cb5c3722b52 already in progress: creating tablet
I20250901 01:52:29.540201 2113 ts_tablet_manager.cc:1905] T a59ac57aebe24860aa02f53ce0485910 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:29.542706 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 822382a1781b4304bce29355ce83730e (table default.Table_b300f31bdab14e24ad40f265ac4de311 [id=4a54bc023da7481fbb0ce573be49f4c3]) successfully deleted
I20250901 01:52:29.542935 1485 master_service.cc:948] ReplaceTablet: received request to replace tablet fecf7d538b4c4c619182d5eccb44dd8b from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:29.549986 2118 log.cc:1199] T fa7fb88ded21432a8757f37ec3db5e71 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/fa7fb88ded21432a8757f37ec3db5e71
I20250901 01:52:29.550565 2118 ts_tablet_manager.cc:1939] T fa7fb88ded21432a8757f37ec3db5e71 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:29.550962 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 103) took 3710 ms. Trace:
I20250901 01:52:29.552313 2118 rpcz_store.cc:276] 0901 01:52:25.841017 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.841066 (+ 49us) service_pool.cc:225] Handling call
0901 01:52:29.502779 (+3661713us) ts_tablet_manager.cc:1145] Deleting tablet fa7fb88ded21432a8757f37ec3db5e71
0901 01:52:29.502788 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:29.531725 (+ 28937us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:29.534726 (+ 3001us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:29.550943 (+ 16217us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3661541}
I20250901 01:52:29.553269 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 105) took 3571 ms. Trace:
I20250901 01:52:29.553406 2118 rpcz_store.cc:276] 0901 01:52:25.982012 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.982843 (+ 831us) service_pool.cc:225] Handling call
0901 01:52:29.553139 (+3570296us) ts_tablet_manager.cc:1145] Deleting tablet 75a703c8e20a4a2fa0ab6ec58d41ef92
0901 01:52:29.553150 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:29.553251 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3570125}
I20250901 01:52:29.555656 2118 tablet_replica.cc:331] T e8e7d2a72f704f52a8f8d46911a8a9e4 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:29.555930 2118 raft_consensus.cc:2241] T e8e7d2a72f704f52a8f8d46911a8a9e4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:29.556149 2118 raft_consensus.cc:2270] T e8e7d2a72f704f52a8f8d46911a8a9e4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:29.557996 2118 ts_tablet_manager.cc:1905] T e8e7d2a72f704f52a8f8d46911a8a9e4 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:29.562678 2408 raft_consensus.cc:491] T 0e4009963cfe42d6ba164fc828897f09 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:29.562824 2408 raft_consensus.cc:513] T 0e4009963cfe42d6ba164fc828897f09 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:29.563470 2408 leader_election.cc:290] T 0e4009963cfe42d6ba164fc828897f09 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:29.565043 858 leader_election.cc:336] T 0e4009963cfe42d6ba164fc828897f09 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:29.567449 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_46397941370d4d4aae2b901598a472ad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.569229 1490 master_service.cc:948] ReplaceTablet: received request to replace tablet 69f61ec283484f9d8b3eebcea27f741b from {username='slave'} at 127.0.0.1:33478
W20250901 01:52:29.570047 1472 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 75a703c8e20a4a2fa0ab6ec58d41ef92 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 75a703c8e20a4a2fa0ab6ec58d41ef92 already in progress: creating tablet
I20250901 01:52:29.570482 1483 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_d5a6ac03c9ff4969ada556d1318b64f0" } new_table_name: "default.Table_63ef1dcd92db46d298446117a62cfba2" modify_external_catalogs: true
I20250901 01:52:29.576058 1488 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 69f61ec283484f9d8b3eebcea27f741b
W20250901 01:52:29.584755 1488 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 69f61ec283484f9d8b3eebcea27f741b: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:29.585180 1488 catalog_manager.cc:6652] ReplaceTablet: tablet 69f61ec283484f9d8b3eebcea27f741b deleted and replaced by tablet 05a547a8866149cb95d23e960a737237
I20250901 01:52:29.576773 1485 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet fecf7d538b4c4c619182d5eccb44dd8b
W20250901 01:52:29.586138 1485 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet fecf7d538b4c4c619182d5eccb44dd8b: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:29.586462 1485 catalog_manager.cc:6652] ReplaceTablet: tablet fecf7d538b4c4c619182d5eccb44dd8b deleted and replaced by tablet 65e5cdc10e9642b7b0df775cb02861fa
I20250901 01:52:29.586493 1035 tablet_service.cc:1515] Processing DeleteTablet for tablet 69f61ec283484f9d8b3eebcea27f741b with delete_type TABLET_DATA_DELETED (replaced by tablet 05a547a8866149cb95d23e960a737237) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:29.588063 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 69f61ec283484f9d8b3eebcea27f741b with delete_type TABLET_DATA_DELETED (replaced by tablet 05a547a8866149cb95d23e960a737237) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:29.588096 887 tablet_service.cc:1515] Processing DeleteTablet for tablet fecf7d538b4c4c619182d5eccb44dd8b with delete_type TABLET_DATA_DELETED (replaced by tablet 65e5cdc10e9642b7b0df775cb02861fa) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:29.574342 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5034bcf54291489f9e662cc46713e474"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:29.588342 1035 tablet_service.cc:1515] Processing DeleteTablet for tablet fecf7d538b4c4c619182d5eccb44dd8b with delete_type TABLET_DATA_DELETED (replaced by tablet 65e5cdc10e9642b7b0df775cb02861fa) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:29.589120 1490 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_259a716264db43c49671b6fef6665f0e" } modify_external_catalogs: true
I20250901 01:52:29.589849 1490 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_259a716264db43c49671b6fef6665f0e" } modify_external_catalogs: true
I20250901 01:52:29.590850 2118 ts_tablet_manager.cc:1918] T e8e7d2a72f704f52a8f8d46911a8a9e4 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:29.590965 2118 log.cc:1199] T e8e7d2a72f704f52a8f8d46911a8a9e4 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/e8e7d2a72f704f52a8f8d46911a8a9e4
I20250901 01:52:29.591274 2118 ts_tablet_manager.cc:1939] T e8e7d2a72f704f52a8f8d46911a8a9e4 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:29.591557 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 107) took 3609 ms. Trace:
I20250901 01:52:29.591631 2118 rpcz_store.cc:276] 0901 01:52:25.982223 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.983041 (+ 818us) service_pool.cc:225] Handling call
0901 01:52:29.555598 (+3572557us) ts_tablet_manager.cc:1145] Deleting tablet e8e7d2a72f704f52a8f8d46911a8a9e4
0901 01:52:29.555608 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:29.581338 (+ 25730us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:29.590719 (+ 9381us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:29.591541 (+ 822us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3572411}
I20250901 01:52:29.589300 1488 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_8c439dfdc16046b682876df01dadd3e0" } new_table_name: "default.Table_a2a8011bb7624d9ea71e7b3169762e43" modify_external_catalogs: true
I20250901 01:52:29.589622 1485 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_0e2cd82b00f1466eaef7f287402e644d" } new_table_name: "default.Table_66b111271c15410aab23682f8f6081ac" modify_external_catalogs: true
I20250901 01:52:29.596165 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet e8e7d2a72f704f52a8f8d46911a8a9e4 (table default.Table_0ece0b3619db43da9d3bb2b0a72a7ef9 [id=623e11cd89c94142beb5c1a46e1adcd9]) successfully deleted
I20250901 01:52:29.597301 2118 tablet_replica.cc:331] T e2aa32450a584395ae07b49f76a7b26d P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:29.598368 2118 raft_consensus.cc:2241] T e2aa32450a584395ae07b49f76a7b26d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:29.601104 2113 ts_tablet_manager.cc:1918] T a59ac57aebe24860aa02f53ce0485910 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:29.598363 1491 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_bb44f1b7c92140309a212038f8bf1286" } new_table_name: "default.Table_a525265208584713b1b3465d63130cd9" modify_external_catalogs: true
I20250901 01:52:29.607394 2118 raft_consensus.cc:2270] T e2aa32450a584395ae07b49f76a7b26d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:29.608042 2118 ts_tablet_manager.cc:1905] T e2aa32450a584395ae07b49f76a7b26d P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:29.604925 2529 raft_consensus.cc:491] T f9b293bf6a17453b9ea9892a6970d0b0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:29.608695 2529 raft_consensus.cc:513] T f9b293bf6a17453b9ea9892a6970d0b0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:29.609293 2529 leader_election.cc:290] T f9b293bf6a17453b9ea9892a6970d0b0 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:29.605139 2409 raft_consensus.cc:491] T 824229183e024feaa975c7f0387c9c1a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:29.609583 2409 raft_consensus.cc:513] T 824229183e024feaa975c7f0387c9c1a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:29.610100 2409 leader_election.cc:290] T 824229183e024feaa975c7f0387c9c1a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:29.610667 2113 log.cc:1199] T a59ac57aebe24860aa02f53ce0485910 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/a59ac57aebe24860aa02f53ce0485910
I20250901 01:52:29.611058 2113 ts_tablet_manager.cc:1939] T a59ac57aebe24860aa02f53ce0485910 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
W20250901 01:52:29.618037 992 leader_election.cc:343] T 12c04e0c1f804179ae23509cdb24d395 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 12c04e0c1f804179ae23509cdb24d395
I20250901 01:52:29.618676 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 109) took 4492 ms. Trace:
I20250901 01:52:29.618788 2113 rpcz_store.cc:276] 0901 01:52:25.126687 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.126738 (+ 51us) service_pool.cc:225] Handling call
0901 01:52:29.512270 (+4385532us) ts_tablet_manager.cc:1145] Deleting tablet a59ac57aebe24860aa02f53ce0485910
0901 01:52:29.512281 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:29.546393 (+ 34112us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:29.600975 (+ 54582us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:29.618657 (+ 17682us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6680,"tablet-delete.queue_time_us":4385242}
I20250901 01:52:29.623389 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet a59ac57aebe24860aa02f53ce0485910 (table default.Table_b8f8ee74648c406eaf589d3ab80934db [id=b8f69313439543be9c11eea88dfa69c5]) successfully deleted
I20250901 01:52:29.623449 2113 tablet_replica.cc:331] T abe27e749fa0485195e0064a49d5431e P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:29.623729 2113 raft_consensus.cc:2241] T abe27e749fa0485195e0064a49d5431e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:29.623940 2113 raft_consensus.cc:2270] T abe27e749fa0485195e0064a49d5431e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:29.624519 2113 ts_tablet_manager.cc:1905] T abe27e749fa0485195e0064a49d5431e P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:29.635656 2118 ts_tablet_manager.cc:1918] T e2aa32450a584395ae07b49f76a7b26d P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:29.635941 2118 log.cc:1199] T e2aa32450a584395ae07b49f76a7b26d P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/e2aa32450a584395ae07b49f76a7b26d
I20250901 01:52:29.636374 2118 ts_tablet_manager.cc:1939] T e2aa32450a584395ae07b49f76a7b26d P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:29.637055 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 108) took 3654 ms. Trace:
I20250901 01:52:29.619668 2221 raft_consensus.cc:357] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227 [term 2 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3add7edb97a40ef9f7c6bc011d13227" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 43871 } } peers { permanent_uuid: "d2ae2cb6af0f4b3a8f47ab1cdd6567d3" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42375 } } peers { permanent_uuid: "7ce2288f784044eea010664e3def133f" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 44231 } }
I20250901 01:52:29.637226 2118 rpcz_store.cc:276] 0901 01:52:25.982331 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.983215 (+ 884us) service_pool.cc:225] Handling call
0901 01:52:29.596675 (+3613460us) ts_tablet_manager.cc:1145] Deleting tablet e2aa32450a584395ae07b49f76a7b26d
0901 01:52:29.597153 (+ 478us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:29.625038 (+ 27885us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:29.635503 (+ 10465us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:29.637033 (+ 1530us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8473,"tablet-delete.queue_time_us":3613322}
I20250901 01:52:29.638098 2118 tablet_replica.cc:331] T becbaae4fd7e44faa679e9479465a7c4 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:29.638407 2118 raft_consensus.cc:2241] T becbaae4fd7e44faa679e9479465a7c4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:29.643306 2118 raft_consensus.cc:2270] T becbaae4fd7e44faa679e9479465a7c4 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:29.644073 2118 ts_tablet_manager.cc:1905] T becbaae4fd7e44faa679e9479465a7c4 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:29.644882 992 leader_election.cc:304] T ff013ea889d44cba8797834e4611b489 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:29.645196 2409 raft_consensus.cc:2802] T ff013ea889d44cba8797834e4611b489 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:29.645262 2409 raft_consensus.cc:491] T ff013ea889d44cba8797834e4611b489 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:29.645315 2409 raft_consensus.cc:3058] T ff013ea889d44cba8797834e4611b489 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:29.646924 2113 ts_tablet_manager.cc:1918] T abe27e749fa0485195e0064a49d5431e P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:29.647014 2113 log.cc:1199] T abe27e749fa0485195e0064a49d5431e P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/abe27e749fa0485195e0064a49d5431e
I20250901 01:52:29.647298 2113 ts_tablet_manager.cc:1939] T abe27e749fa0485195e0064a49d5431e P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:29.647805 2409 raft_consensus.cc:513] T ff013ea889d44cba8797834e4611b489 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:29.648365 2409 leader_election.cc:290] T ff013ea889d44cba8797834e4611b489 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:29.649065 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 110) took 4521 ms. Trace:
I20250901 01:52:29.649199 2113 rpcz_store.cc:276] 0901 01:52:25.127139 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.127176 (+ 37us) service_pool.cc:225] Handling call
0901 01:52:29.623392 (+4496216us) ts_tablet_manager.cc:1145] Deleting tablet abe27e749fa0485195e0064a49d5431e
0901 01:52:29.623406 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:29.631210 (+ 7804us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:29.646823 (+ 15613us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:29.649038 (+ 2215us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4496024}
I20250901 01:52:29.649900 2113 tablet_replica.cc:331] T 824229183e024feaa975c7f0387c9c1a P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:29.649860 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet abe27e749fa0485195e0064a49d5431e (table default.Table_30943b75394441a5a55b195055d1e0cc [id=7e7a4771910546f7ba8c4369c768c943]) successfully deleted
I20250901 01:52:29.650144 2113 raft_consensus.cc:2241] T 824229183e024feaa975c7f0387c9c1a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:29.652545 2113 raft_consensus.cc:2270] T 824229183e024feaa975c7f0387c9c1a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:29.654608 2113 ts_tablet_manager.cc:1905] T 824229183e024feaa975c7f0387c9c1a P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:29.656859 2118 ts_tablet_manager.cc:1918] T becbaae4fd7e44faa679e9479465a7c4 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:29.528834 1151 raft_consensus.cc:738] T 0dbdcb76eeba4fefa56e7835c23c0316 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:29.662779 2118 log.cc:1199] T becbaae4fd7e44faa679e9479465a7c4 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/becbaae4fd7e44faa679e9479465a7c4
I20250901 01:52:29.663329 2118 ts_tablet_manager.cc:1939] T becbaae4fd7e44faa679e9479465a7c4 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
W20250901 01:52:29.661855 992 consensus_peers.cc:489] T eaa521c52a2f4408b59770bd2a05907c P fd27ab6665304aaea53adbd102dd4f58 -> Peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Couldn't send request to peer 00c282f779c1477d920f24dc0dc16628. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eaa521c52a2f4408b59770bd2a05907c. This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:29.663494 1151 consensus_queue.cc:260] T 0dbdcb76eeba4fefa56e7835c23c0316 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:29.663766 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 109) took 3681 ms. Trace:
I20250901 01:52:29.663900 2118 rpcz_store.cc:276] 0901 01:52:25.982424 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.983375 (+ 951us) service_pool.cc:225] Handling call
0901 01:52:29.638034 (+3654659us) ts_tablet_manager.cc:1145] Deleting tablet becbaae4fd7e44faa679e9479465a7c4
0901 01:52:29.638048 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:29.647880 (+ 9832us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:29.656739 (+ 8859us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:29.663745 (+ 7006us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3654519}
W20250901 01:52:29.664261 992 consensus_peers.cc:489] T b2e8b2cd10154de08ddd30c214592a4a P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250901 01:52:29.664364 992 consensus_peers.cc:489] T 500e3c33598b45a4afa983da7cb0845a P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250901 01:52:29.664469 992 consensus_peers.cc:489] T 6789505328b1409ea288e8b4eaf94935 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:29.664495 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet becbaae4fd7e44faa679e9479465a7c4 (table default.Table_0e73f80424cd48408f18c2eff1411706 [id=47c13af47c944d0fa79dee115033265d]) successfully deleted
I20250901 01:52:29.664964 2118 tablet_replica.cc:331] T 31614db4c53a4f3c96fd190d326ce601 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:29.665287 2118 raft_consensus.cc:2241] T 31614db4c53a4f3c96fd190d326ce601 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:29.665517 2118 raft_consensus.cc:2270] T 31614db4c53a4f3c96fd190d326ce601 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:29.665729 1151 ts_tablet_manager.cc:1428] T 0dbdcb76eeba4fefa56e7835c23c0316 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.158s user 0.024s sys 0.000s
I20250901 01:52:29.666065 2118 ts_tablet_manager.cc:1905] T 31614db4c53a4f3c96fd190d326ce601 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:29.668699 1151 tablet_bootstrap.cc:492] T 56475ba491a24301a534304811f30140 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:29.670872 1151 tablet_bootstrap.cc:654] T 56475ba491a24301a534304811f30140 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:29.674906 2118 ts_tablet_manager.cc:1918] T 31614db4c53a4f3c96fd190d326ce601 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:29.676182 2113 ts_tablet_manager.cc:1918] T 824229183e024feaa975c7f0387c9c1a P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:29.676398 2113 log.cc:1199] T 824229183e024feaa975c7f0387c9c1a P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/824229183e024feaa975c7f0387c9c1a
I20250901 01:52:29.676898 2113 ts_tablet_manager.cc:1939] T 824229183e024feaa975c7f0387c9c1a P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:29.677438 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 111) took 4549 ms. Trace:
I20250901 01:52:29.677605 2113 rpcz_store.cc:276] 0901 01:52:25.127515 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.127554 (+ 39us) service_pool.cc:225] Handling call
0901 01:52:29.649850 (+4522296us) ts_tablet_manager.cc:1145] Deleting tablet 824229183e024feaa975c7f0387c9c1a
0901 01:52:29.649861 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:29.657623 (+ 7762us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:29.676074 (+ 18451us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:29.677417 (+ 1343us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4522096}
I20250901 01:52:29.678231 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 112) took 4550 ms. Trace:
I20250901 01:52:29.678349 2113 rpcz_store.cc:276] 0901 01:52:25.127866 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.127908 (+ 42us) service_pool.cc:225] Handling call
0901 01:52:29.678140 (+4550232us) ts_tablet_manager.cc:1145] Deleting tablet 2ff914352c7c43de800f3d54ecf63072
0901 01:52:29.678148 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:29.678216 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4550096}
I20250901 01:52:29.678608 2118 log.cc:1199] T 31614db4c53a4f3c96fd190d326ce601 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/31614db4c53a4f3c96fd190d326ce601
I20250901 01:52:29.678572 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 824229183e024feaa975c7f0387c9c1a (table default.Table_5e65c5f068b84995a2d7a76fa65dda0f [id=b9d81ddd38fe47838877b010be2b40d0]) successfully deleted
I20250901 01:52:29.678776 2113 tablet_replica.cc:331] T 40d74d7b5a3b4d1dae09eebd9e07a0c4 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
W20250901 01:52:29.678892 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 2ff914352c7c43de800f3d54ecf63072 because the tablet was not found. No further retry: Not found: Tablet not found: 2ff914352c7c43de800f3d54ecf63072
I20250901 01:52:29.678999 2118 ts_tablet_manager.cc:1939] T 31614db4c53a4f3c96fd190d326ce601 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:29.679453 2113 raft_consensus.cc:2241] T 40d74d7b5a3b4d1dae09eebd9e07a0c4 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:29.679680 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 110) took 3697 ms. Trace:
I20250901 01:52:29.679769 2113 raft_consensus.cc:2270] T 40d74d7b5a3b4d1dae09eebd9e07a0c4 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:29.679813 2118 rpcz_store.cc:276] 0901 01:52:25.982515 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.983538 (+ 1023us) service_pool.cc:225] Handling call
0901 01:52:29.664901 (+3681363us) ts_tablet_manager.cc:1145] Deleting tablet 31614db4c53a4f3c96fd190d326ce601
0901 01:52:29.664919 (+ 18us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:29.669083 (+ 4164us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:29.674809 (+ 5726us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:29.679660 (+ 4851us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3681232}
I20250901 01:52:29.680321 2113 ts_tablet_manager.cc:1905] T 40d74d7b5a3b4d1dae09eebd9e07a0c4 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:29.680341 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 31614db4c53a4f3c96fd190d326ce601 (table default.Table_e59cac5d3f0a4dd6b9846e7449b60005 [id=fb3be765c8e84e8cb4823153f82b374c]) successfully deleted
I20250901 01:52:29.680706 2118 tablet_replica.cc:331] T 2599f98abfdc4871b573edfc2ad97d22 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:29.680943 2118 raft_consensus.cc:2241] T 2599f98abfdc4871b573edfc2ad97d22 P 00c282f779c1477d920f24dc0dc16628 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:29.681289 2118 raft_consensus.cc:2270] T 2599f98abfdc4871b573edfc2ad97d22 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:29.682024 2118 ts_tablet_manager.cc:1905] T 2599f98abfdc4871b573edfc2ad97d22 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:29.682679 2221 raft_consensus.cc:738] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3add7edb97a40ef9f7c6bc011d13227, State: Initialized, Role: FOLLOWER
I20250901 01:52:29.682888 1151 tablet_bootstrap.cc:492] T 56475ba491a24301a534304811f30140 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:29.683048 1151 ts_tablet_manager.cc:1397] T 56475ba491a24301a534304811f30140 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
W20250901 01:52:29.683090 1465 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 5f5396045979450193d461ba1334a5c9 (table default.Table_2b946a63bd574ac0971fa3d3b9c9a6a4 [id=9c3930e9fcfe4b49ba4bea67a2b0ad07], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 5f5396045979450193d461ba1334a5c9 consensus state has no leader
I20250901 01:52:29.683821 1151 raft_consensus.cc:357] T 56475ba491a24301a534304811f30140 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:29.684083 1151 raft_consensus.cc:383] T 56475ba491a24301a534304811f30140 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:29.684180 1151 raft_consensus.cc:738] T 56475ba491a24301a534304811f30140 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
W20250901 01:52:29.683035 859 leader_election.cc:343] T abf6e3be60f74872b2308dc95bf9cd37 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:29.684543 1151 consensus_queue.cc:260] T 56475ba491a24301a534304811f30140 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:29.684630 859 leader_election.cc:304] T abf6e3be60f74872b2308dc95bf9cd37 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:29.685016 859 leader_election.cc:343] T 4fca1435d82447f289fda4035aa2ed89 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:29.685106 1151 ts_tablet_manager.cc:1428] T 56475ba491a24301a534304811f30140 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250901 01:52:29.685125 859 leader_election.cc:304] T 4fca1435d82447f289fda4035aa2ed89 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:29.685389 1151 tablet_bootstrap.cc:492] T 33f637f6a8134215bb1995ec38440073 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
W20250901 01:52:29.685482 859 leader_election.cc:343] T d0b7c84ec9394a20ad77add3808e555a P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:29.685559 859 leader_election.cc:304] T d0b7c84ec9394a20ad77add3808e555a P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:29.685873 859 leader_election.cc:304] T 14b466b2a5a249aa841f1b8b06301edc P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:29.686311 2323 raft_consensus.cc:2747] T abf6e3be60f74872b2308dc95bf9cd37 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:29.686467 2323 raft_consensus.cc:2747] T 4fca1435d82447f289fda4035aa2ed89 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:29.686596 2323 raft_consensus.cc:2747] T d0b7c84ec9394a20ad77add3808e555a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:29.686695 2323 raft_consensus.cc:2802] T 14b466b2a5a249aa841f1b8b06301edc P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:29.686789 2323 raft_consensus.cc:695] T 14b466b2a5a249aa841f1b8b06301edc P 00c282f779c1477d920f24dc0dc16628 [term 1 LEADER]: Becoming Leader. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Running, Role: LEADER
I20250901 01:52:29.686997 2323 consensus_queue.cc:237] T 14b466b2a5a249aa841f1b8b06301edc P 00c282f779c1477d920f24dc0dc16628 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:29.687168 1151 tablet_bootstrap.cc:654] T 33f637f6a8134215bb1995ec38440073 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:29.697741 2118 ts_tablet_manager.cc:1918] T 2599f98abfdc4871b573edfc2ad97d22 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:29.697948 2118 log.cc:1199] T 2599f98abfdc4871b573edfc2ad97d22 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/2599f98abfdc4871b573edfc2ad97d22
I20250901 01:52:29.698371 2118 ts_tablet_manager.cc:1939] T 2599f98abfdc4871b573edfc2ad97d22 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:29.698750 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 104) took 3718 ms. Trace:
I20250901 01:52:29.698889 2118 rpcz_store.cc:276] 0901 01:52:25.980747 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.984634 (+ 3887us) service_pool.cc:225] Handling call
0901 01:52:29.680656 (+3696022us) ts_tablet_manager.cc:1145] Deleting tablet 2599f98abfdc4871b573edfc2ad97d22
0901 01:52:29.680668 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:29.687732 (+ 7064us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:29.697618 (+ 9886us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:29.698733 (+ 1115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3695851}
I20250901 01:52:29.699451 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 2599f98abfdc4871b573edfc2ad97d22 (table default.Table_0e63d21acfcb4c028c9ea6406b6a04e4 [id=81323e6fbe424655b8e185f52657fadc]) successfully deleted
I20250901 01:52:29.700027 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 106) took 3717 ms. Trace:
I20250901 01:52:29.700146 2118 rpcz_store.cc:276] 0901 01:52:25.982128 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.984849 (+ 2721us) service_pool.cc:225] Handling call
0901 01:52:29.699932 (+3715083us) ts_tablet_manager.cc:1145] Deleting tablet 6741c56499b44f648fb032c8cf50205b
0901 01:52:29.699941 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:29.700009 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3714970}
W20250901 01:52:29.700708 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 6741c56499b44f648fb032c8cf50205b because the tablet was not found. No further retry: Not found: Tablet not found: 6741c56499b44f648fb032c8cf50205b
I20250901 01:52:29.701049 2118 tablet_replica.cc:331] T 879fa66e55544118aafe504587393c66 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:29.700945 2221 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 632, Last appended: 2.633, Last appended by leader: 633, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3add7edb97a40ef9f7c6bc011d13227" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 43871 } } peers { permanent_uuid: "d2ae2cb6af0f4b3a8f47ab1cdd6567d3" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42375 } } peers { permanent_uuid: "7ce2288f784044eea010664e3def133f" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 44231 } }
I20250901 01:52:29.701289 2118 raft_consensus.cc:2241] T 879fa66e55544118aafe504587393c66 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:29.701519 2118 raft_consensus.cc:2270] T 879fa66e55544118aafe504587393c66 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:29.701678 2113 ts_tablet_manager.cc:1918] T 40d74d7b5a3b4d1dae09eebd9e07a0c4 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:29.702036 2113 log.cc:1199] T 40d74d7b5a3b4d1dae09eebd9e07a0c4 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/40d74d7b5a3b4d1dae09eebd9e07a0c4
I20250901 01:52:29.702230 2118 ts_tablet_manager.cc:1905] T 879fa66e55544118aafe504587393c66 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:29.703284 1151 tablet_bootstrap.cc:492] T 33f637f6a8134215bb1995ec38440073 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:29.703464 1151 ts_tablet_manager.cc:1397] T 33f637f6a8134215bb1995ec38440073 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.001s
I20250901 01:52:29.704162 1151 raft_consensus.cc:357] T 33f637f6a8134215bb1995ec38440073 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:29.704376 1151 raft_consensus.cc:383] T 33f637f6a8134215bb1995ec38440073 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:29.704562 1151 raft_consensus.cc:738] T 33f637f6a8134215bb1995ec38440073 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:29.704838 1151 consensus_queue.cc:260] T 33f637f6a8134215bb1995ec38440073 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:29.705549 1151 ts_tablet_manager.cc:1428] T 33f637f6a8134215bb1995ec38440073 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250901 01:52:29.705890 1151 tablet_bootstrap.cc:492] T 1a60c10e42104502ac2d3b80b8126ceb P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:29.707801 1484 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 81ce448b148447ddbc41875d863e0518 (table default.Table_6e461d9cb0ae4dd095975bbc21533794 [id=49673fa73f8c436e9b8026efc4f3cbbc]). Expected version 1 got 0
I20250901 01:52:29.708523 2113 ts_tablet_manager.cc:1939] T 40d74d7b5a3b4d1dae09eebd9e07a0c4 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:29.709390 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 113) took 4581 ms. Trace:
I20250901 01:52:29.710227 2118 ts_tablet_manager.cc:1918] T 879fa66e55544118aafe504587393c66 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:29.711386 2113 rpcz_store.cc:276] 0901 01:52:25.128161 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.128219 (+ 58us) service_pool.cc:225] Handling call
0901 01:52:29.678733 (+4550514us) ts_tablet_manager.cc:1145] Deleting tablet 40d74d7b5a3b4d1dae09eebd9e07a0c4
0901 01:52:29.678741 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:29.697319 (+ 18578us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:29.700354 (+ 3035us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:29.709372 (+ 9018us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4550297}
I20250901 01:52:29.711325 1151 tablet_bootstrap.cc:654] T 1a60c10e42104502ac2d3b80b8126ceb P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:29.713020 2113 tablet_replica.cc:331] T c9a4c80bec1d41458974a0c449aecaf7 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:29.714206 2552 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3add7edb97a40ef9f7c6bc011d13227" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 43871 } } peers { permanent_uuid: "d2ae2cb6af0f4b3a8f47ab1cdd6567d3" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42375 } } peers { permanent_uuid: "7ce2288f784044eea010664e3def133f" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 44231 } } }
I20250901 01:52:29.714701 2552 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227 [sys.catalog]: This master's current role is: FOLLOWER
I20250901 01:52:29.716470 2113 raft_consensus.cc:2241] T c9a4c80bec1d41458974a0c449aecaf7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:29.716724 2113 raft_consensus.cc:2270] T c9a4c80bec1d41458974a0c449aecaf7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:29.716940 2221 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227 [sys.catalog]: configured and running, proceeding with master startup.
I20250901 01:52:29.717221 2118 log.cc:1199] T 879fa66e55544118aafe504587393c66 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/879fa66e55544118aafe504587393c66
I20250901 01:52:29.717675 2118 ts_tablet_manager.cc:1939] T 879fa66e55544118aafe504587393c66 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:29.717767 2113 ts_tablet_manager.cc:1905] T c9a4c80bec1d41458974a0c449aecaf7 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:29.718444 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 111) took 3735 ms. Trace:
I20250901 01:52:29.718411 2146 raft_consensus.cc:3058] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227 [term 2 FOLLOWER]: Advancing to term 4
I20250901 01:52:29.720510 918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9b293bf6a17453b9ea9892a6970d0b0" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:29.724102 919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff013ea889d44cba8797834e4611b489" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:29.724393 2118 rpcz_store.cc:276] 0901 01:52:25.982606 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.984980 (+ 2374us) service_pool.cc:225] Handling call
0901 01:52:29.701004 (+3716024us) ts_tablet_manager.cc:1145] Deleting tablet 879fa66e55544118aafe504587393c66
0901 01:52:29.701014 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:29.707260 (+ 6246us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:29.710133 (+ 2873us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:29.718424 (+ 8291us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3715893}
I20250901 01:52:29.724495 918 raft_consensus.cc:2466] T f9b293bf6a17453b9ea9892a6970d0b0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:29.724540 919 raft_consensus.cc:3058] T ff013ea889d44cba8797834e4611b489 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:29.725742 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 112) took 3743 ms. Trace:
I20250901 01:52:29.725854 2529 raft_consensus.cc:491] T d10c193452954c198bba686169b8025b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:29.725937 2118 rpcz_store.cc:276] 0901 01:52:25.982680 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.985130 (+ 2450us) service_pool.cc:225] Handling call
0901 01:52:29.725648 (+3740518us) ts_tablet_manager.cc:1145] Deleting tablet 4902062977ea42149e3466e999117322
0901 01:52:29.725659 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:29.725725 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3740401}
I20250901 01:52:29.726013 2529 raft_consensus.cc:513] T d10c193452954c198bba686169b8025b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:29.726311 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 113) took 3743 ms. Trace:
I20250901 01:52:29.726374 2118 rpcz_store.cc:276] 0901 01:52:25.982794 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.985263 (+ 2469us) service_pool.cc:225] Handling call
0901 01:52:29.726239 (+3740976us) ts_tablet_manager.cc:1145] Deleting tablet b81a12b38ac64efabd44aeb192d181a0
0901 01:52:29.726247 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:29.726299 (+ 52us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3740826}
I20250901 01:52:29.726711 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 114) took 3737 ms. Trace:
I20250901 01:52:29.726707 2529 leader_election.cc:290] T d10c193452954c198bba686169b8025b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:29.726771 2118 rpcz_store.cc:276] 0901 01:52:25.989393 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.989451 (+ 58us) service_pool.cc:225] Handling call
0901 01:52:29.726648 (+3737197us) ts_tablet_manager.cc:1145] Deleting tablet 5e2472ee5d6e4d3bbf1f24027ba531ca
0901 01:52:29.726655 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:29.726699 (+ 44us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3737031}
I20250901 01:52:29.727114 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 115) took 3731 ms. Trace:
I20250901 01:52:29.727173 2118 rpcz_store.cc:276] 0901 01:52:25.996013 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.996066 (+ 53us) service_pool.cc:225] Handling call
0901 01:52:29.727029 (+3730963us) ts_tablet_manager.cc:1145] Deleting tablet 438ae684af234c868a419fbab43b0b98
0901 01:52:29.727037 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:29.727102 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3730805}
I20250901 01:52:29.727744 1151 tablet_bootstrap.cc:492] T 1a60c10e42104502ac2d3b80b8126ceb P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:29.727712 1058 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4009963cfe42d6ba164fc828897f09" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:29.727871 1151 ts_tablet_manager.cc:1397] T 1a60c10e42104502ac2d3b80b8126ceb P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.000s
I20250901 01:52:29.727993 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 117) took 3724 ms. Trace:
I20250901 01:52:29.728078 2118 rpcz_store.cc:276] 0901 01:52:26.003140 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.003194 (+ 54us) service_pool.cc:225] Handling call
0901 01:52:29.727446 (+3724252us) ts_tablet_manager.cc:1145] Deleting tablet 88ceba7f48b04d7885cc0164668700a5
0901 01:52:29.727454 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:29.727975 (+ 521us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3724083}
I20250901 01:52:29.728204 919 raft_consensus.cc:2466] T ff013ea889d44cba8797834e4611b489 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:29.728469 2118 tablet_replica.cc:331] T 617cc33d0d23439cb5f1e0c31bf11281 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:29.728582 1151 raft_consensus.cc:357] T 1a60c10e42104502ac2d3b80b8126ceb P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:29.728667 2118 raft_consensus.cc:2241] T 617cc33d0d23439cb5f1e0c31bf11281 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:29.728767 1151 raft_consensus.cc:383] T 1a60c10e42104502ac2d3b80b8126ceb P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:29.728804 2118 raft_consensus.cc:2270] T 617cc33d0d23439cb5f1e0c31bf11281 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:29.728854 1151 raft_consensus.cc:738] T 1a60c10e42104502ac2d3b80b8126ceb P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:29.711987 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 40d74d7b5a3b4d1dae09eebd9e07a0c4 (table default.Table_1adc70cccd2f4cb3a79df96861cb4181 [id=24fffc7ebf5a47bdb37211910301acf7]) successfully deleted
I20250901 01:52:29.725389 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 879fa66e55544118aafe504587393c66 (table default.Table_b362441111a241e19cc1a09bdeefea49 [id=f1d75be759db40ebb72ff32560f1d038]) successfully deleted
I20250901 01:52:29.729135 1151 consensus_queue.cc:260] T 1a60c10e42104502ac2d3b80b8126ceb P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:29.729314 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 4902062977ea42149e3466e999117322 because the tablet was not found. No further retry: Not found: Tablet not found: 4902062977ea42149e3466e999117322
I20250901 01:52:29.729460 2118 ts_tablet_manager.cc:1905] T 617cc33d0d23439cb5f1e0c31bf11281 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:29.729588 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet b81a12b38ac64efabd44aeb192d181a0 because the tablet was not found. No further retry: Not found: Tablet not found: b81a12b38ac64efabd44aeb192d181a0
I20250901 01:52:29.729684 1151 ts_tablet_manager.cc:1428] T 1a60c10e42104502ac2d3b80b8126ceb P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20250901 01:52:29.729802 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 5e2472ee5d6e4d3bbf1f24027ba531ca because the tablet was not found. No further retry: Not found: Tablet not found: 5e2472ee5d6e4d3bbf1f24027ba531ca
I20250901 01:52:29.729996 1151 tablet_bootstrap.cc:492] T 48a211e174054866b030cdfb0cf947f4 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
W20250901 01:52:29.730031 1472 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 438ae684af234c868a419fbab43b0b98 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 438ae684af234c868a419fbab43b0b98 already in progress: creating tablet
W20250901 01:52:29.730243 1472 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 88ceba7f48b04d7885cc0164668700a5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 88ceba7f48b04d7885cc0164668700a5 already in progress: creating tablet
I20250901 01:52:29.730962 1484 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet d1f2d98712ff4ea6aec665512de068bd (table default.Table_82d91a1ba2eb4265896824028dec8d3a [id=3f7624d2901445e39155c12fa68e34e9]). Expected version 1 got 0
I20250901 01:52:29.731285 1484 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet ed1b41340ce742a88c3e4fd85cd25ca6 (table default.Table_ce2f3b515c8f4c1bb557c5ed84b48dba [id=0baa4993b72e4a8d94ee3059c9d2db6c]). Expected version 2 got 0
I20250901 01:52:29.731906 1151 tablet_bootstrap.cc:654] T 48a211e174054866b030cdfb0cf947f4 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:29.732044 1484 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet e87d1281f35648b0b115b67c95e244e7 (table default.Table_f840789a58984799bd8057128f6ea439 [id=4453959e273c473fadd737f56f7db3dd]). Expected version 1 got 0
I20250901 01:52:29.735706 2113 ts_tablet_manager.cc:1918] T c9a4c80bec1d41458974a0c449aecaf7 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:29.737310 2113 log.cc:1199] T c9a4c80bec1d41458974a0c449aecaf7 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/c9a4c80bec1d41458974a0c449aecaf7
I20250901 01:52:29.737747 2113 ts_tablet_manager.cc:1939] T c9a4c80bec1d41458974a0c449aecaf7 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:29.738526 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 114) took 4609 ms. Trace:
I20250901 01:52:29.743376 2113 rpcz_store.cc:276] 0901 01:52:25.128573 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.128617 (+ 44us) service_pool.cc:225] Handling call
0901 01:52:29.712954 (+4584337us) ts_tablet_manager.cc:1145] Deleting tablet c9a4c80bec1d41458974a0c449aecaf7
0901 01:52:29.712965 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:29.720992 (+ 8027us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:29.735607 (+ 14615us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:29.738496 (+ 2889us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4584149}
I20250901 01:52:29.744001 2113 tablet_replica.cc:331] T 121803e79ab64ec0b948b448646b9d0a P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:29.744233 2113 raft_consensus.cc:2241] T 121803e79ab64ec0b948b448646b9d0a P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:29.744517 2113 raft_consensus.cc:2270] T 121803e79ab64ec0b948b448646b9d0a P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:29.745206 2113 ts_tablet_manager.cc:1905] T 121803e79ab64ec0b948b448646b9d0a P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:29.747974 992 leader_election.cc:336] T ff013ea889d44cba8797834e4611b489 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:29.748211 992 consensus_peers.cc:489] T d58f69fad01f403eab74a47d3695ed83 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250901 01:52:29.748353 992 consensus_peers.cc:489] T 7cc1efcb6bff4aba8015d2d3b7f4c923 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250901 01:52:29.743950 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet c9a4c80bec1d41458974a0c449aecaf7 (table default.Table_117c6c43b552476dbb8b7b5ccdce47b7 [id=a4c947d85200437395c66bb4f7e607af]) successfully deleted
I20250901 01:52:29.748903 1484 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet ea846f88b3534beebd0001b7f17b5ca9 (table default.Table_dd92add170f94aaf966f77d81e2c3883 [id=85b64046d2fa4f1d871ab43e5c800fb8]). Expected version 1 got 0
I20250901 01:52:29.749593 1484 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 616ff34e8d314e0aa0e11ce60be11789 (table default.Table_0f4b4094d2974b7591875375eda6bed0 [id=5ee13d46e06a4fa59b1070547573e523]). Expected version 2 got 1
W20250901 01:52:29.749817 992 leader_election.cc:336] T 12c04e0c1f804179ae23509cdb24d395 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:29.749873 2118 ts_tablet_manager.cc:1918] T 617cc33d0d23439cb5f1e0c31bf11281 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:29.749965 992 leader_election.cc:304] T 12c04e0c1f804179ae23509cdb24d395 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:29.750047 2118 log.cc:1199] T 617cc33d0d23439cb5f1e0c31bf11281 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/617cc33d0d23439cb5f1e0c31bf11281
I20250901 01:52:29.750339 2118 ts_tablet_manager.cc:1939] T 617cc33d0d23439cb5f1e0c31bf11281 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:29.750324 1761 raft_consensus.cc:2747] T 12c04e0c1f804179ae23509cdb24d395 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:29.750931 1484 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet c6f7c525f59b49938b697de98d4b5835 (table default.Table_db61e055dbd7450898e82c4c0c4763be [id=aabca67f6f754820977ac9c4c20f3e4f]). Expected version 1 got 0
W20250901 01:52:29.751142 992 consensus_peers.cc:489] T 7b16ca1daf394a6da65a4af8da82ec33 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250901 01:52:29.751555 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 118) took 3747 ms. Trace:
I20250901 01:52:29.750626 2323 raft_consensus.cc:491] T 5afae316ff164413be0d25d4bc0f58dc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:29.751420 1151 tablet_bootstrap.cc:492] T 48a211e174054866b030cdfb0cf947f4 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
W20250901 01:52:29.751881 992 consensus_peers.cc:489] T 7954393c8c1c4457ab8bb723c03a9c8b P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:29.751802 2323 raft_consensus.cc:513] T 5afae316ff164413be0d25d4bc0f58dc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:29.752279 1484 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet ad45c90ba0da43c9860745e534664a98 (table default.Table_1cd3f3fb6b244b3f8796bfca9f7622c8 [id=6fd739b0e725424ca844dbb652ea8028]). Expected version 2 got 0
I20250901 01:52:29.752192 1151 ts_tablet_manager.cc:1397] T 48a211e174054866b030cdfb0cf947f4 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.000s
I20250901 01:52:29.752595 2323 leader_election.cc:290] T 5afae316ff164413be0d25d4bc0f58dc P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:29.752810 992 consensus_peers.cc:489] T 72ba689f9230411995dc96e2b45601df P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250901 01:52:29.714470 2549 raft_consensus.cc:491] T 2dacb608897d47d28512df87dc320927 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:29.752708 2118 rpcz_store.cc:276] 0901 01:52:26.003597 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.003644 (+ 47us) service_pool.cc:225] Handling call
0901 01:52:29.728403 (+3724759us) ts_tablet_manager.cc:1145] Deleting tablet 617cc33d0d23439cb5f1e0c31bf11281
0901 01:52:29.728411 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:29.732739 (+ 4328us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:29.749760 (+ 17021us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:29.751522 (+ 1762us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3724597}
I20250901 01:52:29.753369 2118 tablet_replica.cc:331] T 246b33338de647689d7404c70676e76f P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:29.753154 2549 raft_consensus.cc:513] T 2dacb608897d47d28512df87dc320927 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:29.753839 2118 raft_consensus.cc:2241] T 246b33338de647689d7404c70676e76f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250901 01:52:29.753830 858 leader_election.cc:336] T 5afae316ff164413be0d25d4bc0f58dc P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:29.754045 2118 raft_consensus.cc:2270] T 246b33338de647689d7404c70676e76f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:29.754838 2118 ts_tablet_manager.cc:1905] T 246b33338de647689d7404c70676e76f P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:29.754870 2323 raft_consensus.cc:491] T 764bfeb2a7d14c40815b5ec1b07bdd62 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:29.754980 2323 raft_consensus.cc:513] T 764bfeb2a7d14c40815b5ec1b07bdd62 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:29.755450 2323 leader_election.cc:290] T 764bfeb2a7d14c40815b5ec1b07bdd62 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:29.755633 858 leader_election.cc:336] T 2dacb608897d47d28512df87dc320927 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:29.755409 2549 leader_election.cc:290] T 2dacb608897d47d28512df87dc320927 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:29.753463 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 617cc33d0d23439cb5f1e0c31bf11281 (table default.Table_ce1fd26ac65a4ed085d023e10a849e5c [id=fb088d31bba64f04871fbfd355d6263b]) successfully deleted
I20250901 01:52:29.756322 1484 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 8ca5cf4e99014c0d84a3acad4ef713af (table default.Table_ed6fd284be6f4ae3a008e21e7ce17930 [id=303e76c918bc4c39aa47a4d24849a358]). Expected version 1 got 0
I20250901 01:52:29.757133 1151 raft_consensus.cc:357] T 48a211e174054866b030cdfb0cf947f4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:29.757372 1151 raft_consensus.cc:383] T 48a211e174054866b030cdfb0cf947f4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:29.757468 1151 raft_consensus.cc:738] T 48a211e174054866b030cdfb0cf947f4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
W20250901 01:52:29.757602 858 leader_election.cc:336] T 764bfeb2a7d14c40815b5ec1b07bdd62 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:29.757701 1151 consensus_queue.cc:260] T 48a211e174054866b030cdfb0cf947f4 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:29.758292 1151 ts_tablet_manager.cc:1428] T 48a211e174054866b030cdfb0cf947f4 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250901 01:52:29.758628 1151 tablet_bootstrap.cc:492] T 4902062977ea42149e3466e999117322 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:29.760255 1151 tablet_bootstrap.cc:654] T 4902062977ea42149e3466e999117322 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:29.760782 1484 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 61ddcf86c9f34726939cc116434f0822 (table default.Table_63d0f2ead9d946d39ce40162ebd769d6 [id=f9dcf4937b1143248bfa9d88053fce17]). Expected version 3 got 0
I20250901 01:52:29.761348 1150 tablet_bootstrap.cc:492] T e1c8fe96a6ba4edba4f09c00f7add75e P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:29.761524 1150 ts_tablet_manager.cc:1397] T e1c8fe96a6ba4edba4f09c00f7add75e P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.251s user 0.033s sys 0.029s
I20250901 01:52:29.762216 2118 ts_tablet_manager.cc:1918] T 246b33338de647689d7404c70676e76f P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:29.763520 1484 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 5f5396045979450193d461ba1334a5c9 (table default.Table_2b946a63bd574ac0971fa3d3b9c9a6a4 [id=9c3930e9fcfe4b49ba4bea67a2b0ad07]). Expected version 2 got 0
I20250901 01:52:29.765064 1484 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 2973b59bb1df41d18d9a5b6a29ae336f (table default.Table_7790af96855e45c0a316ba3a4a87c9ed [id=746845bd84c6444b8a1e5683eaba4a68]). Expected version 1 got 0
I20250901 01:52:29.768393 1484 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet d000ba05e12843b4925e622ab55471c0 (table default.Table_e0902dfbdd984f85b3fdac8d283731f7 [id=b43a92ed46164d199b445287ef039bc6]). Expected version 1 got 0
I20250901 01:52:29.764155 2113 ts_tablet_manager.cc:1918] T 121803e79ab64ec0b948b448646b9d0a P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:29.768847 2113 log.cc:1199] T 121803e79ab64ec0b948b448646b9d0a P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/121803e79ab64ec0b948b448646b9d0a
I20250901 01:52:29.769328 2113 ts_tablet_manager.cc:1939] T 121803e79ab64ec0b948b448646b9d0a P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:29.762338 1150 raft_consensus.cc:357] T e1c8fe96a6ba4edba4f09c00f7add75e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:29.770116 2118 log.cc:1199] T 246b33338de647689d7404c70676e76f P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/246b33338de647689d7404c70676e76f
I20250901 01:52:29.770167 1150 raft_consensus.cc:383] T e1c8fe96a6ba4edba4f09c00f7add75e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:29.770270 1150 raft_consensus.cc:738] T e1c8fe96a6ba4edba4f09c00f7add75e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:29.770469 2118 ts_tablet_manager.cc:1939] T 246b33338de647689d7404c70676e76f P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:29.770578 1150 consensus_queue.cc:260] T e1c8fe96a6ba4edba4f09c00f7add75e P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:29.770843 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 119) took 3758 ms. Trace:
I20250901 01:52:29.770949 2118 rpcz_store.cc:276] 0901 01:52:26.011935 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.011986 (+ 51us) service_pool.cc:225] Handling call
0901 01:52:29.753322 (+3741336us) ts_tablet_manager.cc:1145] Deleting tablet 246b33338de647689d7404c70676e76f
0901 01:52:29.753333 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:29.758972 (+ 5639us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:29.762141 (+ 3169us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:29.770828 (+ 8687us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3741186}
I20250901 01:52:29.771158 1150 ts_tablet_manager.cc:1428] T e1c8fe96a6ba4edba4f09c00f7add75e P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20250901 01:52:29.771514 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 120) took 3758 ms. Trace:
I20250901 01:52:29.771646 2118 rpcz_store.cc:276] 0901 01:52:26.013183 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.013228 (+ 45us) service_pool.cc:225] Handling call
0901 01:52:29.771401 (+3758173us) ts_tablet_manager.cc:1145] Deleting tablet b11e8f77243e4764b96548d1132b8e5a
0901 01:52:29.771410 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:29.771500 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3758034}
I20250901 01:52:29.773232 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 115) took 4644 ms. Trace:
I20250901 01:52:29.773247 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 121) took 3757 ms. Trace:
I20250901 01:52:29.773324 2118 rpcz_store.cc:276] 0901 01:52:26.015931 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.016121 (+ 190us) service_pool.cc:225] Handling call
0901 01:52:29.772019 (+3755898us) ts_tablet_manager.cc:1145] Deleting tablet 4410da9e95454ecf95faf80221b0a71c
0901 01:52:29.773023 (+ 1004us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:29.773230 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3755707}
I20250901 01:52:29.773802 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 122) took 3757 ms. Trace:
I20250901 01:52:29.773921 2118 rpcz_store.cc:276] 0901 01:52:26.016075 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.016343 (+ 268us) service_pool.cc:225] Handling call
0901 01:52:29.773714 (+3757371us) ts_tablet_manager.cc:1145] Deleting tablet 3c12d6a70ac24dce82e851782e6fe5aa
0901 01:52:29.773722 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:29.773789 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3757213}
I20250901 01:52:29.774307 2118 tablet_replica.cc:331] T f62d96c37cca4143a4b098b7c69979e6 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:29.774540 2118 raft_consensus.cc:2241] T f62d96c37cca4143a4b098b7c69979e6 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:29.774606 2113 rpcz_store.cc:276] 0901 01:52:25.128924 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.128962 (+ 38us) service_pool.cc:225] Handling call
0901 01:52:29.743950 (+4614988us) ts_tablet_manager.cc:1145] Deleting tablet 121803e79ab64ec0b948b448646b9d0a
0901 01:52:29.743960 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:29.761203 (+ 17243us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:29.764071 (+ 2868us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:29.773213 (+ 9142us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4614772}
I20250901 01:52:29.774487 1151 tablet_bootstrap.cc:492] T 4902062977ea42149e3466e999117322 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:29.775015 2118 raft_consensus.cc:2270] T f62d96c37cca4143a4b098b7c69979e6 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:29.775133 1151 ts_tablet_manager.cc:1397] T 4902062977ea42149e3466e999117322 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.003s
I20250901 01:52:29.775290 1150 tablet_bootstrap.cc:492] T b2b172757e50404aae78e818921a7d9d P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:29.775609 2118 ts_tablet_manager.cc:1905] T f62d96c37cca4143a4b098b7c69979e6 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:29.776010 1151 raft_consensus.cc:357] T 4902062977ea42149e3466e999117322 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:29.776280 1151 raft_consensus.cc:383] T 4902062977ea42149e3466e999117322 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:29.776388 1151 raft_consensus.cc:738] T 4902062977ea42149e3466e999117322 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:29.776958 1150 tablet_bootstrap.cc:654] T b2b172757e50404aae78e818921a7d9d P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:29.776701 1151 consensus_queue.cc:260] T 4902062977ea42149e3466e999117322 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:29.779858 1151 ts_tablet_manager.cc:1428] T 4902062977ea42149e3466e999117322 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20250901 01:52:29.780166 1151 tablet_bootstrap.cc:492] T 1495de70f0844f82a0eea99c059f076b P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:29.781834 1151 tablet_bootstrap.cc:654] T 1495de70f0844f82a0eea99c059f076b P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:29.782326 2113 tablet_replica.cc:331] T 82d163681fda47f38d79863bd28f576d P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:29.783322 2113 raft_consensus.cc:2241] T 82d163681fda47f38d79863bd28f576d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:29.783850 2113 raft_consensus.cc:2270] T 82d163681fda47f38d79863bd28f576d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:29.784369 2113 ts_tablet_manager.cc:1905] T 82d163681fda47f38d79863bd28f576d P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:29.779042 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d10c193452954c198bba686169b8025b" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:29.781715 2118 ts_tablet_manager.cc:1918] T f62d96c37cca4143a4b098b7c69979e6 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250901 01:52:29.792575 2118 log.cc:1199] T f62d96c37cca4143a4b098b7c69979e6 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/f62d96c37cca4143a4b098b7c69979e6
I20250901 01:52:29.792575 916 raft_consensus.cc:2466] T d10c193452954c198bba686169b8025b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:29.793056 2118 ts_tablet_manager.cc:1939] T f62d96c37cca4143a4b098b7c69979e6 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:29.794173 2323 raft_consensus.cc:491] T d000ba05e12843b4925e622ab55471c0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:29.794293 2323 raft_consensus.cc:513] T d000ba05e12843b4925e622ab55471c0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:29.794914 2323 leader_election.cc:290] T d000ba05e12843b4925e622ab55471c0 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:29.796149 858 leader_election.cc:336] T d000ba05e12843b4925e622ab55471c0 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:29.800150 2113 ts_tablet_manager.cc:1918] T 82d163681fda47f38d79863bd28f576d P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:29.800761 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 123) took 3784 ms. Trace:
I20250901 01:52:29.800871 2118 rpcz_store.cc:276] 0901 01:52:26.016723 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.016765 (+ 42us) service_pool.cc:225] Handling call
0901 01:52:29.774268 (+3757503us) ts_tablet_manager.cc:1145] Deleting tablet f62d96c37cca4143a4b098b7c69979e6
0901 01:52:29.774276 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:29.778666 (+ 4390us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:29.781627 (+ 2961us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:29.800728 (+ 19101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3757364}
I20250901 01:52:29.801196 2118 tablet_replica.cc:331] T f9b293bf6a17453b9ea9892a6970d0b0 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:29.802696 2118 raft_consensus.cc:2241] T f9b293bf6a17453b9ea9892a6970d0b0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:29.802865 2118 raft_consensus.cc:2270] T f9b293bf6a17453b9ea9892a6970d0b0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:29.803347 2118 ts_tablet_manager.cc:1905] T f9b293bf6a17453b9ea9892a6970d0b0 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:29.807848 2323 raft_consensus.cc:491] T 5f52698fb56d472d9e91fb10cb10c69a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:29.807961 2323 raft_consensus.cc:513] T 5f52698fb56d472d9e91fb10cb10c69a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:29.808517 2323 leader_election.cc:290] T 5f52698fb56d472d9e91fb10cb10c69a P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:29.808897 2113 log.cc:1199] T 82d163681fda47f38d79863bd28f576d P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/82d163681fda47f38d79863bd28f576d
W20250901 01:52:29.809885 858 leader_election.cc:336] T 5f52698fb56d472d9e91fb10cb10c69a P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:29.810365 2113 ts_tablet_manager.cc:1939] T 82d163681fda47f38d79863bd28f576d P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:29.811257 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 116) took 4681 ms. Trace:
I20250901 01:52:29.822002 2118 ts_tablet_manager.cc:1918] T f9b293bf6a17453b9ea9892a6970d0b0 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:29.822145 2118 log.cc:1199] T f9b293bf6a17453b9ea9892a6970d0b0 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/f9b293bf6a17453b9ea9892a6970d0b0
I20250901 01:52:29.822465 2118 ts_tablet_manager.cc:1939] T f9b293bf6a17453b9ea9892a6970d0b0 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:29.822875 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 124) took 3803 ms. Trace:
I20250901 01:52:29.822960 2118 rpcz_store.cc:276] 0901 01:52:26.019897 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.019941 (+ 44us) service_pool.cc:225] Handling call
0901 01:52:29.801152 (+3781211us) ts_tablet_manager.cc:1145] Deleting tablet f9b293bf6a17453b9ea9892a6970d0b0
0901 01:52:29.801162 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:29.806775 (+ 5613us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:29.821895 (+ 15120us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:29.822856 (+ 961us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1284,"tablet-delete.queue_time_us":3780270}
I20250901 01:52:29.823334 2549 raft_consensus.cc:491] T df01d5df9fc2424eaf9de4e68e781803 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:29.823469 2549 raft_consensus.cc:513] T df01d5df9fc2424eaf9de4e68e781803 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:29.825480 858 leader_election.cc:336] T df01d5df9fc2424eaf9de4e68e781803 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:29.825649 2549 leader_election.cc:290] T df01d5df9fc2424eaf9de4e68e781803 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:29.826525 1150 tablet_bootstrap.cc:492] T b2b172757e50404aae78e818921a7d9d P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:29.826639 1150 ts_tablet_manager.cc:1397] T b2b172757e50404aae78e818921a7d9d P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.051s user 0.000s sys 0.003s
I20250901 01:52:29.827330 1150 raft_consensus.cc:357] T b2b172757e50404aae78e818921a7d9d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:29.827478 1150 raft_consensus.cc:383] T b2b172757e50404aae78e818921a7d9d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:29.827522 1150 raft_consensus.cc:738] T b2b172757e50404aae78e818921a7d9d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:29.827728 1150 consensus_queue.cc:260] T b2b172757e50404aae78e818921a7d9d P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:29.828495 1150 ts_tablet_manager.cc:1428] T b2b172757e50404aae78e818921a7d9d P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250901 01:52:29.828784 1150 tablet_bootstrap.cc:492] T ae21b9ae2e3b472eb67fb5f96388a50f P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:29.836633 2113 rpcz_store.cc:276] 0901 01:52:25.129292 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.129330 (+ 38us) service_pool.cc:225] Handling call
0901 01:52:29.782277 (+4652947us) ts_tablet_manager.cc:1145] Deleting tablet 82d163681fda47f38d79863bd28f576d
0901 01:52:29.782286 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:29.794682 (+ 12396us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:29.800049 (+ 5367us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:29.811237 (+ 11188us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4652795}
I20250901 01:52:29.837119 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 125) took 3811 ms. Trace:
I20250901 01:52:29.837248 2118 rpcz_store.cc:276] 0901 01:52:26.025183 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.025229 (+ 46us) service_pool.cc:225] Handling call
0901 01:52:29.836981 (+3811752us) ts_tablet_manager.cc:1145] Deleting tablet c754129977904f978f87fd89d7092e36
0901 01:52:29.836991 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:29.837093 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3757384}
I20250901 01:52:29.837656 2118 tablet_replica.cc:331] T 72ba689f9230411995dc96e2b45601df P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:29.837877 2113 tablet_replica.cc:331] T 4a87e68850c4459eb2c8eafac89c6aa9 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:29.837872 2118 raft_consensus.cc:2241] T 72ba689f9230411995dc96e2b45601df P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:29.838083 2118 raft_consensus.cc:2270] T 72ba689f9230411995dc96e2b45601df P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:29.838634 2118 ts_tablet_manager.cc:1905] T 72ba689f9230411995dc96e2b45601df P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:29.771410 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 246b33338de647689d7404c70676e76f (table default.Table_aee74198b51d4f089538c4f2bf3ce4dc [id=a7b1d2a8ace14ef8bcce0a3bed41e85d]) successfully deleted
W20250901 01:52:29.843430 1472 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.2:40561, user_credentials={real_user=slave}} blocked reactor thread for 71998us
I20250901 01:52:29.843009 2409 raft_consensus.cc:491] T 615b2306fca84003b264da9d5f95ddab P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250901 01:52:29.843948 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 4410da9e95454ecf95faf80221b0a71c because the tablet was not found. No further retry: Not found: Tablet not found: 4410da9e95454ecf95faf80221b0a71c
I20250901 01:52:29.844009 2409 raft_consensus.cc:513] T 615b2306fca84003b264da9d5f95ddab P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:29.844147 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 3c12d6a70ac24dce82e851782e6fe5aa because the tablet was not found. No further retry: Not found: Tablet not found: 3c12d6a70ac24dce82e851782e6fe5aa
I20250901 01:52:29.844295 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet f62d96c37cca4143a4b098b7c69979e6 (table default.Table_2ddf898959684cdc995c1351d84a9661 [id=1f65b3e09c9748f189ecb2b56c5d1132]) successfully deleted
I20250901 01:52:29.844540 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet f9b293bf6a17453b9ea9892a6970d0b0 (table default.Table_a7fa05538bbd4766aa24d557677a692c [id=b67bfe518a094e0781ef931524b55990]) successfully deleted
I20250901 01:52:29.844666 2409 leader_election.cc:290] T 615b2306fca84003b264da9d5f95ddab P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:29.844736 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet c754129977904f978f87fd89d7092e36 because the tablet was not found. No further retry: Not found: Tablet not found: c754129977904f978f87fd89d7092e36
I20250901 01:52:29.775059 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 121803e79ab64ec0b948b448646b9d0a (table default.Table_117c6c43b552476dbb8b7b5ccdce47b7 [id=a4c947d85200437395c66bb4f7e607af]) successfully deleted
W20250901 01:52:29.844978 1473 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.3:39447, user_credentials={real_user=slave}} blocked reactor thread for 69947.2us
I20250901 01:52:29.838308 2113 raft_consensus.cc:2241] T 4a87e68850c4459eb2c8eafac89c6aa9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:29.845207 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 82d163681fda47f38d79863bd28f576d (table default.Table_dfd01d6b61ac43538c37f74330579cbd [id=b067d050196e4185b76c016c93f27c02]) successfully deleted
I20250901 01:52:29.845276 2113 raft_consensus.cc:2270] T 4a87e68850c4459eb2c8eafac89c6aa9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:29.846091 2113 ts_tablet_manager.cc:1905] T 4a87e68850c4459eb2c8eafac89c6aa9 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:29.847640 1150 tablet_bootstrap.cc:654] T ae21b9ae2e3b472eb67fb5f96388a50f P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:29.854008 1484 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 5d68ea97c71942c7b55b967ba870e0d7 (table default.Table_f610cb75906d4c5cb3b57cfdb80ecefc [id=36915a3e843340ba8ef3578ac35a9af4]). Expected version 1 got 0
I20250901 01:52:29.854601 1484 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet ae21b9ae2e3b472eb67fb5f96388a50f (table default.Table_046289f393ff432aae0b033bdec1fb17 [id=db57b18a9ce24e068b362f5fbe940531]). Expected version 1 got 0
I20250901 01:52:29.855482 2529 consensus_queue.cc:1035] T 6789505328b1409ea288e8b4eaf94935 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:29.856182 2528 tablet.cc:1722] T f80f13db501c42c7950499df8fcb6fc8 P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:29.856228 1484 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet e9f1f892658b4736859ff507536920a1 (table default.Table_8825dd28afbe464193eb1b668cc9326a [id=9aa62cb634314a0a849612c54c1220d9]). Expected version 1 got 0
I20250901 01:52:29.857043 1484 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 93bcac1fd95d44dc927abfca81692921 (table default.Table_b8f8ee74648c406eaf589d3ab80934db [id=b8f69313439543be9c11eea88dfa69c5]). Expected version 1 got 0
W20250901 01:52:29.857893 992 consensus_peers.cc:489] T 50e3da53433f4c14b57db8a733fa53bc P fd27ab6665304aaea53adbd102dd4f58 -> Peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Couldn't send request to peer 00c282f779c1477d920f24dc0dc16628. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 50e3da53433f4c14b57db8a733fa53bc. This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:29.858228 1484 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 44af4764d4f64e44aec84ccb84be6f5f (table default.Table_9896b980158948339a0b91f5882ed56b [id=a541b55be0dc420a8353ef4793b925b5]). Expected version 1 got 0
I20250901 01:52:29.860518 2528 tablet.cc:1722] T ea846f88b3534beebd0001b7f17b5ca9 P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:29.860905 1484 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 14b466b2a5a249aa841f1b8b06301edc (table default.Table_8c439dfdc16046b682876df01dadd3e0 [id=78cd71321e9a480ca2d7d88327f157ee]). Expected version 2 got 0
W20250901 01:52:29.861029 992 leader_election.cc:343] T 824229183e024feaa975c7f0387c9c1a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 824229183e024feaa975c7f0387c9c1a
I20250901 01:52:29.861470 1484 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 0e4009963cfe42d6ba164fc828897f09 (table default.Table_1adc70cccd2f4cb3a79df96861cb4181 [id=24fffc7ebf5a47bdb37211910301acf7]). Expected version 1 got 0
I20250901 01:52:29.861789 992 leader_election.cc:304] T f9b293bf6a17453b9ea9892a6970d0b0 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:29.868579 2528 tablet.cc:1722] T 89d40f0331114490b51de3ba6d773571 P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:29.868934 992 leader_election.cc:304] T ff013ea889d44cba8797834e4611b489 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:29.869383 1484 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 6c0abbfc8d904dce989ce51961e4237d (table default.Table_3f9422b08b0349b9a5466218f0fe1b06 [id=f0dfb3811e744bd4ba6d59195cebde6f]). Expected version 2 got 0
I20250901 01:52:29.869764 1993 raft_consensus.cc:2802] T f9b293bf6a17453b9ea9892a6970d0b0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:29.869863 1993 raft_consensus.cc:491] T f9b293bf6a17453b9ea9892a6970d0b0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:29.869926 1993 raft_consensus.cc:3058] T f9b293bf6a17453b9ea9892a6970d0b0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:29.870116 1484 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 508406601c994e42a5c5b1f665e74957 (table default.Table_345b7f9fe8214aa6878f20a166125364 [id=4aabc328f6374ec5b1dea26f778208b2]). Expected version 1 got 0
I20250901 01:52:29.870666 1484 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet aec951f2cc84499b9162b9fa24d5440e (table default.Table_156d1feafd5246ff8ff1bb7b9ae444a5 [id=55c31cd42fac43e6b8eddc048f565d0a]). Expected version 1 got 0
I20250901 01:52:29.872905 2518 raft_consensus.cc:2802] T ff013ea889d44cba8797834e4611b489 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:29.873018 2518 raft_consensus.cc:695] T ff013ea889d44cba8797834e4611b489 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:29.873303 2518 consensus_queue.cc:237] T ff013ea889d44cba8797834e4611b489 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:29.873699 1484 catalog_manager.cc:5582] T 3d90c6ef06d1436d8e2bb1333e7d9985 P 00c282f779c1477d920f24dc0dc16628 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00c282f779c1477d920f24dc0dc16628 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "00c282f779c1477d920f24dc0dc16628" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: HEALTHY } } }
I20250901 01:52:29.874234 1484 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet f80f13db501c42c7950499df8fcb6fc8 (table default.Table_ce86974fd8974de9a5a737954fac2ad7 [id=ff772d20b287476f94716379b86d2a71]). Expected version 1 got 0
I20250901 01:52:29.874400 1484 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet b164b4a67a9143ceaff69b0ce49d63fe (table default.Table_da2c70a8f2fe49a0a8184e88c0a9e0ea [id=c1b1e9e3876244f68d31c63ce7bdebf1]). Expected version 1 got 0
I20250901 01:52:29.872334 1993 raft_consensus.cc:513] T f9b293bf6a17453b9ea9892a6970d0b0 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:29.874874 1993 leader_election.cc:290] T f9b293bf6a17453b9ea9892a6970d0b0 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:29.874913 1484 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 3618a3eb4f174c43805ce7dc0e5315cb (table default.Table_83f2c0acedbc4c6982caf5313646e90b [id=31a2e50c32644321825da40176127015]). Expected version 1 got 0
I20250901 01:52:29.875072 1484 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 858218c4427c41e980b2b5c969a5af40 (table default.Table_6d47266af0ed433eb1f2abbc81d18e69 [id=264cc469a73841e79538843479a165ab]). Expected version 1 got 0
I20250901 01:52:29.875080 1993 consensus_queue.cc:1035] T 7954393c8c1c4457ab8bb723c03a9c8b P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:29.875322 1484 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 89d40f0331114490b51de3ba6d773571 (table default.Table_4dacb3b1d2e94e42ac8937eb6eb58375 [id=8ee0aa1ba24042dd8ded004644913b9c]). Expected version 1 got 0
I20250901 01:52:29.875499 1484 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 7954393c8c1c4457ab8bb723c03a9c8b (table default.Table_7050639ebd0147959ac4a251f534eb96 [id=543f2074598d4c29972e564be1123fca]). Expected version 5 got 0
I20250901 01:52:29.876166 2528 tablet.cc:1722] T ad45c90ba0da43c9860745e534664a98 P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250901 01:52:29.882543 1150 tablet_bootstrap.cc:492] T ae21b9ae2e3b472eb67fb5f96388a50f P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:29.882736 1150 ts_tablet_manager.cc:1397] T ae21b9ae2e3b472eb67fb5f96388a50f P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.054s user 0.002s sys 0.003s
I20250901 01:52:29.883169 1484 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 615b2306fca84003b264da9d5f95ddab (table default.Table_302f6e7cf97d46b9a621eb852e6aefe0 [id=cecdd8d6ca6d445c9fd82decd2a95366]). Expected version 2 got 0
I20250901 01:52:29.883482 1484 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet d0c3c363dda444519a17dca63ad9caa7 (table default.Table_42422c44ca254d8ca989134c2433391f [id=8c1b263824ec4c15a5ec424364c78895]). Expected version 3 got 0
I20250901 01:52:29.883793 1484 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 33f637f6a8134215bb1995ec38440073 (table default.Table_3d5afc22c4e14d6990aea31564dfde20 [id=557afe29976d40f6bfb6ae4ca348f887]). Expected version 2 got 0
I20250901 01:52:29.884200 1150 raft_consensus.cc:357] T ae21b9ae2e3b472eb67fb5f96388a50f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:29.885584 1150 raft_consensus.cc:383] T ae21b9ae2e3b472eb67fb5f96388a50f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:29.887017 1150 raft_consensus.cc:738] T ae21b9ae2e3b472eb67fb5f96388a50f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:29.887445 1484 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 4cde5f4f02044b7bbd7d43b49aac4ae8 (table default.Table_d53a456dbca64daf92aeb5fb4517b417 [id=77ae3376281349e087a3b3cee41f2a09]). Expected version 1 got 0
I20250901 01:52:29.888008 1150 consensus_queue.cc:260] T ae21b9ae2e3b472eb67fb5f96388a50f P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:29.888772 1484 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 45f6e8ba99b74898a2922b36768c6c36 (table default.Table_938421db3cf74cf8b878089bac931818 [id=9ce26dcf307149b595a43b6ac4562fba]). Expected version 1 got 0
I20250901 01:52:29.891973 1150 ts_tablet_manager.cc:1428] T ae21b9ae2e3b472eb67fb5f96388a50f P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20250901 01:52:29.884687 2118 ts_tablet_manager.cc:1918] T 72ba689f9230411995dc96e2b45601df P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 01:52:29.893924 2118 log.cc:1199] T 72ba689f9230411995dc96e2b45601df P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/72ba689f9230411995dc96e2b45601df
I20250901 01:52:29.894325 2118 ts_tablet_manager.cc:1939] T 72ba689f9230411995dc96e2b45601df P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:29.896297 1150 tablet_bootstrap.cc:492] T 69f61ec283484f9d8b3eebcea27f741b P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:29.897830 970 heartbeater.cc:344] Connected to a master server at 127.0.104.62:43871
I20250901 01:52:29.897125 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 126) took 3805 ms. Trace:
I20250901 01:52:29.898044 2118 rpcz_store.cc:276] 0901 01:52:26.091850 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.091895 (+ 45us) service_pool.cc:225] Handling call
0901 01:52:29.837593 (+3745698us) ts_tablet_manager.cc:1145] Deleting tablet 72ba689f9230411995dc96e2b45601df
0901 01:52:29.837613 (+ 20us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:29.859839 (+ 22226us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:29.884566 (+ 24727us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:29.897096 (+ 12530us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3745503}
I20250901 01:52:29.898530 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 72ba689f9230411995dc96e2b45601df (table default.Table_604f2c5c58d74154bacf2a065a467f32 [id=351a05134f4948039c920cafbe050914]) successfully deleted
I20250901 01:52:29.899734 1150 tablet_bootstrap.cc:654] T 69f61ec283484f9d8b3eebcea27f741b P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:29.900657 2118 tablet_replica.cc:331] T 606fac5c6efe48d1877104b01c23ba5b P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:29.900872 2118 raft_consensus.cc:2241] T 606fac5c6efe48d1877104b01c23ba5b P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:29.901064 2118 raft_consensus.cc:2270] T 606fac5c6efe48d1877104b01c23ba5b P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:29.901588 2118 ts_tablet_manager.cc:1905] T 606fac5c6efe48d1877104b01c23ba5b P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:29.902348 2546 mvcc.cc:204] Tried to move back new op lower bound from 7195408563780915200 to 7195408555434221568. Current Snapshot: MvccSnapshot[applied={T|T < 7195408563780915200 or (T in {7195408563780915200})}]
I20250901 01:52:29.905906 1151 tablet_bootstrap.cc:492] T 1495de70f0844f82a0eea99c059f076b P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:29.906128 1151 ts_tablet_manager.cc:1397] T 1495de70f0844f82a0eea99c059f076b P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.126s user 0.003s sys 0.006s
I20250901 01:52:29.906947 1151 raft_consensus.cc:357] T 1495de70f0844f82a0eea99c059f076b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:29.907184 1151 raft_consensus.cc:383] T 1495de70f0844f82a0eea99c059f076b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:29.907269 1151 raft_consensus.cc:738] T 1495de70f0844f82a0eea99c059f076b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:29.907519 1151 consensus_queue.cc:260] T 1495de70f0844f82a0eea99c059f076b P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:29.908109 1151 ts_tablet_manager.cc:1428] T 1495de70f0844f82a0eea99c059f076b P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250901 01:52:29.913879 1151 tablet_bootstrap.cc:492] T 4cde5f4f02044b7bbd7d43b49aac4ae8 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:29.916198 1151 tablet_bootstrap.cc:654] T 4cde5f4f02044b7bbd7d43b49aac4ae8 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:29.916679 992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:40561, user_credentials={real_user=slave}} blocked reactor thread for 46843.4us
I20250901 01:52:29.917158 1993 consensus_queue.cc:1035] T aec951f2cc84499b9162b9fa24d5440e P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:29.917273 1150 tablet_bootstrap.cc:492] T 69f61ec283484f9d8b3eebcea27f741b P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:29.917392 1150 ts_tablet_manager.cc:1397] T 69f61ec283484f9d8b3eebcea27f741b P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.001s
I20250901 01:52:29.917490 992 leader_election.cc:304] T d10c193452954c198bba686169b8025b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:29.918169 1150 raft_consensus.cc:357] T 69f61ec283484f9d8b3eebcea27f741b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:29.918506 1150 raft_consensus.cc:383] T 69f61ec283484f9d8b3eebcea27f741b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:29.918866 2574 tablet.cc:1722] T 616ff34e8d314e0aa0e11ce60be11789 P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250901 01:52:29.919129 2409 raft_consensus.cc:2802] T d10c193452954c198bba686169b8025b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:29.919211 2409 raft_consensus.cc:491] T d10c193452954c198bba686169b8025b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:29.919289 2409 raft_consensus.cc:3058] T d10c193452954c198bba686169b8025b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:29.918496 2573 tablet.cc:1722] T 3618a3eb4f174c43805ce7dc0e5315cb P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:29.922305 2409 raft_consensus.cc:513] T d10c193452954c198bba686169b8025b P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:29.923013 2409 leader_election.cc:290] T d10c193452954c198bba686169b8025b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:29.923529 859 leader_election.cc:343] T 0e4009963cfe42d6ba164fc828897f09 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:29.923681 859 leader_election.cc:304] T 0e4009963cfe42d6ba164fc828897f09 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:29.924340 2549 raft_consensus.cc:2747] T 0e4009963cfe42d6ba164fc828897f09 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:29.925595 992 leader_election.cc:336] T ff013ea889d44cba8797834e4611b489 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:29.925838 992 leader_election.cc:336] T 824229183e024feaa975c7f0387c9c1a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:29.925913 992 leader_election.cc:304] T 824229183e024feaa975c7f0387c9c1a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
W20250901 01:52:29.926054 992 raft_consensus.cc:2684] T 824229183e024feaa975c7f0387c9c1a P fd27ab6665304aaea53adbd102dd4f58: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250901 01:52:29.926898 992 consensus_peers.cc:489] T f8a5886ddb644a86b76df44aab309057 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250901 01:52:29.927027 992 consensus_peers.cc:489] T 92f0ad59fbbb435ea872c669a52f854a P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250901 01:52:29.927135 992 consensus_peers.cc:489] T fae2779fac164b9f8052c3666db803b8 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250901 01:52:29.927214 992 leader_election.cc:336] T f9b293bf6a17453b9ea9892a6970d0b0 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:29.931808 2118 ts_tablet_manager.cc:1918] T 606fac5c6efe48d1877104b01c23ba5b P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 01:52:29.932281 2118 log.cc:1199] T 606fac5c6efe48d1877104b01c23ba5b P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/606fac5c6efe48d1877104b01c23ba5b
I20250901 01:52:29.933473 2118 ts_tablet_manager.cc:1939] T 606fac5c6efe48d1877104b01c23ba5b P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:29.935940 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 127) took 3707 ms. Trace:
I20250901 01:52:29.938678 2113 ts_tablet_manager.cc:1918] T 4a87e68850c4459eb2c8eafac89c6aa9 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:29.942698 2113 log.cc:1199] T 4a87e68850c4459eb2c8eafac89c6aa9 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/4a87e68850c4459eb2c8eafac89c6aa9
I20250901 01:52:29.948882 2113 ts_tablet_manager.cc:1939] T 4a87e68850c4459eb2c8eafac89c6aa9 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:29.954257 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 117) took 4824 ms. Trace:
I20250901 01:52:29.949960 1151 tablet_bootstrap.cc:492] T 4cde5f4f02044b7bbd7d43b49aac4ae8 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:29.955020 1151 ts_tablet_manager.cc:1397] T 4cde5f4f02044b7bbd7d43b49aac4ae8 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.047s user 0.004s sys 0.000s
I20250901 01:52:29.956080 1151 raft_consensus.cc:357] T 4cde5f4f02044b7bbd7d43b49aac4ae8 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:29.956434 1151 raft_consensus.cc:383] T 4cde5f4f02044b7bbd7d43b49aac4ae8 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:29.956552 1151 raft_consensus.cc:738] T 4cde5f4f02044b7bbd7d43b49aac4ae8 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:29.956844 1151 consensus_queue.cc:260] T 4cde5f4f02044b7bbd7d43b49aac4ae8 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:29.957854 1151 ts_tablet_manager.cc:1428] T 4cde5f4f02044b7bbd7d43b49aac4ae8 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250901 01:52:29.958187 1151 tablet_bootstrap.cc:492] T a9b7f2e6548043a7865e25387cdd06c9 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:29.959936 2113 rpcz_store.cc:276] 0901 01:52:25.129592 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.129635 (+ 43us) service_pool.cc:225] Handling call
0901 01:52:29.837832 (+4708197us) ts_tablet_manager.cc:1145] Deleting tablet 4a87e68850c4459eb2c8eafac89c6aa9
0901 01:52:29.837844 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:29.926452 (+ 88608us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:29.938551 (+ 12099us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:29.954228 (+ 15677us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4708011}
I20250901 01:52:29.960682 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 4a87e68850c4459eb2c8eafac89c6aa9 (table default.Table_302f6e7cf97d46b9a621eb852e6aefe0 [id=cecdd8d6ca6d445c9fd82decd2a95366]) successfully deleted
I20250901 01:52:29.961126 2113 tablet_replica.cc:331] T 12c04e0c1f804179ae23509cdb24d395 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:29.961364 2113 raft_consensus.cc:2241] T 12c04e0c1f804179ae23509cdb24d395 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:29.962231 2113 raft_consensus.cc:2270] T 12c04e0c1f804179ae23509cdb24d395 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:29.962771 2113 ts_tablet_manager.cc:1905] T 12c04e0c1f804179ae23509cdb24d395 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:29.974440 2113 ts_tablet_manager.cc:1918] T 12c04e0c1f804179ae23509cdb24d395 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:29.936913 919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "615b2306fca84003b264da9d5f95ddab" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:29.931900 1150 raft_consensus.cc:738] T 69f61ec283484f9d8b3eebcea27f741b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:29.981166 919 raft_consensus.cc:2466] T 615b2306fca84003b264da9d5f95ddab P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:29.950851 2357 raft_consensus.cc:491] T 2973b59bb1df41d18d9a5b6a29ae336f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:29.953035 2585 tablet.cc:1722] T 8ca5cf4e99014c0d84a3acad4ef713af P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:29.981274 2582 tablet.cc:1722] T ea846f88b3534beebd0001b7f17b5ca9 P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:29.981380 2357 raft_consensus.cc:513] T 2973b59bb1df41d18d9a5b6a29ae336f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:29.961730 2323 raft_consensus.cc:491] T 3239e58531d445fa80165cf678bc277c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:29.981494 2118 rpcz_store.cc:276] 0901 01:52:26.228111 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.230390 (+ 2279us) service_pool.cc:225] Handling call
0901 01:52:29.898352 (+3667962us) ts_tablet_manager.cc:1145] Deleting tablet 606fac5c6efe48d1877104b01c23ba5b
0901 01:52:29.898362 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:29.907793 (+ 9431us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:29.931689 (+ 23896us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:29.935923 (+ 4234us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3666857}
I20250901 01:52:29.981756 2323 raft_consensus.cc:513] T 3239e58531d445fa80165cf678bc277c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:29.981950 2357 leader_election.cc:290] T 2973b59bb1df41d18d9a5b6a29ae336f P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:29.982051 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 606fac5c6efe48d1877104b01c23ba5b (table default.Table_7c12d66048a6470e8c227dc1d9f45fa4 [id=91472f220fe44297b79d3eec8b9b80e8]) successfully deleted
I20250901 01:52:29.981299 1150 consensus_queue.cc:260] T 69f61ec283484f9d8b3eebcea27f741b P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:29.982291 2323 leader_election.cc:290] T 3239e58531d445fa80165cf678bc277c P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:29.961843 2549 raft_consensus.cc:491] T 5f5396045979450193d461ba1334a5c9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:29.982867 2549 raft_consensus.cc:513] T 5f5396045979450193d461ba1334a5c9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:29.983403 2549 leader_election.cc:290] T 5f5396045979450193d461ba1334a5c9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:29.983788 858 leader_election.cc:336] T 3239e58531d445fa80165cf678bc277c P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:29.984382 858 leader_election.cc:336] T 2973b59bb1df41d18d9a5b6a29ae336f P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:29.984863 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 128) took 3730 ms. Trace:
I20250901 01:52:29.982753 1150 ts_tablet_manager.cc:1428] T 69f61ec283484f9d8b3eebcea27f741b P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.065s user 0.014s sys 0.000s
I20250901 01:52:29.962093 1151 tablet_bootstrap.cc:654] T a9b7f2e6548043a7865e25387cdd06c9 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:29.993003 2118 rpcz_store.cc:276] 0901 01:52:26.254796 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.255073 (+ 277us) service_pool.cc:225] Handling call
0901 01:52:29.984741 (+3729668us) ts_tablet_manager.cc:1145] Deleting tablet 87e97a9544a2485897b1053c8929a39a
0901 01:52:29.984764 (+ 23us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:29.984848 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3726651}
I20250901 01:52:29.993278 1150 tablet_bootstrap.cc:492] T f23df6f253a943fa8e146bb8c811ef51 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:29.993359 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 129) took 3722 ms. Trace:
I20250901 01:52:29.993430 2118 rpcz_store.cc:276] 0901 01:52:26.270596 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.270656 (+ 60us) service_pool.cc:225] Handling call
0901 01:52:29.993255 (+3722599us) ts_tablet_manager.cc:1145] Deleting tablet 0f37a24e693847f98b574950379081f3
0901 01:52:29.993265 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:29.993341 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3722409}
I20250901 01:52:29.993685 2118 tablet_replica.cc:331] T ddbc15685fca4c708329420109ad0c57 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:29.993875 2118 raft_consensus.cc:2241] T ddbc15685fca4c708329420109ad0c57 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:29.994093 2118 raft_consensus.cc:2270] T ddbc15685fca4c708329420109ad0c57 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:29.994695 1150 tablet_bootstrap.cc:654] T f23df6f253a943fa8e146bb8c811ef51 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:29.994764 2118 ts_tablet_manager.cc:1905] T ddbc15685fca4c708329420109ad0c57 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:29.984954 858 leader_election.cc:336] T 5f5396045979450193d461ba1334a5c9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:29.989388 2585 tablet.cc:1722] T ad45c90ba0da43c9860745e534664a98 P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250901 01:52:29.996239 2113 log.cc:1199] T 12c04e0c1f804179ae23509cdb24d395 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/12c04e0c1f804179ae23509cdb24d395
W20250901 01:52:29.996220 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 87e97a9544a2485897b1053c8929a39a because the tablet was not found. No further retry: Not found: Tablet not found: 87e97a9544a2485897b1053c8929a39a
I20250901 01:52:29.980710 2590 tablet.cc:1722] T f80f13db501c42c7950499df8fcb6fc8 P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250901 01:52:29.996497 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 0f37a24e693847f98b574950379081f3 because the tablet was not found. No further retry: Not found: Tablet not found: 0f37a24e693847f98b574950379081f3
I20250901 01:52:30.000314 2585 tablet.cc:1722] T 89d40f0331114490b51de3ba6d773571 P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:30.000581 2113 ts_tablet_manager.cc:1939] T 12c04e0c1f804179ae23509cdb24d395 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:30.004703 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 118) took 4874 ms. Trace:
I20250901 01:52:30.004909 2590 tablet.cc:1722] T 3618a3eb4f174c43805ce7dc0e5315cb P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:30.007701 918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d10c193452954c198bba686169b8025b" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:30.007879 918 raft_consensus.cc:3058] T d10c193452954c198bba686169b8025b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:30.008224 2113 rpcz_store.cc:276] 0901 01:52:25.129936 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.129976 (+ 40us) service_pool.cc:225] Handling call
0901 01:52:29.961067 (+4831091us) ts_tablet_manager.cc:1145] Deleting tablet 12c04e0c1f804179ae23509cdb24d395
0901 01:52:29.961081 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:29.971415 (+ 10334us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:29.974328 (+ 2913us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:30.004681 (+ 30353us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4830897}
I20250901 01:52:30.009084 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 12c04e0c1f804179ae23509cdb24d395 (table default.Table_3d223b93128649e49a295d940feef90a [id=348190236c694eab9a318040bb19c1f2]) successfully deleted
I20250901 01:52:30.009527 2113 tablet_replica.cc:331] T 670d90a37b934579b353bb4e4820127b P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:30.009951 1056 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d000ba05e12843b4925e622ab55471c0" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:30.010283 1058 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "df01d5df9fc2424eaf9de4e68e781803" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:30.010373 918 raft_consensus.cc:2466] T d10c193452954c198bba686169b8025b P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:30.010668 1053 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5afae316ff164413be0d25d4bc0f58dc" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:30.011027 1051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2dacb608897d47d28512df87dc320927" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:30.011330 1057 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "764bfeb2a7d14c40815b5ec1b07bdd62" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:30.011605 1055 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f52698fb56d472d9e91fb10cb10c69a" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:30.009965 2113 raft_consensus.cc:2241] T 670d90a37b934579b353bb4e4820127b P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:30.015775 2585 tablet.cc:1722] T 616ff34e8d314e0aa0e11ce60be11789 P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250901 01:52:30.018034 2113 raft_consensus.cc:2270] T 670d90a37b934579b353bb4e4820127b P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:30.019466 2113 ts_tablet_manager.cc:1905] T 670d90a37b934579b353bb4e4820127b P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:30.022778 2552 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ce2288f784044eea010664e3def133f. Latest consensus state: current_term: 4 leader_uuid: "7ce2288f784044eea010664e3def133f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3add7edb97a40ef9f7c6bc011d13227" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 43871 } } peers { permanent_uuid: "d2ae2cb6af0f4b3a8f47ab1cdd6567d3" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42375 } } peers { permanent_uuid: "7ce2288f784044eea010664e3def133f" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 44231 } } }
I20250901 01:52:30.022686 2571 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "7ce2288f784044eea010664e3def133f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3add7edb97a40ef9f7c6bc011d13227" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 43871 } } peers { permanent_uuid: "d2ae2cb6af0f4b3a8f47ab1cdd6567d3" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42375 } } peers { permanent_uuid: "7ce2288f784044eea010664e3def133f" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 44231 } } }
I20250901 01:52:30.023025 2571 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227 [sys.catalog]: This master's current role is: FOLLOWER
I20250901 01:52:30.023021 2552 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227 [sys.catalog]: This master's current role is: FOLLOWER
I20250901 01:52:30.023666 1150 tablet_bootstrap.cc:492] T f23df6f253a943fa8e146bb8c811ef51 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:30.023856 1150 ts_tablet_manager.cc:1397] T f23df6f253a943fa8e146bb8c811ef51 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.031s user 0.000s sys 0.015s
I20250901 01:52:30.024643 1150 raft_consensus.cc:357] T f23df6f253a943fa8e146bb8c811ef51 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.024874 1150 raft_consensus.cc:383] T f23df6f253a943fa8e146bb8c811ef51 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:30.024976 1150 raft_consensus.cc:738] T f23df6f253a943fa8e146bb8c811ef51 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:30.025175 2146 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54886 (request call id 1) took 5167 ms. Trace:
I20250901 01:52:30.021514 2574 tablet.cc:1722] T 8ca5cf4e99014c0d84a3acad4ef713af P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:30.021895 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2973b59bb1df41d18d9a5b6a29ae336f" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:30.022264 1054 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3239e58531d445fa80165cf678bc277c" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:30.022461 1055 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f5396045979450193d461ba1334a5c9" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:30.022907 992 leader_election.cc:336] T d10c193452954c198bba686169b8025b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:30.028450 1150 consensus_queue.cc:260] T f23df6f253a943fa8e146bb8c811ef51 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.029430 1150 ts_tablet_manager.cc:1428] T f23df6f253a943fa8e146bb8c811ef51 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250901 01:52:30.029762 1150 tablet_bootstrap.cc:492] T 088ef4861dc14f81ae470c3aaee5b4e1 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:30.031374 1150 tablet_bootstrap.cc:654] T 088ef4861dc14f81ae470c3aaee5b4e1 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:30.040228 992 consensus_peers.cc:489] T 72ba689f9230411995dc96e2b45601df P fd27ab6665304aaea53adbd102dd4f58 -> Peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Couldn't send request to peer 00c282f779c1477d920f24dc0dc16628. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 72ba689f9230411995dc96e2b45601df. This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:30.040558 2412 consensus_queue.cc:1035] T 482a9a2cb81f448bb2823892e35920d4 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:30.048620 992 leader_election.cc:304] T 615b2306fca84003b264da9d5f95ddab P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:30.049158 2409 raft_consensus.cc:2802] T 615b2306fca84003b264da9d5f95ddab P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:30.049234 2409 raft_consensus.cc:491] T 615b2306fca84003b264da9d5f95ddab P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:30.049293 2409 raft_consensus.cc:3058] T 615b2306fca84003b264da9d5f95ddab P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:30.050045 2596 catalog_manager.cc:797] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250901 01:52:30.051791 2409 raft_consensus.cc:513] T 615b2306fca84003b264da9d5f95ddab P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.051940 1150 tablet_bootstrap.cc:492] T 088ef4861dc14f81ae470c3aaee5b4e1 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:30.052080 1150 ts_tablet_manager.cc:1397] T 088ef4861dc14f81ae470c3aaee5b4e1 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.000s
I20250901 01:52:30.052377 2409 leader_election.cc:290] T 615b2306fca84003b264da9d5f95ddab P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:30.052803 1150 raft_consensus.cc:357] T 088ef4861dc14f81ae470c3aaee5b4e1 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.053025 1150 raft_consensus.cc:383] T 088ef4861dc14f81ae470c3aaee5b4e1 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:30.053499 1151 tablet_bootstrap.cc:492] T a9b7f2e6548043a7865e25387cdd06c9 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:30.053524 1150 raft_consensus.cc:738] T 088ef4861dc14f81ae470c3aaee5b4e1 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:30.053623 1151 ts_tablet_manager.cc:1397] T a9b7f2e6548043a7865e25387cdd06c9 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.095s user 0.007s sys 0.002s
I20250901 01:52:30.053872 1150 consensus_queue.cc:260] T 088ef4861dc14f81ae470c3aaee5b4e1 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.054364 1150 ts_tablet_manager.cc:1428] T 088ef4861dc14f81ae470c3aaee5b4e1 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250901 01:52:30.054317 1151 raft_consensus.cc:357] T a9b7f2e6548043a7865e25387cdd06c9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.054471 1151 raft_consensus.cc:383] T a9b7f2e6548043a7865e25387cdd06c9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:30.054520 1151 raft_consensus.cc:738] T a9b7f2e6548043a7865e25387cdd06c9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:30.054672 1151 consensus_queue.cc:260] T a9b7f2e6548043a7865e25387cdd06c9 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.054877 1150 tablet_bootstrap.cc:492] T e9f1f892658b4736859ff507536920a1 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:30.055220 1151 ts_tablet_manager.cc:1428] T a9b7f2e6548043a7865e25387cdd06c9 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250901 01:52:30.055601 1151 tablet_bootstrap.cc:492] T b81a12b38ac64efabd44aeb192d181a0 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:30.056694 1150 tablet_bootstrap.cc:654] T e9f1f892658b4736859ff507536920a1 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:30.057147 1151 tablet_bootstrap.cc:654] T b81a12b38ac64efabd44aeb192d181a0 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:30.059262 992 leader_election.cc:343] T f9b293bf6a17453b9ea9892a6970d0b0 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: f9b293bf6a17453b9ea9892a6970d0b0
W20250901 01:52:30.060813 992 consensus_peers.cc:489] T d58f69fad01f403eab74a47d3695ed83 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Couldn't send request to peer 00c282f779c1477d920f24dc0dc16628. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d58f69fad01f403eab74a47d3695ed83. This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:30.060971 992 leader_election.cc:304] T d10c193452954c198bba686169b8025b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:30.061877 2409 raft_consensus.cc:2802] T d10c193452954c198bba686169b8025b P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:30.062135 2409 raft_consensus.cc:695] T d10c193452954c198bba686169b8025b P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:30.062407 2409 consensus_queue.cc:237] T d10c193452954c198bba686169b8025b P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.072093 1150 tablet_bootstrap.cc:492] T e9f1f892658b4736859ff507536920a1 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:30.072319 1150 ts_tablet_manager.cc:1397] T e9f1f892658b4736859ff507536920a1 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.000s
I20250901 01:52:30.073720 1150 raft_consensus.cc:357] T e9f1f892658b4736859ff507536920a1 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.073988 1150 raft_consensus.cc:383] T e9f1f892658b4736859ff507536920a1 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:30.074093 1150 raft_consensus.cc:738] T e9f1f892658b4736859ff507536920a1 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:30.074379 1150 consensus_queue.cc:260] T e9f1f892658b4736859ff507536920a1 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.075559 1150 ts_tablet_manager.cc:1428] T e9f1f892658b4736859ff507536920a1 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250901 01:52:30.075892 1150 tablet_bootstrap.cc:492] T 56f2c4d2e6ab4bb4bdd6d025843589b6 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:30.077687 1150 tablet_bootstrap.cc:654] T 56f2c4d2e6ab4bb4bdd6d025843589b6 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:30.083082 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet ed1b41340ce742a88c3e4fd85cd25ca6 (table default.Table_ce2f3b515c8f4c1bb557c5ed84b48dba [id=0baa4993b72e4a8d94ee3059c9d2db6c]). Expected version 2 got 0
I20250901 01:52:30.083418 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 14b466b2a5a249aa841f1b8b06301edc (table default.Table_8c439dfdc16046b682876df01dadd3e0 [id=78cd71321e9a480ca2d7d88327f157ee]). Expected version 2 got 0
W20250901 01:52:30.086859 992 consensus_peers.cc:489] T 6c05eff658924ca4a9a1e7a37f917d67 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Couldn't send request to peer 00c282f779c1477d920f24dc0dc16628. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6c05eff658924ca4a9a1e7a37f917d67. This is attempt 1: this message will repeat every 5th retry.
W20250901 01:52:30.087055 992 consensus_peers.cc:489] T e249474dcbf642189560c91ec3a99fdf P fd27ab6665304aaea53adbd102dd4f58 -> Peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Couldn't send request to peer 00c282f779c1477d920f24dc0dc16628. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e249474dcbf642189560c91ec3a99fdf. This is attempt 1: this message will repeat every 5th retry.
W20250901 01:52:30.087260 992 consensus_peers.cc:489] T 7cc1efcb6bff4aba8015d2d3b7f4c923 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Couldn't send request to peer 00c282f779c1477d920f24dc0dc16628. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7cc1efcb6bff4aba8015d2d3b7f4c923. This is attempt 1: this message will repeat every 5th retry.
W20250901 01:52:30.087997 992 consensus_peers.cc:489] T 482a9a2cb81f448bb2823892e35920d4 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250901 01:52:30.088126 992 leader_election.cc:336] T f9b293bf6a17453b9ea9892a6970d0b0 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:30.088207 992 leader_election.cc:304] T f9b293bf6a17453b9ea9892a6970d0b0 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:30.088409 1993 raft_consensus.cc:2747] T f9b293bf6a17453b9ea9892a6970d0b0 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250901 01:52:30.088486 1150 tablet_bootstrap.cc:492] T 56f2c4d2e6ab4bb4bdd6d025843589b6 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
W20250901 01:52:30.088506 992 leader_election.cc:336] T 615b2306fca84003b264da9d5f95ddab P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:30.088611 1150 ts_tablet_manager.cc:1397] T 56f2c4d2e6ab4bb4bdd6d025843589b6 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.003s
W20250901 01:52:30.088706 992 consensus_peers.cc:489] T aec951f2cc84499b9162b9fa24d5440e P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250901 01:52:30.088984 992 leader_election.cc:336] T d10c193452954c198bba686169b8025b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:30.089730 1150 raft_consensus.cc:357] T 56f2c4d2e6ab4bb4bdd6d025843589b6 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.090000 1150 raft_consensus.cc:383] T 56f2c4d2e6ab4bb4bdd6d025843589b6 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:30.090114 1150 raft_consensus.cc:738] T 56f2c4d2e6ab4bb4bdd6d025843589b6 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:30.090448 1150 consensus_queue.cc:260] T 56f2c4d2e6ab4bb4bdd6d025843589b6 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.091176 1150 ts_tablet_manager.cc:1428] T 56f2c4d2e6ab4bb4bdd6d025843589b6 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250901 01:52:30.091300 2113 ts_tablet_manager.cc:1918] T 670d90a37b934579b353bb4e4820127b P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:30.091545 2113 log.cc:1199] T 670d90a37b934579b353bb4e4820127b P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/670d90a37b934579b353bb4e4820127b
I20250901 01:52:30.091567 1150 tablet_bootstrap.cc:492] T 36e10328c34242ebb4dbd8f294162696 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:30.092106 2113 ts_tablet_manager.cc:1939] T 670d90a37b934579b353bb4e4820127b P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:30.092583 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 119) took 4962 ms. Trace:
I20250901 01:52:30.093668 1150 tablet_bootstrap.cc:654] T 36e10328c34242ebb4dbd8f294162696 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:30.099658 2105 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "00c282f779c1477d920f24dc0dc16628" instance_seqno: 1756691527381543) as {username='slave'} at 127.0.104.2:52751; Asking this server to re-register.
W20250901 01:52:30.039927 859 leader_election.cc:343] T d000ba05e12843b4925e622ab55471c0 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:30.036695 2118 ts_tablet_manager.cc:1918] T ddbc15685fca4c708329420109ad0c57 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 01:52:30.101157 2118 log.cc:1199] T ddbc15685fca4c708329420109ad0c57 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/ddbc15685fca4c708329420109ad0c57
I20250901 01:52:30.101123 859 leader_election.cc:304] T d000ba05e12843b4925e622ab55471c0 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:30.101454 859 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:39447, user_credentials={real_user=slave}} blocked reactor thread for 61541.5us
I20250901 01:52:30.101683 2118 ts_tablet_manager.cc:1939] T ddbc15685fca4c708329420109ad0c57 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
W20250901 01:52:30.101833 859 leader_election.cc:343] T df01d5df9fc2424eaf9de4e68e781803 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:30.101970 859 leader_election.cc:304] T df01d5df9fc2424eaf9de4e68e781803 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:30.102358 859 leader_election.cc:343] T 5afae316ff164413be0d25d4bc0f58dc P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:30.102466 859 leader_election.cc:304] T 5afae316ff164413be0d25d4bc0f58dc P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:30.102910 859 leader_election.cc:343] T 2dacb608897d47d28512df87dc320927 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:30.103013 859 leader_election.cc:304] T 2dacb608897d47d28512df87dc320927 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:30.103370 859 leader_election.cc:343] T 764bfeb2a7d14c40815b5ec1b07bdd62 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:30.103464 859 leader_election.cc:304] T 764bfeb2a7d14c40815b5ec1b07bdd62 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:30.103844 2549 raft_consensus.cc:2747] T d000ba05e12843b4925e622ab55471c0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:30.104008 2549 raft_consensus.cc:2747] T df01d5df9fc2424eaf9de4e68e781803 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:30.104130 2549 raft_consensus.cc:2747] T 5afae316ff164413be0d25d4bc0f58dc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:30.104218 2549 raft_consensus.cc:2747] T 2dacb608897d47d28512df87dc320927 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:30.104310 2549 raft_consensus.cc:2747] T 764bfeb2a7d14c40815b5ec1b07bdd62 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:30.107640 970 heartbeater.cc:461] Registering TS with master...
I20250901 01:52:30.083868 1492 catalog_manager.cc:5582] T 6789505328b1409ea288e8b4eaf94935 P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: UNKNOWN } } }
I20250901 01:52:30.113070 1492 catalog_manager.cc:5582] T 7954393c8c1c4457ab8bb723c03a9c8b P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: UNKNOWN } } }
I20250901 01:52:30.025885 2146 rpcz_store.cc:276] 0901 01:52:24.857680 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:24.857753 (+ 73us) service_pool.cc:225] Handling call
0901 01:52:29.717178 (+4859425us) raft_consensus.cc:1469] Updating replica for 38 ops
0901 01:52:29.724314 (+ 7136us) raft_consensus.cc:1532] Early marking committed up to index 633
0901 01:52:29.786319 (+ 62005us) raft_consensus.cc:1537] Triggering prepare for 38 ops
0901 01:52:29.786814 (+ 495us) write_op.cc:270] Start()
0901 01:52:29.786863 (+ 49us) write_op.cc:276] Timestamp: P: 1756691543471824 usec, L: 0
0901 01:52:29.787150 (+ 287us) write_op.cc:270] Start()
0901 01:52:29.787183 (+ 33us) write_op.cc:276] Timestamp: P: 1756691543544508 usec, L: 0
0901 01:52:29.787340 (+ 157us) write_op.cc:270] Start()
0901 01:52:29.787358 (+ 18us) write_op.cc:276] Timestamp: P: 1756691543545217 usec, L: 0
0901 01:52:29.787495 (+ 137us) write_op.cc:270] Start()
0901 01:52:29.787517 (+ 22us) write_op.cc:276] Timestamp: P: 1756691543545953 usec, L: 0
0901 01:52:29.787648 (+ 131us) write_op.cc:270] Start()
0901 01:52:29.787664 (+ 16us) write_op.cc:276] Timestamp: P: 1756691543546512 usec, L: 0
0901 01:52:29.787801 (+ 137us) write_op.cc:270] Start()
0901 01:52:29.787821 (+ 20us) write_op.cc:276] Timestamp: P: 1756691543547049 usec, L: 0
0901 01:52:29.787938 (+ 117us) write_op.cc:270] Start()
0901 01:52:29.787975 (+ 37us) write_op.cc:276] Timestamp: P: 1756691543547540 usec, L: 0
0901 01:52:29.788088 (+ 113us) write_op.cc:270] Start()
0901 01:52:29.788108 (+ 20us) write_op.cc:276] Timestamp: P: 1756691543577668 usec, L: 0
0901 01:52:29.788238 (+ 130us) write_op.cc:270] Start()
0901 01:52:29.788255 (+ 17us) write_op.cc:276] Timestamp: P: 1756691543607736 usec, L: 0
0901 01:52:29.788384 (+ 129us) write_op.cc:270] Start()
0901 01:52:29.788429 (+ 45us) write_op.cc:276] Timestamp: P: 1756691543621633 usec, L: 0
0901 01:52:29.788561 (+ 132us) write_op.cc:270] Start()
0901 01:52:29.788579 (+ 18us) write_op.cc:276] Timestamp: P: 1756691543623491 usec, L: 0
0901 01:52:29.788737 (+ 158us) write_op.cc:270] Start()
0901 01:52:29.788758 (+ 21us) write_op.cc:276] Timestamp: P: 1756691543624483 usec, L: 0
0901 01:52:29.788896 (+ 138us) write_op.cc:270] Start()
0901 01:52:29.788912 (+ 16us) write_op.cc:276] Timestamp: P: 1756691543655600 usec, L: 0
0901 01:52:29.789013 (+ 101us) write_op.cc:270] Start()
0901 01:52:29.789027 (+ 14us) write_op.cc:276] Timestamp: P: 1756691543657455 usec, L: 0
0901 01:52:29.789119 (+ 92us) write_op.cc:270] Start()
0901 01:52:29.789151 (+ 32us) write_op.cc:276] Timestamp: P: 1756691543658345 usec, L: 0
0901 01:52:29.802922 (+ 13771us) write_op.cc:270] Start()
0901 01:52:29.802973 (+ 51us) write_op.cc:276] Timestamp: P: 1756691543849214 usec, L: 0
0901 01:52:29.803160 (+ 187us) write_op.cc:270] Start()
0901 01:52:29.803180 (+ 20us) write_op.cc:276] Timestamp: P: 1756691543854197 usec, L: 0
0901 01:52:29.809374 (+ 6194us) write_op.cc:270] Start()
0901 01:52:29.809406 (+ 32us) write_op.cc:276] Timestamp: P: 1756691543860970 usec, L: 0
0901 01:52:29.814840 (+ 5434us) write_op.cc:270] Start()
0901 01:52:29.814872 (+ 32us) write_op.cc:276] Timestamp: P: 1756691543869847 usec, L: 0
0901 01:52:29.815022 (+ 150us) write_op.cc:270] Start()
0901 01:52:29.815042 (+ 20us) write_op.cc:276] Timestamp: P: 1756691543876431 usec, L: 0
0901 01:52:29.815158 (+ 116us) write_op.cc:270] Start()
0901 01:52:29.815175 (+ 17us) write_op.cc:276] Timestamp: P: 1756691543891825 usec, L: 0
0901 01:52:29.815285 (+ 110us) write_op.cc:270] Start()
0901 01:52:29.815302 (+ 17us) write_op.cc:276] Timestamp: P: 1756691543898894 usec, L: 0
0901 01:52:29.815413 (+ 111us) write_op.cc:270] Start()
0901 01:52:29.815429 (+ 16us) write_op.cc:276] Timestamp: P: 1756691543911766 usec, L: 0
0901 01:52:29.815542 (+ 113us) write_op.cc:270] Start()
0901 01:52:29.815571 (+ 29us) write_op.cc:276] Timestamp: P: 1756691544215037 usec, L: 0
0901 01:52:29.818947 (+ 3376us) write_op.cc:270] Start()
0901 01:52:29.818976 (+ 29us) write_op.cc:276] Timestamp: P: 1756691544232858 usec, L: 0
0901 01:52:29.819151 (+ 175us) write_op.cc:270] Start()
0901 01:52:29.819170 (+ 19us) write_op.cc:276] Timestamp: P: 1756691544432633 usec, L: 0
0901 01:52:29.819285 (+ 115us) write_op.cc:270] Start()
0901 01:52:29.819301 (+ 16us) write_op.cc:276] Timestamp: P: 1756691544445199 usec, L: 0
0901 01:52:29.819399 (+ 98us) write_op.cc:270] Start()
0901 01:52:29.819413 (+ 14us) write_op.cc:276] Timestamp: P: 1756691544505001 usec, L: 0
0901 01:52:29.819520 (+ 107us) write_op.cc:270] Start()
0901 01:52:29.819536 (+ 16us) write_op.cc:276] Timestamp: P: 1756691544511448 usec, L: 0
0901 01:52:29.819657 (+ 121us) write_op.cc:270] Start()
0901 01:52:29.819674 (+ 17us) write_op.cc:276] Timestamp: P: 1756691544518603 usec, L: 0
0901 01:52:29.819814 (+ 140us) write_op.cc:270] Start()
0901 01:52:29.819833 (+ 19us) write_op.cc:276] Timestamp: P: 1756691544583946 usec, L: 0
0901 01:52:29.819961 (+ 128us) write_op.cc:270] Start()
0901 01:52:29.819976 (+ 15us) write_op.cc:276] Timestamp: P: 1756691544625212 usec, L: 0
0901 01:52:29.820080 (+ 104us) write_op.cc:270] Start()
0901 01:52:29.820094 (+ 14us) write_op.cc:276] Timestamp: P: 1756691544629587 usec, L: 0
0901 01:52:29.820227 (+ 133us) write_op.cc:270] Start()
0901 01:52:29.820247 (+ 20us) write_op.cc:276] Timestamp: P: 1756691544630962 usec, L: 0
0901 01:52:29.832105 (+ 11858us) write_op.cc:270] Start()
0901 01:52:29.832149 (+ 44us) write_op.cc:276] Timestamp: P: 1756691544707557 usec, L: 0
0901 01:52:29.832346 (+ 197us) write_op.cc:270] Start()
0901 01:52:29.832364 (+ 18us) write_op.cc:276] Timestamp: P: 1756691544712287 usec, L: 0
0901 01:52:29.832503 (+ 139us) write_op.cc:270] Start()
0901 01:52:29.832541 (+ 38us) write_op.cc:276] Timestamp: P: 1756691544713117 usec, L: 0
0901 01:52:29.833747 (+ 1206us) log.cc:844] Serialized 28259 byte log entry
0901 01:52:29.850581 (+ 16834us) raft_consensus.cc:1643] Marking committed up to 669
0901 01:52:29.850760 (+ 179us) log.cc:844] Serialized 15 byte log entry
0901 01:52:29.864047 (+ 13287us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:29.864053 (+ 6us) write_op.cc:454] Released schema lock
0901 01:52:29.865075 (+ 1022us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:29.865080 (+ 5us) write_op.cc:454] Released schema lock
0901 01:52:29.883731 (+ 18651us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:29.883739 (+ 8us) write_op.cc:454] Released schema lock
0901 01:52:29.885284 (+ 1545us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:29.885289 (+ 5us) write_op.cc:454] Released schema lock
0901 01:52:29.886364 (+ 1075us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:29.886369 (+ 5us) write_op.cc:454] Released schema lock
0901 01:52:29.887672 (+ 1303us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:29.887677 (+ 5us) write_op.cc:454] Released schema lock
0901 01:52:29.888879 (+ 1202us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:29.888884 (+ 5us) write_op.cc:454] Released schema lock
0901 01:52:29.889520 (+ 636us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:29.889525 (+ 5us) write_op.cc:454] Released schema lock
0901 01:52:29.890305 (+ 780us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:29.890309 (+ 4us) write_op.cc:454] Released schema lock
0901 01:52:29.892092 (+ 1783us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:29.892098 (+ 6us) write_op.cc:454] Released schema lock
0901 01:52:29.893008 (+ 910us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:29.893014 (+ 6us) write_op.cc:454] Released schema lock
0901 01:52:29.893719 (+ 705us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:29.893724 (+ 5us) write_op.cc:454] Released schema lock
0901 01:52:29.895514 (+ 1790us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:29.895519 (+ 5us) write_op.cc:454] Released schema lock
0901 01:52:29.896101 (+ 582us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:29.896106 (+ 5us) write_op.cc:454] Released schema lock
0901 01:52:29.896656 (+ 550us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:29.896663 (+ 7us) write_op.cc:454] Released schema lock
0901 01:52:29.897701 (+ 1038us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:29.897707 (+ 6us) write_op.cc:454] Released schema lock
0901 01:52:29.898814 (+ 1107us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:29.898819 (+ 5us) write_op.cc:454] Released schema lock
0901 01:52:29.899672 (+ 853us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:29.899677 (+ 5us) write_op.cc:454] Released schema lock
0901 01:52:29.900300 (+ 623us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:29.900305 (+ 5us) write_op.cc:454] Released schema lock
0901 01:52:29.901195 (+ 890us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:29.901200 (+ 5us) write_op.cc:454] Released schema lock
0901 01:52:29.902318 (+ 1118us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:29.902323 (+ 5us) write_op.cc:454] Released schema lock
0901 01:52:29.904708 (+ 2385us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:29.904715 (+ 7us) write_op.cc:454] Released schema lock
0901 01:52:29.915343 (+ 10628us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:29.915352 (+ 9us) write_op.cc:454] Released schema lock
0901 01:52:29.918424 (+ 3072us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:29.918429 (+ 5us) write_op.cc:454] Released schema lock
0901 01:52:29.921344 (+ 2915us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:29.921350 (+ 6us) write_op.cc:454] Released schema lock
0901 01:52:30.021188 (+ 99838us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:30.021197 (+ 9us) write_op.cc:454] Released schema lock
0901 01:52:30.021481 (+ 284us) raft_consensus.cc:1695] Filling consensus response to leader.
0901 01:52:30.021546 (+ 65us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0901 01:52:30.021576 (+ 30us) raft_consensus.cc:1683] finished
0901 01:52:30.021581 (+ 5us) raft_consensus.cc:1689] UpdateReplicas() finished
0901 01:52:30.022618 (+ 1037us) spinlock_profiling.cc:243] Waited 297 ms on lock 0x616000080554. stack: 00007f58f21237d6 00007f58f2123590 00007f58f11163ad 0000000000972d94 0000000000972d5d 00007f5900eee14d 00007f5900eee243 00007f58fdbad941 00007f5905925ec5 00007f5905939d25 00007f5905939ad1 00007f58feeee32d 00007f58fdbbe74e 00007f58fdbbe49d 0000000000a774dc 00007f58f2169b5f
0901 01:52:30.022619 (+ 1us) spinlock_profiling.cc:243] Waited 159 ms on lock 0x616000080554. stack: 00007f58f21237d6 00007f58f2123590 00007f58f11163ad 0000000000972d94 0000000000972d5d 00007f5900eee14d 00007f5900eee243 00007f58fdbad941 00007f5905925ec5 00007f5905939d25 00007f5905939ad1 00007f58feeee32d 00007f58fdbbe74e 00007f58fdbbe49d 0000000000a774dc 00007f58f2169b5f
0901 01:52:30.025090 (+ 2471us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0901 01:52:29.787788 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:29.787920 (+ 132us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:29.787927 (+ 7us) write_op.cc:435] Acquired schema lock
0901 01:52:29.787929 (+ 2us) tablet.cc:660] Decoding operations
0901 01:52:29.788060 (+ 131us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:29.788098 (+ 38us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:29.788103 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0901 01:52:29.788170 (+ 67us) tablet.cc:699] Row locks acquired
0901 01:52:29.788173 (+ 3us) write_op.cc:260] PREPARE: finished
0901 01:52:29.863305 (+ 75132us) op_driver.cc:464] REPLICATION: finished
0901 01:52:29.863403 (+ 98us) write_op.cc:301] APPLY: starting
0901 01:52:29.863420 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:29.863463 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:29.863465 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:29.863524 (+ 59us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:29.863559 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0901 01:52:29.863592 (+ 33us) write_op.cc:312] APPLY: finished
0901 01:52:29.863674 (+ 82us) log.cc:844] Serialized 23 byte log entry
0901 01:52:29.863740 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:29.863761 (+ 21us) write_op.cc:454] Released schema lock
0901 01:52:29.863768 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0901 01:52:29.788237 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:29.788310 (+ 73us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:29.788315 (+ 5us) write_op.cc:435] Acquired schema lock
0901 01:52:29.788317 (+ 2us) tablet.cc:660] Decoding operations
0901 01:52:29.788383 (+ 66us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:29.788391 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:29.788395 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0901 01:52:29.788482 (+ 87us) tablet.cc:699] Row locks acquired
0901 01:52:29.788486 (+ 4us) write_op.cc:260] PREPARE: finished
0901 01:52:29.864139 (+ 75653us) op_driver.cc:464] REPLICATION: finished
0901 01:52:29.864457 (+ 318us) write_op.cc:301] APPLY: starting
0901 01:52:29.864470 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:29.864516 (+ 46us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:29.864519 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:29.864611 (+ 92us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:29.864665 (+ 54us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0901 01:52:29.864690 (+ 25us) write_op.cc:312] APPLY: finished
0901 01:52:29.864745 (+ 55us) log.cc:844] Serialized 29 byte log entry
0901 01:52:29.864816 (+ 71us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:29.864839 (+ 23us) write_op.cc:454] Released schema lock
0901 01:52:29.864844 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0901 01:52:29.788547 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:29.788615 (+ 68us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:29.788619 (+ 4us) write_op.cc:435] Acquired schema lock
0901 01:52:29.788622 (+ 3us) tablet.cc:660] Decoding operations
0901 01:52:29.788687 (+ 65us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:29.788699 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:29.788705 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
0901 01:52:29.788760 (+ 55us) tablet.cc:699] Row locks acquired
0901 01:52:29.788762 (+ 2us) write_op.cc:260] PREPARE: finished
0901 01:52:29.865155 (+ 76393us) op_driver.cc:464] REPLICATION: finished
0901 01:52:29.868190 (+ 3035us) write_op.cc:301] APPLY: starting
0901 01:52:29.868206 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:29.868237 (+ 31us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:29.868240 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:29.868327 (+ 87us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:29.868374 (+ 47us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0901 01:52:29.868434 (+ 60us) write_op.cc:312] APPLY: finished
0901 01:52:29.868499 (+ 65us) log.cc:844] Serialized 29 byte log entry
0901 01:52:29.868555 (+ 56us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:29.868578 (+ 23us) write_op.cc:454] Released schema lock
0901 01:52:29.868583 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0901 01:52:29.788814 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:29.788877 (+ 63us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:29.788880 (+ 3us) write_op.cc:435] Acquired schema lock
0901 01:52:29.788883 (+ 3us) tablet.cc:660] Decoding operations
0901 01:52:29.788972 (+ 89us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:29.788981 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:29.788985 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0901 01:52:29.789058 (+ 73us) tablet.cc:699] Row locks acquired
0901 01:52:29.789060 (+ 2us) write_op.cc:260] PREPARE: finished
0901 01:52:29.883880 (+ 94820us) op_driver.cc:464] REPLICATION: finished
0901 01:52:29.884623 (+ 743us) write_op.cc:301] APPLY: starting
0901 01:52:29.884641 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:29.884690 (+ 49us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:29.884693 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:29.884767 (+ 74us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:29.884818 (+ 51us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0901 01:52:29.884852 (+ 34us) write_op.cc:312] APPLY: finished
0901 01:52:29.884930 (+ 78us) log.cc:844] Serialized 29 byte log entry
0901 01:52:29.885005 (+ 75us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:29.885028 (+ 23us) write_op.cc:454] Released schema lock
0901 01:52:29.885035 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0901 01:52:29.789107 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:29.789192 (+ 85us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:29.789196 (+ 4us) write_op.cc:435] Acquired schema lock
0901 01:52:29.789198 (+ 2us) tablet.cc:660] Decoding operations
0901 01:52:29.789251 (+ 53us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:29.789259 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:29.789263 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0901 01:52:29.789295 (+ 32us) tablet.cc:699] Row locks acquired
0901 01:52:29.789297 (+ 2us) write_op.cc:260] PREPARE: finished
0901 01:52:29.885384 (+ 96087us) op_driver.cc:464] REPLICATION: finished
0901 01:52:29.885806 (+ 422us) write_op.cc:301] APPLY: starting
0901 01:52:29.885818 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:29.885850 (+ 32us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:29.885854 (+ 4us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:29.885898 (+ 44us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:29.885927 (+ 29us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0901 01:52:29.885967 (+ 40us) write_op.cc:312] APPLY: finished
0901 01:52:29.886029 (+ 62us) log.cc:844] Serialized 23 byte log entry
0901 01:52:29.886084 (+ 55us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:29.886096 (+ 12us) write_op.cc:454] Released schema lock
0901 01:52:29.886101 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0901 01:52:29.802788 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:29.802929 (+ 141us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:29.802936 (+ 7us) write_op.cc:435] Acquired schema lock
0901 01:52:29.802939 (+ 3us) tablet.cc:660] Decoding operations
0901 01:52:29.803076 (+ 137us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:29.803092 (+ 16us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:29.803099 (+ 7us) tablet.cc:683] Acquiring locks for 1 operations
0901 01:52:29.803168 (+ 69us) tablet.cc:699] Row locks acquired
0901 01:52:29.803170 (+ 2us) write_op.cc:260] PREPARE: finished
0901 01:52:29.886432 (+ 83262us) op_driver.cc:464] REPLICATION: finished
0901 01:52:29.887208 (+ 776us) write_op.cc:301] APPLY: starting
0901 01:52:29.887219 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:29.887245 (+ 26us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:29.887248 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:29.887283 (+ 35us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:29.887307 (+ 24us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0901 01:52:29.887338 (+ 31us) write_op.cc:312] APPLY: finished
0901 01:52:29.887395 (+ 57us) log.cc:844] Serialized 23 byte log entry
0901 01:52:29.887451 (+ 56us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:29.887466 (+ 15us) write_op.cc:454] Released schema lock
0901 01:52:29.887473 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0901 01:52:29.803255 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:29.803338 (+ 83us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:29.803342 (+ 4us) write_op.cc:435] Acquired schema lock
0901 01:52:29.803344 (+ 2us) tablet.cc:660] Decoding operations
0901 01:52:29.803431 (+ 87us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:29.803444 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:29.803447 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0901 01:52:29.803497 (+ 50us) tablet.cc:699] Row locks acquired
0901 01:52:29.803511 (+ 14us) write_op.cc:260] PREPARE: finished
0901 01:52:29.887735 (+ 84224us) op_driver.cc:464] REPLICATION: finished
0901 01:52:29.888432 (+ 697us) write_op.cc:301] APPLY: starting
0901 01:52:29.888443 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:29.888469 (+ 26us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:29.888471 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:29.888505 (+ 34us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:29.888527 (+ 22us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0901 01:52:29.888552 (+ 25us) write_op.cc:312] APPLY: finished
0901 01:52:29.888616 (+ 64us) log.cc:844] Serialized 23 byte log entry
0901 01:52:29.888662 (+ 46us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:29.888673 (+ 11us) write_op.cc:454] Released schema lock
0901 01:52:29.888679 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0901 01:52:29.803575 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:29.803649 (+ 74us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:29.803655 (+ 6us) write_op.cc:435] Acquired schema lock
0901 01:52:29.803657 (+ 2us) tablet.cc:660] Decoding operations
0901 01:52:29.803722 (+ 65us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:29.803744 (+ 22us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:29.803748 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0901 01:52:29.803787 (+ 39us) tablet.cc:699] Row locks acquired
0901 01:52:29.803789 (+ 2us) write_op.cc:260] PREPARE: finished
0901 01:52:29.888942 (+ 85153us) op_driver.cc:464] REPLICATION: finished
0901 01:52:29.889109 (+ 167us) write_op.cc:301] APPLY: starting
0901 01:52:29.889118 (+ 9us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:29.889142 (+ 24us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:29.889144 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:29.889175 (+ 31us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:29.889196 (+ 21us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0901 01:52:29.889227 (+ 31us) write_op.cc:312] APPLY: finished
0901 01:52:29.889271 (+ 44us) log.cc:844] Serialized 23 byte log entry
0901 01:52:29.889310 (+ 39us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:29.889328 (+ 18us) write_op.cc:454] Released schema lock
0901 01:52:29.889332 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0901 01:52:29.803871 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:29.803971 (+ 100us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:29.803975 (+ 4us) write_op.cc:435] Acquired schema lock
0901 01:52:29.803977 (+ 2us) tablet.cc:660] Decoding operations
0901 01:52:29.804048 (+ 71us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:29.804057 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:29.804061 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0901 01:52:29.804113 (+ 52us) tablet.cc:699] Row locks acquired
0901 01:52:29.804115 (+ 2us) write_op.cc:260] PREPARE: finished
0901 01:52:29.889594 (+ 85479us) op_driver.cc:464] REPLICATION: finished
0901 01:52:29.889803 (+ 209us) write_op.cc:301] APPLY: starting
0901 01:52:29.889813 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:29.889841 (+ 28us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:29.889844 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:29.889924 (+ 80us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:29.889972 (+ 48us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0901 01:52:29.889993 (+ 21us) write_op.cc:312] APPLY: finished
0901 01:52:29.890042 (+ 49us) log.cc:844] Serialized 29 byte log entry
0901 01:52:29.890092 (+ 50us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:29.890112 (+ 20us) write_op.cc:454] Released schema lock
0901 01:52:29.890117 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0901 01:52:29.804167 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:29.804232 (+ 65us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:29.804236 (+ 4us) write_op.cc:435] Acquired schema lock
0901 01:52:29.804238 (+ 2us) tablet.cc:660] Decoding operations
0901 01:52:29.804296 (+ 58us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:29.804304 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:29.804308 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0901 01:52:29.804352 (+ 44us) tablet.cc:699] Row locks acquired
0901 01:52:29.804354 (+ 2us) write_op.cc:260] PREPARE: finished
0901 01:52:29.890366 (+ 86012us) op_driver.cc:464] REPLICATION: finished
0901 01:52:29.891122 (+ 756us) write_op.cc:301] APPLY: starting
0901 01:52:29.891133 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:29.891166 (+ 33us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:29.891169 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:29.891272 (+ 103us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:29.891318 (+ 46us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0901 01:52:29.891340 (+ 22us) write_op.cc:312] APPLY: finished
0901 01:52:29.891387 (+ 47us) log.cc:844] Serialized 29 byte log entry
0901 01:52:29.891433 (+ 46us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:29.891446 (+ 13us) write_op.cc:454] Released schema lock
0901 01:52:29.891451 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0901 01:52:29.804398 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:29.804490 (+ 92us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:29.804494 (+ 4us) write_op.cc:435] Acquired schema lock
0901 01:52:29.804496 (+ 2us) tablet.cc:660] Decoding operations
0901 01:52:29.804574 (+ 78us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:29.804582 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:29.804587 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0901 01:52:29.804675 (+ 88us) tablet.cc:699] Row locks acquired
0901 01:52:29.804677 (+ 2us) write_op.cc:260] PREPARE: finished
0901 01:52:29.892169 (+ 87492us) op_driver.cc:464] REPLICATION: finished
0901 01:52:29.892249 (+ 80us) write_op.cc:301] APPLY: starting
0901 01:52:29.892259 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:29.892287 (+ 28us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:29.892290 (+ 3us) tablet.cc:1370] star
I20250901 01:52:30.122692 2113 rpcz_store.cc:276] 0901 01:52:25.130268 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.130305 (+ 37us) service_pool.cc:225] Handling call
0901 01:52:30.009463 (+4879158us) ts_tablet_manager.cc:1145] Deleting tablet 670d90a37b934579b353bb4e4820127b
0901 01:52:30.009474 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:30.067441 (+ 57967us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:30.091177 (+ 23736us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:30.092516 (+ 1339us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4878988}
I20250901 01:52:30.125226 2113 tablet_replica.cc:331] T 50e3da53433f4c14b57db8a733fa53bc P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:30.126013 2113 raft_consensus.cc:2241] T 50e3da53433f4c14b57db8a733fa53bc P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:30.128015 2518 raft_consensus.cc:491] T ad1a6be9fcc045d3bb4e71a2f6bd5ccd P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:30.128809 1993 raft_consensus.cc:491] T 33f637f6a8134215bb1995ec38440073 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:30.128988 1993 raft_consensus.cc:513] T 33f637f6a8134215bb1995ec38440073 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.129704 2113 raft_consensus.cc:2270] T 50e3da53433f4c14b57db8a733fa53bc P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:30.123423 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 670d90a37b934579b353bb4e4820127b (table default.Table_77728af9eee44f29a1e2131d5b75b7ca [id=a90d1f10531042c3bb7a196b8be61c5f]) successfully deleted
I20250901 01:52:30.102502 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 130) took 3545 ms. Trace:
W20250901 01:52:30.113597 859 leader_election.cc:343] T 5f52698fb56d472d9e91fb10cb10c69a P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:30.136606 859 leader_election.cc:304] T 5f52698fb56d472d9e91fb10cb10c69a P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:30.136761 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 7954393c8c1c4457ab8bb723c03a9c8b (table default.Table_7050639ebd0147959ac4a251f534eb96 [id=543f2074598d4c29972e564be1123fca]). Expected version 5 got 0
I20250901 01:52:30.137051 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 6c0abbfc8d904dce989ce51961e4237d (table default.Table_3f9422b08b0349b9a5466218f0fe1b06 [id=f0dfb3811e744bd4ba6d59195cebde6f]). Expected version 2 got 0
I20250901 01:52:30.137276 2323 raft_consensus.cc:2747] T 5f52698fb56d472d9e91fb10cb10c69a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:30.137414 1492 catalog_manager.cc:5582] T aec951f2cc84499b9162b9fa24d5440e P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: UNKNOWN } } }
W20250901 01:52:30.137791 859 leader_election.cc:343] T 2973b59bb1df41d18d9a5b6a29ae336f P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:30.137837 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet aec951f2cc84499b9162b9fa24d5440e (table default.Table_156d1feafd5246ff8ff1bb7b9ae444a5 [id=55c31cd42fac43e6b8eddc048f565d0a]). Expected version 1 got 0
I20250901 01:52:30.137897 859 leader_election.cc:304] T 2973b59bb1df41d18d9a5b6a29ae336f P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:30.138110 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet c6f7c525f59b49938b697de98d4b5835 (table default.Table_db61e055dbd7450898e82c4c0c4763be [id=aabca67f6f754820977ac9c4c20f3e4f]). Expected version 1 got 0
I20250901 01:52:30.138139 2408 raft_consensus.cc:2747] T 2973b59bb1df41d18d9a5b6a29ae336f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:30.138383 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet ea846f88b3534beebd0001b7f17b5ca9 (table default.Table_dd92add170f94aaf966f77d81e2c3883 [id=85b64046d2fa4f1d871ab43e5c800fb8]). Expected version 1 got 0
W20250901 01:52:30.138682 859 leader_election.cc:343] T 3239e58531d445fa80165cf678bc277c P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:30.138676 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet d1f2d98712ff4ea6aec665512de068bd (table default.Table_82d91a1ba2eb4265896824028dec8d3a [id=3f7624d2901445e39155c12fa68e34e9]). Expected version 1 got 0
I20250901 01:52:30.138871 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 3618a3eb4f174c43805ce7dc0e5315cb (table default.Table_83f2c0acedbc4c6982caf5313646e90b [id=31a2e50c32644321825da40176127015]). Expected version 1 got 0
I20250901 01:52:30.139016 859 leader_election.cc:304] T 3239e58531d445fa80165cf678bc277c P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:30.139084 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 33f637f6a8134215bb1995ec38440073 (table default.Table_3d5afc22c4e14d6990aea31564dfde20 [id=557afe29976d40f6bfb6ae4ca348f887]). Expected version 2 got 0
I20250901 01:52:30.139300 2323 raft_consensus.cc:2747] T 3239e58531d445fa80165cf678bc277c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:30.139298 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet d0c3c363dda444519a17dca63ad9caa7 (table default.Table_42422c44ca254d8ca989134c2433391f [id=8c1b263824ec4c15a5ec424364c78895]). Expected version 3 got 0
I20250901 01:52:30.139603 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 45f6e8ba99b74898a2922b36768c6c36 (table default.Table_938421db3cf74cf8b878089bac931818 [id=9ce26dcf307149b595a43b6ac4562fba]). Expected version 1 got 0
W20250901 01:52:30.139672 859 leader_election.cc:343] T 5f5396045979450193d461ba1334a5c9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:30.139869 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet e9f1f892658b4736859ff507536920a1 (table default.Table_8825dd28afbe464193eb1b668cc9326a [id=9aa62cb634314a0a849612c54c1220d9]). Expected version 1 got 0
I20250901 01:52:30.139868 859 leader_election.cc:304] T 5f5396045979450193d461ba1334a5c9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:30.140134 2323 raft_consensus.cc:2747] T 5f5396045979450193d461ba1334a5c9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:30.136569 2118 rpcz_store.cc:276] 0901 01:52:26.557317 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.558131 (+ 814us) service_pool.cc:225] Handling call
0901 01:52:29.993638 (+3435507us) ts_tablet_manager.cc:1145] Deleting tablet ddbc15685fca4c708329420109ad0c57
0901 01:52:29.993647 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:30.033770 (+ 40123us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:30.036610 (+ 2840us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:30.102483 (+ 65873us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3435315}
I20250901 01:52:30.140601 2118 tablet_replica.cc:331] T da5f69c5b0ca4257ad2ff7918ee2af2e P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:30.121888 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "615b2306fca84003b264da9d5f95ddab" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:30.140719 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet ddbc15685fca4c708329420109ad0c57 (table default.Table_942175dadf4b496195b2fa53f39f6ddc [id=2bd332b452f84f25af4225af8e6c6740]) successfully deleted
I20250901 01:52:30.140872 2118 raft_consensus.cc:2241] T da5f69c5b0ca4257ad2ff7918ee2af2e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:30.140915 916 raft_consensus.cc:3058] T 615b2306fca84003b264da9d5f95ddab P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:30.141021 2118 raft_consensus.cc:2270] T da5f69c5b0ca4257ad2ff7918ee2af2e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:30.141544 2118 ts_tablet_manager.cc:1905] T da5f69c5b0ca4257ad2ff7918ee2af2e P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:30.143520 916 raft_consensus.cc:2466] T 615b2306fca84003b264da9d5f95ddab P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:30.152415 2347 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "7ce2288f784044eea010664e3def133f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3add7edb97a40ef9f7c6bc011d13227" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 43871 } } peers { permanent_uuid: "d2ae2cb6af0f4b3a8f47ab1cdd6567d3" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42375 } } peers { permanent_uuid: "7ce2288f784044eea010664e3def133f" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 44231 } } }
I20250901 01:52:30.152937 2347 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f [sys.catalog]: This master's current role is: LEADER
I20250901 01:52:30.152635 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 44af4764d4f64e44aec84ccb84be6f5f (table default.Table_9896b980158948339a0b91f5882ed56b [id=a541b55be0dc420a8353ef4793b925b5]). Expected version 1 got 0
I20250901 01:52:30.156476 2113 ts_tablet_manager.cc:1905] T 50e3da53433f4c14b57db8a733fa53bc P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:30.156872 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 5f5396045979450193d461ba1334a5c9 (table default.Table_2b946a63bd574ac0971fa3d3b9c9a6a4 [id=9c3930e9fcfe4b49ba4bea67a2b0ad07]). Expected version 2 got 0
I20250901 01:52:30.157193 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 4cde5f4f02044b7bbd7d43b49aac4ae8 (table default.Table_d53a456dbca64daf92aeb5fb4517b417 [id=77ae3376281349e087a3b3cee41f2a09]). Expected version 1 got 0
I20250901 01:52:30.157879 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet b164b4a67a9143ceaff69b0ce49d63fe (table default.Table_da2c70a8f2fe49a0a8184e88c0a9e0ea [id=c1b1e9e3876244f68d31c63ce7bdebf1]). Expected version 1 got 0
I20250901 01:52:30.158049 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet ae21b9ae2e3b472eb67fb5f96388a50f (table default.Table_046289f393ff432aae0b033bdec1fb17 [id=db57b18a9ce24e068b362f5fbe940531]). Expected version 1 got 0
I20250901 01:52:30.158452 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 0e4009963cfe42d6ba164fc828897f09 (table default.Table_1adc70cccd2f4cb3a79df96861cb4181 [id=24fffc7ebf5a47bdb37211910301acf7]). Expected version 1 got 0
I20250901 01:52:30.158630 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet f80f13db501c42c7950499df8fcb6fc8 (table default.Table_ce86974fd8974de9a5a737954fac2ad7 [id=ff772d20b287476f94716379b86d2a71]). Expected version 1 got 0
I20250901 01:52:30.159902 2529 raft_consensus.cc:491] T c754129977904f978f87fd89d7092e36 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:30.160116 2409 raft_consensus.cc:491] T 1495de70f0844f82a0eea99c059f076b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:30.162894 1150 tablet_bootstrap.cc:492] T 36e10328c34242ebb4dbd8f294162696 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:30.163460 1150 ts_tablet_manager.cc:1397] T 36e10328c34242ebb4dbd8f294162696 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.072s user 0.010s sys 0.001s
I20250901 01:52:30.164288 1150 raft_consensus.cc:357] T 36e10328c34242ebb4dbd8f294162696 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.164599 1150 raft_consensus.cc:383] T 36e10328c34242ebb4dbd8f294162696 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250901 01:52:30.164583 992 consensus_peers.cc:489] T f65ec84958eb488d8b0f812b59b5b3d5 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250901 01:52:30.164759 1150 raft_consensus.cc:738] T 36e10328c34242ebb4dbd8f294162696 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:30.165068 1150 consensus_queue.cc:260] T 36e10328c34242ebb4dbd8f294162696 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.165810 1150 ts_tablet_manager.cc:1428] T 36e10328c34242ebb4dbd8f294162696 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250901 01:52:30.166273 1150 tablet_bootstrap.cc:492] T 44af4764d4f64e44aec84ccb84be6f5f P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:30.167922 1150 tablet_bootstrap.cc:654] T 44af4764d4f64e44aec84ccb84be6f5f P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:30.168275 2105 ts_manager.cc:194] Registered new tserver with Master: 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:30.171819 1993 leader_election.cc:290] T 33f637f6a8134215bb1995ec38440073 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:30.173266 2575 raft_consensus.cc:491] T 1a60c10e42104502ac2d3b80b8126ceb P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:30.173492 2575 raft_consensus.cc:513] T 1a60c10e42104502ac2d3b80b8126ceb P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.174345 2575 leader_election.cc:290] T 1a60c10e42104502ac2d3b80b8126ceb P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:30.174582 2529 raft_consensus.cc:513] T c754129977904f978f87fd89d7092e36 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.175244 2529 leader_election.cc:290] T c754129977904f978f87fd89d7092e36 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:30.175434 2409 raft_consensus.cc:513] T 1495de70f0844f82a0eea99c059f076b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.176064 2409 leader_election.cc:290] T 1495de70f0844f82a0eea99c059f076b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:30.176280 2545 raft_consensus.cc:491] T 48a211e174054866b030cdfb0cf947f4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:30.176477 2545 raft_consensus.cc:513] T 48a211e174054866b030cdfb0cf947f4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.177186 2545 leader_election.cc:290] T 48a211e174054866b030cdfb0cf947f4 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:30.177274 1150 tablet_bootstrap.cc:492] T 44af4764d4f64e44aec84ccb84be6f5f P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:30.177383 1150 ts_tablet_manager.cc:1397] T 44af4764d4f64e44aec84ccb84be6f5f P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.001s
I20250901 01:52:30.177448 2518 raft_consensus.cc:513] T ad1a6be9fcc045d3bb4e71a2f6bd5ccd P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.178961 1151 tablet_bootstrap.cc:492] T b81a12b38ac64efabd44aeb192d181a0 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:30.179095 1151 ts_tablet_manager.cc:1397] T b81a12b38ac64efabd44aeb192d181a0 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.124s user 0.007s sys 0.015s
I20250901 01:52:30.179836 1151 raft_consensus.cc:357] T b81a12b38ac64efabd44aeb192d181a0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.179991 1151 raft_consensus.cc:383] T b81a12b38ac64efabd44aeb192d181a0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:30.180095 1151 raft_consensus.cc:738] T b81a12b38ac64efabd44aeb192d181a0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:30.180356 1151 consensus_queue.cc:260] T b81a12b38ac64efabd44aeb192d181a0 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.183152 1151 ts_tablet_manager.cc:1428] T b81a12b38ac64efabd44aeb192d181a0 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250901 01:52:30.182716 2518 leader_election.cc:290] T ad1a6be9fcc045d3bb4e71a2f6bd5ccd P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:30.183907 2118 ts_tablet_manager.cc:1918] T da5f69c5b0ca4257ad2ff7918ee2af2e P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:30.184098 1151 tablet_bootstrap.cc:492] T 31614db4c53a4f3c96fd190d326ce601 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:30.159387 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 93bcac1fd95d44dc927abfca81692921 (table default.Table_b8f8ee74648c406eaf589d3ab80934db [id=b8f69313439543be9c11eea88dfa69c5]). Expected version 1 got 0
I20250901 01:52:30.189381 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 508406601c994e42a5c5b1f665e74957 (table default.Table_345b7f9fe8214aa6878f20a166125364 [id=4aabc328f6374ec5b1dea26f778208b2]). Expected version 1 got 0
I20250901 01:52:30.189649 2118 log.cc:1199] T da5f69c5b0ca4257ad2ff7918ee2af2e P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/da5f69c5b0ca4257ad2ff7918ee2af2e
I20250901 01:52:30.190076 2118 ts_tablet_manager.cc:1939] T da5f69c5b0ca4257ad2ff7918ee2af2e P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:30.187211 1150 raft_consensus.cc:357] T 44af4764d4f64e44aec84ccb84be6f5f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.190590 1150 raft_consensus.cc:383] T 44af4764d4f64e44aec84ccb84be6f5f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:30.193240 1150 raft_consensus.cc:738] T 44af4764d4f64e44aec84ccb84be6f5f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:30.193518 1150 consensus_queue.cc:260] T 44af4764d4f64e44aec84ccb84be6f5f P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.194211 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 5d68ea97c71942c7b55b967ba870e0d7 (table default.Table_f610cb75906d4c5cb3b57cfdb80ecefc [id=36915a3e843340ba8ef3578ac35a9af4]). Expected version 1 got 0
I20250901 01:52:30.194542 1151 tablet_bootstrap.cc:654] T 31614db4c53a4f3c96fd190d326ce601 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:30.195715 1150 ts_tablet_manager.cc:1428] T 44af4764d4f64e44aec84ccb84be6f5f P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
W20250901 01:52:30.195988 992 consensus_peers.cc:489] T 606fac5c6efe48d1877104b01c23ba5b P fd27ab6665304aaea53adbd102dd4f58 -> Peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Couldn't send request to peer 00c282f779c1477d920f24dc0dc16628. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 606fac5c6efe48d1877104b01c23ba5b. This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:30.196028 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet d000ba05e12843b4925e622ab55471c0 (table default.Table_e0902dfbdd984f85b3fdac8d283731f7 [id=b43a92ed46164d199b445287ef039bc6]). Expected version 1 got 0
I20250901 01:52:30.196394 2613 raft_consensus.cc:491] T a9b7f2e6548043a7865e25387cdd06c9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:30.196518 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 2973b59bb1df41d18d9a5b6a29ae336f (table default.Table_7790af96855e45c0a316ba3a4a87c9ed [id=746845bd84c6444b8a1e5683eaba4a68]). Expected version 1 got 0
I20250901 01:52:30.196610 992 leader_election.cc:304] T 615b2306fca84003b264da9d5f95ddab P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:30.196589 2613 raft_consensus.cc:513] T a9b7f2e6548043a7865e25387cdd06c9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.196883 1761 raft_consensus.cc:2802] T 615b2306fca84003b264da9d5f95ddab P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:30.197019 1761 raft_consensus.cc:695] T 615b2306fca84003b264da9d5f95ddab P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:30.197248 2613 leader_election.cc:290] T a9b7f2e6548043a7865e25387cdd06c9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:30.197252 1761 consensus_queue.cc:237] T 615b2306fca84003b264da9d5f95ddab P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.197580 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 615b2306fca84003b264da9d5f95ddab (table default.Table_302f6e7cf97d46b9a621eb852e6aefe0 [id=cecdd8d6ca6d445c9fd82decd2a95366]). Expected version 2 got 0
I20250901 01:52:30.198354 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet ad45c90ba0da43c9860745e534664a98 (table default.Table_1cd3f3fb6b244b3f8796bfca9f7622c8 [id=6fd739b0e725424ca844dbb652ea8028]). Expected version 2 got 0
I20250901 01:52:30.198789 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 8ca5cf4e99014c0d84a3acad4ef713af (table default.Table_ed6fd284be6f4ae3a008e21e7ce17930 [id=303e76c918bc4c39aa47a4d24849a358]). Expected version 1 got 0
I20250901 01:52:30.199082 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 61ddcf86c9f34726939cc116434f0822 (table default.Table_63d0f2ead9d946d39ce40162ebd769d6 [id=f9dcf4937b1143248bfa9d88053fce17]). Expected version 3 got 0
I20250901 01:52:30.199492 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 858218c4427c41e980b2b5c969a5af40 (table default.Table_6d47266af0ed433eb1f2abbc81d18e69 [id=264cc469a73841e79538843479a165ab]). Expected version 1 got 0
I20250901 01:52:30.199694 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 89d40f0331114490b51de3ba6d773571 (table default.Table_4dacb3b1d2e94e42ac8937eb6eb58375 [id=8ee0aa1ba24042dd8ded004644913b9c]). Expected version 1 got 0
I20250901 01:52:30.200511 1492 catalog_manager.cc:5582] T 482a9a2cb81f448bb2823892e35920d4 P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: UNKNOWN } } }
I20250901 01:52:30.205031 919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "33f637f6a8134215bb1995ec38440073" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:30.205204 919 raft_consensus.cc:2466] T 33f637f6a8134215bb1995ec38440073 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
W20250901 01:52:30.206051 992 leader_election.cc:336] T 615b2306fca84003b264da9d5f95ddab P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:30.209137 917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1495de70f0844f82a0eea99c059f076b" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:30.209313 917 raft_consensus.cc:2466] T 1495de70f0844f82a0eea99c059f076b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:30.212147 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a9b7f2e6548043a7865e25387cdd06c9" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:30.212282 913 raft_consensus.cc:2466] T a9b7f2e6548043a7865e25387cdd06c9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:30.212388 2113 ts_tablet_manager.cc:1918] T 50e3da53433f4c14b57db8a733fa53bc P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:30.212543 2113 log.cc:1199] T 50e3da53433f4c14b57db8a733fa53bc P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/50e3da53433f4c14b57db8a733fa53bc
I20250901 01:52:30.212985 2113 ts_tablet_manager.cc:1939] T 50e3da53433f4c14b57db8a733fa53bc P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:30.213716 2106 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" instance_seqno: 1756691527859899) as {username='slave'} at 127.0.104.3:54539; Asking this server to re-register.
I20250901 01:52:30.216477 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 132) took 3637 ms. Trace:
I20250901 01:52:30.216567 2118 rpcz_store.cc:276] 0901 01:52:26.578616 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.580474 (+ 1858us) service_pool.cc:225] Handling call
0901 01:52:30.140545 (+3560071us) ts_tablet_manager.cc:1145] Deleting tablet da5f69c5b0ca4257ad2ff7918ee2af2e
0901 01:52:30.140560 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:30.148965 (+ 8405us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:30.183799 (+ 34834us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:30.216461 (+ 32662us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3559838}
I20250901 01:52:30.216869 2118 tablet_replica.cc:331] T 8ed6616ce54548ba83ba299c4ecf3c38 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:30.216876 2608 raft_consensus.cc:491] T 435b1cca5a7d4d49959a310cb250b2a0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:30.216998 2608 raft_consensus.cc:513] T 435b1cca5a7d4d49959a310cb250b2a0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.217069 2118 raft_consensus.cc:2241] T 8ed6616ce54548ba83ba299c4ecf3c38 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:30.217044 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 616ff34e8d314e0aa0e11ce60be11789 (table default.Table_0f4b4094d2974b7591875375eda6bed0 [id=5ee13d46e06a4fa59b1070547573e523]). Expected version 2 got 1
I20250901 01:52:30.217219 2118 raft_consensus.cc:2270] T 8ed6616ce54548ba83ba299c4ecf3c38 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:30.217449 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 81ce448b148447ddbc41875d863e0518 (table default.Table_6e461d9cb0ae4dd095975bbc21533794 [id=49673fa73f8c436e9b8026efc4f3cbbc]). Expected version 1 got 0
I20250901 01:52:30.217597 2608 leader_election.cc:290] T 435b1cca5a7d4d49959a310cb250b2a0 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:30.217725 2118 ts_tablet_manager.cc:1905] T 8ed6616ce54548ba83ba299c4ecf3c38 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:30.219087 1485 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b1ba0e19dc0b462e80c537bf2d183f5b has no consensus state
I20250901 01:52:30.219122 1490 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet b8ca4e66c6a74f9d8d90a69be5d6a58f
I20250901 01:52:30.219257 1489 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 4abf9725be5d42f3893f07f0a9c7614c
W20250901 01:52:30.219317 1490 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet b8ca4e66c6a74f9d8d90a69be5d6a58f: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:30.219394 1482 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 629db80a8ab444e582628578155638fd
W20250901 01:52:30.219585 1482 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 629db80a8ab444e582628578155638fd: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
W20250901 01:52:30.219718 1488 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 14b466b2a5a249aa841f1b8b06301edc consensus state has no leader
I20250901 01:52:30.219902 1490 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_1cd3f3fb6b244b3f8796bfca9f7622c8" } new_table_name: "default.Table_a920a3ffbabf4d83a6563c3423cde12a" modify_external_catalogs: true
I20250901 01:52:30.220021 1488 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_93afd37289414655980b22934695bd55" } modify_external_catalogs: true
I20250901 01:52:30.220180 1488 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_93afd37289414655980b22934695bd55" } modify_external_catalogs: true
I20250901 01:52:30.220222 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7444cdd5505444ea892fefd437c2ffdd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.220348 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_243e5ccef83d4b68a14a87b410f865d5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:30.221225 1491 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c83ce95fd6fa41478de0e2168be201b1 consensus state has no leader
I20250901 01:52:30.221132 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_bf67319cd2ce4ae2b559e917d411ccfd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.221546 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8e1539858bbd4ad38e1c03be60017497"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.221581 1485 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_9d8c1fb24bc24631a2ba2b34a395f24a" } modify_external_catalogs: true
I20250901 01:52:30.221764 1485 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_9d8c1fb24bc24631a2ba2b34a395f24a" } modify_external_catalogs: true
I20250901 01:52:30.221915 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_734b1c4824cb45358816f239544284fc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.222218 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4932719cacd9459fbc580dbc69f10e0d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.222535 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f7eca1e6f9564f7ca5243cff0410c697"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.222817 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_db74d5e57c5543cda95dc8497b9b2862"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.223261 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7eef66c6e75740f1b75857363b102a02"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.223897 1491 master_service.cc:948] ReplaceTablet: received request to replace tablet 26c8ed083cea4dc0acca80d00b11b29b from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:30.224798 1489 master_service.cc:948] ReplaceTablet: received request to replace tablet f80f13db501c42c7950499df8fcb6fc8 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:30.225999 1150 tablet_bootstrap.cc:492] T 3363fd67b1ad44849ecb7b4090b61a72 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:30.228112 969 heartbeater.cc:499] Master 127.0.104.60:44231 was elected leader, sending a full tablet report...
I20250901 01:52:30.229527 1150 tablet_bootstrap.cc:654] T 3363fd67b1ad44849ecb7b4090b61a72 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:30.229985 1482 master_service.cc:948] ReplaceTablet: received request to replace tablet 606fac5c6efe48d1877104b01c23ba5b from {username='slave'} at 127.0.0.1:33478
W20250901 01:52:30.230882 858 leader_election.cc:336] T 435b1cca5a7d4d49959a310cb250b2a0 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:30.231259 2552 raft_consensus.cc:491] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7ce2288f784044eea010664e3def133f)
I20250901 01:52:30.231508 2552 raft_consensus.cc:513] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3add7edb97a40ef9f7c6bc011d13227" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 43871 } } peers { permanent_uuid: "d2ae2cb6af0f4b3a8f47ab1cdd6567d3" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42375 } } peers { permanent_uuid: "7ce2288f784044eea010664e3def133f" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 44231 } }
I20250901 01:52:30.233739 2552 leader_election.cc:290] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d2ae2cb6af0f4b3a8f47ab1cdd6567d3 (127.0.104.61:42375), 7ce2288f784044eea010664e3def133f (127.0.104.60:44231)
I20250901 01:52:30.221138 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2771be40c99745818735d3033de4bd93"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.234742 1483 master_service.cc:948] ReplaceTablet: received request to replace tablet f4fd117c395e41c2ab679c684908c5b7 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:30.236780 1488 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet cc3b8736b9094d7e892c80da4d5b20f6
I20250901 01:52:30.237046 1488 master_service.cc:948] ReplaceTablet: received request to replace tablet 081e2630e284498aa819f85bf8326315 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:30.237638 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 120) took 5107 ms. Trace:
I20250901 01:52:30.237777 2113 rpcz_store.cc:276] 0901 01:52:25.130643 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.130684 (+ 41us) service_pool.cc:225] Handling call
0901 01:52:30.123648 (+4992964us) ts_tablet_manager.cc:1145] Deleting tablet 50e3da53433f4c14b57db8a733fa53bc
0901 01:52:30.123662 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:30.161602 (+ 37940us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:30.185130 (+ 23528us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:30.237595 (+ 52465us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4992799}
I20250901 01:52:30.238193 2113 tablet_replica.cc:331] T f8a5886ddb644a86b76df44aab309057 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:30.238190 534 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e3add7edb97a40ef9f7c6bc011d13227" candidate_term: 5 candidate_status { last_received { term: 4 index: 671 } } ignore_live_leader: false dest_uuid: "d2ae2cb6af0f4b3a8f47ab1cdd6567d3" is_pre_election: true
I20250901 01:52:30.238407 2113 raft_consensus.cc:2241] T f8a5886ddb644a86b76df44aab309057 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:30.238819 2113 raft_consensus.cc:2270] T f8a5886ddb644a86b76df44aab309057 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:30.238835 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 50e3da53433f4c14b57db8a733fa53bc (table default.Table_4ce20bf6ad224378861e083f6a480edf [id=cd8eab7eafac490c896a2a9be18e426f]) successfully deleted
I20250901 01:52:30.239526 2113 ts_tablet_manager.cc:1905] T f8a5886ddb644a86b76df44aab309057 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:30.213613 992 leader_election.cc:336] T 33f637f6a8134215bb1995ec38440073 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:30.215169 2518 raft_consensus.cc:491] T ed1b41340ce742a88c3e4fd85cd25ca6 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:30.228626 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet b8ca4e66c6a74f9d8d90a69be5d6a58f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:29 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:30.228796 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 629db80a8ab444e582628578155638fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:29 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:30.240545 2518 raft_consensus.cc:513] T ed1b41340ce742a88c3e4fd85cd25ca6 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.241197 2518 leader_election.cc:290] T ed1b41340ce742a88c3e4fd85cd25ca6 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:30.243892 992 leader_election.cc:304] T 33f637f6a8134215bb1995ec38440073 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:30.244144 2518 raft_consensus.cc:2802] T 33f637f6a8134215bb1995ec38440073 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:30.244196 1484 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: default.Table_b3e694397be14321a668e46c09c51a71 [id=8199118617cb4707bc3ae9b1363750f7] alter complete (version 3)
I20250901 01:52:30.244257 2518 raft_consensus.cc:491] T 33f637f6a8134215bb1995ec38440073 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:30.244361 2518 raft_consensus.cc:3058] T 33f637f6a8134215bb1995ec38440073 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:30.247164 2518 raft_consensus.cc:513] T 33f637f6a8134215bb1995ec38440073 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.247664 2518 leader_election.cc:290] T 33f637f6a8134215bb1995ec38440073 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:30.248003 992 leader_election.cc:343] T c754129977904f978f87fd89d7092e36 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: c754129977904f978f87fd89d7092e36
I20250901 01:52:30.248194 992 leader_election.cc:304] T 1495de70f0844f82a0eea99c059f076b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:30.248454 2518 raft_consensus.cc:2802] T 1495de70f0844f82a0eea99c059f076b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:30.248533 2518 raft_consensus.cc:491] T 1495de70f0844f82a0eea99c059f076b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:30.248626 2518 raft_consensus.cc:3058] T 1495de70f0844f82a0eea99c059f076b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:30.251343 2518 raft_consensus.cc:513] T 1495de70f0844f82a0eea99c059f076b P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.252084 2518 leader_election.cc:290] T 1495de70f0844f82a0eea99c059f076b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:30.259014 2118 ts_tablet_manager.cc:1918] T 8ed6616ce54548ba83ba299c4ecf3c38 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:30.259153 2118 log.cc:1199] T 8ed6616ce54548ba83ba299c4ecf3c38 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/8ed6616ce54548ba83ba299c4ecf3c38
I20250901 01:52:30.259524 2118 ts_tablet_manager.cc:1939] T 8ed6616ce54548ba83ba299c4ecf3c38 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:30.259886 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 133) took 3659 ms. Trace:
I20250901 01:52:30.259970 2118 rpcz_store.cc:276] 0901 01:52:26.600627 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.600781 (+ 154us) service_pool.cc:225] Handling call
0901 01:52:30.216826 (+3616045us) ts_tablet_manager.cc:1145] Deleting tablet 8ed6616ce54548ba83ba299c4ecf3c38
0901 01:52:30.216834 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:30.225308 (+ 8474us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:30.258872 (+ 33564us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:30.259865 (+ 993us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3615767}
E20250901 01:52:30.264102 1696 catalog_manager.cc:6205] Error selecting replicas for tablet a656adbf1b29431687e0989f0db3459c, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2d3d2227e6a24bd2a86f457e553e698b'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:30.264243 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 494b6488aa0d4355b4caa46004267465, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_eecfab7239cc40c5a2f1fb6ac6d35dc9'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:30.264358 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 0ee3f35392954817b350e5f48653be78, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_442e11b9211c4b209c68f653d7bfc313'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:30.264487 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae4844183a3c4a04a775e6fd878391a3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_aee74198b51d4f089538c4f2bf3ce4dc'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:30.264609 1696 catalog_manager.cc:6205] Error selecting replicas for tablet be8dfbc466564842a2fdb4953eee939b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_75ba3ba70ca24a94920e7bce96190ff2'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:30.264717 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9745582cebfe47b58c21d0191ab4cb9b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77728af9eee44f29a1e2131d5b75b7ca'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:30.264842 1696 catalog_manager.cc:6205] Error selecting replicas for tablet e2ed32b3292141278af28026528e96c2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b5077f55951d48be92a67407ffe26a08'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:30.264921 1696 catalog_manager.cc:6205] Error selecting replicas for tablet dcc60bb000f440f1aca9133928b5ce20, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2fbca8463d6546c9b8f5b018fc60251d'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:30.265015 1696 catalog_manager.cc:6205] Error selecting replicas for tablet a2ca69fabadd4ad69e1670cae7a0c503, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_32c4716e3aa24a87a6b1472984b255c4'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:30.265115 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f8f923ddcd2740bcb7e521f4f57c4034, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_5e65c5f068b84995a2d7a76fa65dda0f'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:30.265228 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 0813cc68d4b0488ba60a523e4479971b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d1302673d68f424ebb54b220911ce385'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:30.265336 1696 catalog_manager.cc:6205] Error selecting replicas for tablet b1ba0e19dc0b462e80c537bf2d183f5b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_66b111271c15410aab23682f8f6081ac'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:30.265484 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 8e10a01e77d14cf28c2c5e0205e595b9, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_83cf2d86a1284544ba2e7aa9ebb66bb7'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:30.265592 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5f0e054f62de4cf6bc6f69ad6664b192, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ef6832d0845044ed9f165ca3541b4c3f'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:30.265712 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 787be682c0e44248a80d7037fb61dc6c, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7c12d66048a6470e8c227dc1d9f45fa4'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:30.265834 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae9a4ad4fdfb4771ba23a91f7007447e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_5e84ae9a9c2a4084a1f9c5ffb4594de2'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:30.268673 887 tablet_service.cc:1515] Processing DeleteTablet for tablet b8ca4e66c6a74f9d8d90a69be5d6a58f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:29 UTC) from {username='slave'} at 127.0.0.1:34472
E20250901 01:52:30.265949 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9074543dea9949e7a085717b40913348, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_53e30205c5fd488eb3b48c685a159d04'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:30.270829 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 629db80a8ab444e582628578155638fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:29 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:30.271458 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet da5f69c5b0ca4257ad2ff7918ee2af2e (table default.Table_ecb9e8f638a043cc87c38afbebce250f [id=61f1e8a842844ed899e5913c76f362ff]) successfully deleted
I20250901 01:52:30.271790 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_324777fa5a8e46eba1b98b8f40aae388"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.272336 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_21abc8c96c174461941957505222b5d1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.271488 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f167062c90b247dfbb2aac549048ae85"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.273813 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_81bdae07b9eb4929837cbbbbb70b4a4b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.274540 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_6557f34b0b6f48e5a88283f5ee6c4c3d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.275179 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e02ab59c8faf499bb19b51d75d5bfb7f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.275890 2608 raft_consensus.cc:491] T f195b040a6ae4b7b83a8308ab0a12b37 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:30.276015 2608 raft_consensus.cc:513] T f195b040a6ae4b7b83a8308ab0a12b37 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.276631 2608 leader_election.cc:290] T f195b040a6ae4b7b83a8308ab0a12b37 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:30.277776 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_252312bcd4a2481f9e16c5039bfdfb69"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.279597 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_37f9ed16e8de463cb98c5ef7729342dd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:30.279907 858 leader_election.cc:336] T f195b040a6ae4b7b83a8308ab0a12b37 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:30.280342 1151 tablet_bootstrap.cc:492] T 31614db4c53a4f3c96fd190d326ce601 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:30.280318 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3113918045494afcbb3ca544ac505848"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.280599 1151 ts_tablet_manager.cc:1397] T 31614db4c53a4f3c96fd190d326ce601 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.097s user 0.010s sys 0.000s
I20250901 01:52:30.280871 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_24b45bd8f50b4b7c88db046cc6a13978"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:30.281358 992 leader_election.cc:343] T 1a60c10e42104502ac2d3b80b8126ceb P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 1a60c10e42104502ac2d3b80b8126ceb
W20250901 01:52:30.283155 992 leader_election.cc:343] T 48a211e174054866b030cdfb0cf947f4 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 48a211e174054866b030cdfb0cf947f4
W20250901 01:52:30.283380 992 leader_election.cc:343] T ad1a6be9fcc045d3bb4e71a2f6bd5ccd P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: ad1a6be9fcc045d3bb4e71a2f6bd5ccd
I20250901 01:52:30.283691 992 leader_election.cc:304] T a9b7f2e6548043a7865e25387cdd06c9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:30.283914 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_6807e18329ae484a8f8a918c4713d3a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:30.270694 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 3a62034e688348ed9e2ac9de009ed3a2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_409dad98d7eb455a8e16edfb4acedb34'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:30.284466 1761 raft_consensus.cc:2802] T a9b7f2e6548043a7865e25387cdd06c9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:30.284574 1761 raft_consensus.cc:491] T a9b7f2e6548043a7865e25387cdd06c9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:30.284677 1761 raft_consensus.cc:3058] T a9b7f2e6548043a7865e25387cdd06c9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:30.284705 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ed1b41340ce742a88c3e4fd85cd25ca6" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:30.284806 1151 raft_consensus.cc:357] T 31614db4c53a4f3c96fd190d326ce601 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.284926 913 raft_consensus.cc:2466] T ed1b41340ce742a88c3e4fd85cd25ca6 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:30.284996 1151 raft_consensus.cc:383] T 31614db4c53a4f3c96fd190d326ce601 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:30.285064 1151 raft_consensus.cc:738] T 31614db4c53a4f3c96fd190d326ce601 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:30.285265 1151 consensus_queue.cc:260] T 31614db4c53a4f3c96fd190d326ce601 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.286170 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "33f637f6a8134215bb1995ec38440073" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:30.286324 913 raft_consensus.cc:3058] T 33f637f6a8134215bb1995ec38440073 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:30.286351 1151 ts_tablet_manager.cc:1428] T 31614db4c53a4f3c96fd190d326ce601 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250901 01:52:30.286662 1151 tablet_bootstrap.cc:492] T 6c0abbfc8d904dce989ce51961e4237d P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:30.287420 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d5ae28856ac24694b4689220830dd88f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.287653 1761 raft_consensus.cc:513] T a9b7f2e6548043a7865e25387cdd06c9 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.288237 1761 leader_election.cc:290] T a9b7f2e6548043a7865e25387cdd06c9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:30.288216 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a28f237638b947969af8f03da3b7c707"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.288311 1151 tablet_bootstrap.cc:654] T 6c0abbfc8d904dce989ce51961e4237d P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:30.288704 1150 tablet_bootstrap.cc:492] T 3363fd67b1ad44849ecb7b4090b61a72 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:30.288828 1150 ts_tablet_manager.cc:1397] T 3363fd67b1ad44849ecb7b4090b61a72 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.063s user 0.007s sys 0.000s
I20250901 01:52:30.288990 913 raft_consensus.cc:2466] T 33f637f6a8134215bb1995ec38440073 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:30.289534 1150 raft_consensus.cc:357] T 3363fd67b1ad44849ecb7b4090b61a72 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.289747 1150 raft_consensus.cc:383] T 3363fd67b1ad44849ecb7b4090b61a72 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:30.289840 1150 raft_consensus.cc:738] T 3363fd67b1ad44849ecb7b4090b61a72 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:30.290117 1150 consensus_queue.cc:260] T 3363fd67b1ad44849ecb7b4090b61a72 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.294572 1150 ts_tablet_manager.cc:1428] T 3363fd67b1ad44849ecb7b4090b61a72 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250901 01:52:30.294872 1150 tablet_bootstrap.cc:492] T 508406601c994e42a5c5b1f665e74957 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:30.293103 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1495de70f0844f82a0eea99c059f076b" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:30.295217 913 raft_consensus.cc:3058] T 1495de70f0844f82a0eea99c059f076b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:30.296938 1150 tablet_bootstrap.cc:654] T 508406601c994e42a5c5b1f665e74957 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:30.297508 913 raft_consensus.cc:2466] T 1495de70f0844f82a0eea99c059f076b P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:30.299089 2113 ts_tablet_manager.cc:1918] T f8a5886ddb644a86b76df44aab309057 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:30.299237 2113 log.cc:1199] T f8a5886ddb644a86b76df44aab309057 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/f8a5886ddb644a86b76df44aab309057
I20250901 01:52:30.299675 2113 ts_tablet_manager.cc:1939] T f8a5886ddb644a86b76df44aab309057 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:30.300472 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 121) took 5134 ms. Trace:
I20250901 01:52:30.300616 2113 rpcz_store.cc:276] 0901 01:52:25.166226 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.166348 (+ 122us) service_pool.cc:225] Handling call
0901 01:52:30.238128 (+5071780us) ts_tablet_manager.cc:1145] Deleting tablet f8a5886ddb644a86b76df44aab309057
0901 01:52:30.238139 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:30.292452 (+ 54313us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:30.298963 (+ 6511us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:30.300386 (+ 1423us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5064060}
I20250901 01:52:30.300969 2113 tablet_replica.cc:331] T 1286c0ab85704837a99b6e5a62184d77 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:30.301187 2113 raft_consensus.cc:2241] T 1286c0ab85704837a99b6e5a62184d77 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:30.301496 2113 raft_consensus.cc:2270] T 1286c0ab85704837a99b6e5a62184d77 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:30.303280 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet f8a5886ddb644a86b76df44aab309057 (table default.Table_79a6a0f0c6944313b9987f23b6a09557 [id=3f3723927dc04cf99499f68d39db8038]) successfully deleted
I20250901 01:52:30.303404 2518 raft_consensus.cc:491] T 56475ba491a24301a534304811f30140 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:30.303551 2518 raft_consensus.cc:513] T 56475ba491a24301a534304811f30140 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:30.304317 2518 leader_election.cc:290] T 56475ba491a24301a534304811f30140 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:30.304895 2113 ts_tablet_manager.cc:1905] T 1286c0ab85704837a99b6e5a62184d77 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:30.305136 1055 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "435b1cca5a7d4d49959a310cb250b2a0" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:30.305532 1054 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f195b040a6ae4b7b83a8308ab0a12b37" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:30.307087 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_0d9ee65a12224e5eba0bdb9833c383c9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.307348 1104 heartbeater.cc:461] Registering TS with master...
I20250901 01:52:30.307969 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5f89bb2ef9e546e0ae84b874f65f648b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.311535 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ad061141e6c841d69a443f63eb233e76"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:30.311818 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 18498c9cb09141d08c1c5c0854b63ebd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77f4810bc83c4d948531da2a45a3792e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:30.311957 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f5e3316adeae47cd848833dd84b972bb, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_af5338b09bbc4c879c7f4f573c32fd8d'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:30.313478 1150 tablet_bootstrap.cc:492] T 508406601c994e42a5c5b1f665e74957 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:30.313704 1150 ts_tablet_manager.cc:1397] T 508406601c994e42a5c5b1f665e74957 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.002s
I20250901 01:52:30.313951 2118 tablet_replica.cc:331] T a289a66b2dfa44dc8c8547ff83c1716e P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:30.314007 2608 raft_consensus.cc:491] T 088ef4861dc14f81ae470c3aaee5b4e1 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:30.314138 2118 raft_consensus.cc:2241] T a289a66b2dfa44dc8c8547ff83c1716e P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:30.314294 2118 raft_consensus.cc:2270] T a289a66b2dfa44dc8c8547ff83c1716e P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:30.314145 2608 raft_consensus.cc:513] T 088ef4861dc14f81ae470c3aaee5b4e1 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.314796 2118 ts_tablet_manager.cc:1905] T a289a66b2dfa44dc8c8547ff83c1716e P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:30.314842 2608 leader_election.cc:290] T 088ef4861dc14f81ae470c3aaee5b4e1 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:30.314734 1150 raft_consensus.cc:357] T 508406601c994e42a5c5b1f665e74957 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.315007 1150 raft_consensus.cc:383] T 508406601c994e42a5c5b1f665e74957 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:30.315099 1150 raft_consensus.cc:738] T 508406601c994e42a5c5b1f665e74957 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:30.315326 1150 consensus_queue.cc:260] T 508406601c994e42a5c5b1f665e74957 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.315835 1150 ts_tablet_manager.cc:1428] T 508406601c994e42a5c5b1f665e74957 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20250901 01:52:30.316038 992 leader_election.cc:336] T ad1a6be9fcc045d3bb4e71a2f6bd5ccd P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:30.316035 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5c951915ce134f26bb714fd3db4f6d72"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.316135 1150 tablet_bootstrap.cc:492] T 75a703c8e20a4a2fa0ab6ec58d41ef92 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:30.316208 992 leader_election.cc:304] T ad1a6be9fcc045d3bb4e71a2f6bd5ccd P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:30.316498 1761 raft_consensus.cc:2747] T ad1a6be9fcc045d3bb4e71a2f6bd5ccd P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:30.316829 992 leader_election.cc:336] T 48a211e174054866b030cdfb0cf947f4 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:30.316962 992 leader_election.cc:304] T 48a211e174054866b030cdfb0cf947f4 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
W20250901 01:52:30.317453 858 leader_election.cc:336] T 088ef4861dc14f81ae470c3aaee5b4e1 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:30.318182 1150 tablet_bootstrap.cc:654] T 75a703c8e20a4a2fa0ab6ec58d41ef92 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:30.318315 1761 raft_consensus.cc:2747] T 48a211e174054866b030cdfb0cf947f4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:30.318789 992 leader_election.cc:336] T 1495de70f0844f82a0eea99c059f076b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:30.319000 992 leader_election.cc:336] T c754129977904f978f87fd89d7092e36 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:30.319089 992 leader_election.cc:304] T c754129977904f978f87fd89d7092e36 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:30.322050 2113 ts_tablet_manager.cc:1918] T 1286c0ab85704837a99b6e5a62184d77 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:30.322212 2113 log.cc:1199] T 1286c0ab85704837a99b6e5a62184d77 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/1286c0ab85704837a99b6e5a62184d77
I20250901 01:52:30.322721 2113 ts_tablet_manager.cc:1939] T 1286c0ab85704837a99b6e5a62184d77 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:30.325098 1761 raft_consensus.cc:2747] T c754129977904f978f87fd89d7092e36 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:30.325695 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 122) took 5132 ms. Trace:
I20250901 01:52:30.325861 2113 rpcz_store.cc:276] 0901 01:52:25.192844 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.192934 (+ 90us) service_pool.cc:225] Handling call
0901 01:52:30.300889 (+5107955us) ts_tablet_manager.cc:1145] Deleting tablet 1286c0ab85704837a99b6e5a62184d77
0901 01:52:30.300898 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:30.310559 (+ 9661us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:30.321908 (+ 11349us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:30.325658 (+ 3750us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1628,"tablet-delete.queue_time_us":5086759}
I20250901 01:52:30.326560 2113 tablet_replica.cc:331] T 9473f0ef978c4bfaa41fb7b0729e8863 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:30.327346 1150 tablet_bootstrap.cc:492] T 75a703c8e20a4a2fa0ab6ec58d41ef92 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:30.327567 1150 ts_tablet_manager.cc:1397] T 75a703c8e20a4a2fa0ab6ec58d41ef92 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20250901 01:52:30.328337 1150 raft_consensus.cc:357] T 75a703c8e20a4a2fa0ab6ec58d41ef92 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:30.328586 1150 raft_consensus.cc:383] T 75a703c8e20a4a2fa0ab6ec58d41ef92 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:30.328678 1150 raft_consensus.cc:738] T 75a703c8e20a4a2fa0ab6ec58d41ef92 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:30.328997 1150 consensus_queue.cc:260] T 75a703c8e20a4a2fa0ab6ec58d41ef92 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
E20250901 01:52:30.329394 1696 catalog_manager.cc:6205] Error selecting replicas for tablet a6b539e160194da3a849d9e2f1ec9859, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_03fb48eaab9b42858ab7cefb722e43a8'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:30.329515 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 1286c0ab85704837a99b6e5a62184d77 (table default.Table_c9401a0bf1b84645bdfd72bc9bcbe5b6 [id=be7660807d4745d19346469b223bea60]) successfully deleted
E20250901 01:52:30.329591 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 6ce9521b925b4f97bb4a810ce77e147e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_9e04f976a93f4d30baccf90fa3622dfe'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:30.329584 1150 ts_tablet_manager.cc:1428] T 75a703c8e20a4a2fa0ab6ec58d41ef92 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250901 01:52:30.329962 1150 tablet_bootstrap.cc:492] T 50472a7aa2f04e0fbd65ac6c86c34b3e P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
E20250901 01:52:30.329986 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2f61edb50d7a42198bfd70639c7bf4de, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_a5c783cbdb694af69047b31d99ff8def'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:30.329699 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_bb745f6959134cdc8a4a6cb6b29d466a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.331756 1150 tablet_bootstrap.cc:654] T 50472a7aa2f04e0fbd65ac6c86c34b3e P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:30.331768 1485 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 7b16ca1daf394a6da65a4af8da82ec33
W20250901 01:52:30.331962 1485 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 7b16ca1daf394a6da65a4af8da82ec33: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:30.332703 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 7b16ca1daf394a6da65a4af8da82ec33 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:30 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:30.332819 1488 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 081e2630e284498aa819f85bf8326315
W20250901 01:52:30.333155 1488 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 081e2630e284498aa819f85bf8326315: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:30.333171 2113 raft_consensus.cc:2241] T 9473f0ef978c4bfaa41fb7b0729e8863 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:30.333333 2113 raft_consensus.cc:2270] T 9473f0ef978c4bfaa41fb7b0729e8863 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:30.333491 1488 catalog_manager.cc:6652] ReplaceTablet: tablet 081e2630e284498aa819f85bf8326315 deleted and replaced by tablet 66719d8e9c614657820440ebe8ce7c1e
I20250901 01:52:30.333853 2113 ts_tablet_manager.cc:1905] T 9473f0ef978c4bfaa41fb7b0729e8863 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:30.334261 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 081e2630e284498aa819f85bf8326315 with delete_type TABLET_DATA_DELETED (replaced by tablet 66719d8e9c614657820440ebe8ce7c1e) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:30.336706 1484 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: default.Table_97526c9a6c9f4acbb6dedd78475571e0 [id=305490ecfddb4d46927f7bb128c7f878] alter complete (version 1)
W20250901 01:52:30.340456 992 leader_election.cc:336] T a9b7f2e6548043a7865e25387cdd06c9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:30.340586 2113 ts_tablet_manager.cc:1918] T 9473f0ef978c4bfaa41fb7b0729e8863 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:30.340705 2113 log.cc:1199] T 9473f0ef978c4bfaa41fb7b0729e8863 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/9473f0ef978c4bfaa41fb7b0729e8863
I20250901 01:52:30.343109 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 7b16ca1daf394a6da65a4af8da82ec33 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:30 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:30.343353 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 081e2630e284498aa819f85bf8326315 with delete_type TABLET_DATA_DELETED (replaced by tablet 66719d8e9c614657820440ebe8ce7c1e) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:30.343829 2113 ts_tablet_manager.cc:1939] T 9473f0ef978c4bfaa41fb7b0729e8863 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
W20250901 01:52:30.340881 992 leader_election.cc:336] T 1a60c10e42104502ac2d3b80b8126ceb P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:30.344652 992 leader_election.cc:304] T 1a60c10e42104502ac2d3b80b8126ceb P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:30.344964 1761 raft_consensus.cc:2747] T 1a60c10e42104502ac2d3b80b8126ceb P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:30.346153 922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a9b7f2e6548043a7865e25387cdd06c9" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
W20250901 01:52:30.346282 859 leader_election.cc:343] T 435b1cca5a7d4d49959a310cb250b2a0 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:30.346330 922 raft_consensus.cc:3058] T a9b7f2e6548043a7865e25387cdd06c9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:30.346390 859 leader_election.cc:304] T 435b1cca5a7d4d49959a310cb250b2a0 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:30.346609 1151 tablet_bootstrap.cc:492] T 6c0abbfc8d904dce989ce51961e4237d P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:30.346745 1151 ts_tablet_manager.cc:1397] T 6c0abbfc8d904dce989ce51961e4237d P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.060s user 0.001s sys 0.002s
W20250901 01:52:30.346766 859 leader_election.cc:343] T f195b040a6ae4b7b83a8308ab0a12b37 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:30.346850 859 leader_election.cc:304] T f195b040a6ae4b7b83a8308ab0a12b37 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:30.347147 2323 raft_consensus.cc:2747] T 435b1cca5a7d4d49959a310cb250b2a0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:30.347275 2323 raft_consensus.cc:2747] T f195b040a6ae4b7b83a8308ab0a12b37 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:30.347514 1151 raft_consensus.cc:357] T 6c0abbfc8d904dce989ce51961e4237d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.347765 1151 raft_consensus.cc:383] T 6c0abbfc8d904dce989ce51961e4237d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:30.347913 992 leader_election.cc:304] T ed1b41340ce742a88c3e4fd85cd25ca6 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:30.348443 1761 raft_consensus.cc:2802] T ed1b41340ce742a88c3e4fd85cd25ca6 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:30.348579 992 leader_election.cc:304] T 33f637f6a8134215bb1995ec38440073 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:30.348578 1761 raft_consensus.cc:491] T ed1b41340ce742a88c3e4fd85cd25ca6 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:30.348726 1761 raft_consensus.cc:3058] T ed1b41340ce742a88c3e4fd85cd25ca6 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:30.348866 992 leader_election.cc:304] T 1495de70f0844f82a0eea99c059f076b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:30.348940 922 raft_consensus.cc:2466] T a9b7f2e6548043a7865e25387cdd06c9 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:30.349094 2613 raft_consensus.cc:2802] T 33f637f6a8134215bb1995ec38440073 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:30.349227 2613 raft_consensus.cc:695] T 33f637f6a8134215bb1995ec38440073 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:30.349516 2613 consensus_queue.cc:237] T 33f637f6a8134215bb1995ec38440073 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.350068 2518 raft_consensus.cc:2802] T 1495de70f0844f82a0eea99c059f076b P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:30.350196 2518 raft_consensus.cc:695] T 1495de70f0844f82a0eea99c059f076b P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:30.350422 2518 consensus_queue.cc:237] T 1495de70f0844f82a0eea99c059f076b P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.352907 2608 raft_consensus.cc:491] T 6c0abbfc8d904dce989ce51961e4237d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:30.353029 2608 raft_consensus.cc:513] T 6c0abbfc8d904dce989ce51961e4237d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.353565 2608 leader_election.cc:290] T 6c0abbfc8d904dce989ce51961e4237d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:30.355336 858 leader_election.cc:336] T 6c0abbfc8d904dce989ce51961e4237d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:30.355798 1761 raft_consensus.cc:513] T ed1b41340ce742a88c3e4fd85cd25ca6 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.357354 1761 leader_election.cc:290] T ed1b41340ce742a88c3e4fd85cd25ca6 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:30.358353 2608 raft_consensus.cc:491] T b2b172757e50404aae78e818921a7d9d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:30.358515 2608 raft_consensus.cc:513] T b2b172757e50404aae78e818921a7d9d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.359143 2608 leader_election.cc:290] T b2b172757e50404aae78e818921a7d9d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:30.359948 2574 tablet.cc:1722] T ad45c90ba0da43c9860745e534664a98 P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20250901 01:52:30.361099 858 leader_election.cc:336] T b2b172757e50404aae78e818921a7d9d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:30.361937 1503 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e3add7edb97a40ef9f7c6bc011d13227" candidate_term: 5 candidate_status { last_received { term: 4 index: 671 } } ignore_live_leader: false dest_uuid: "7ce2288f784044eea010664e3def133f" is_pre_election: true
I20250901 01:52:30.348311 1151 raft_consensus.cc:738] T 6c0abbfc8d904dce989ce51961e4237d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:30.362746 2061 leader_election.cc:304] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3add7edb97a40ef9f7c6bc011d13227; no voters: 7ce2288f784044eea010664e3def133f, d2ae2cb6af0f4b3a8f47ab1cdd6567d3
I20250901 01:52:30.362900 1151 consensus_queue.cc:260] T 6c0abbfc8d904dce989ce51961e4237d P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.363420 1151 ts_tablet_manager.cc:1428] T 6c0abbfc8d904dce989ce51961e4237d P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.017s user 0.000s sys 0.002s
I20250901 01:52:30.363703 1151 tablet_bootstrap.cc:492] T 3239e58531d445fa80165cf678bc277c P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
W20250901 01:52:30.365466 992 consensus_peers.cc:489] T ddbc15685fca4c708329420109ad0c57 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Couldn't send request to peer 00c282f779c1477d920f24dc0dc16628. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ddbc15685fca4c708329420109ad0c57. This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:30.366250 2574 tablet.cc:1722] T f13b972e25214f168421439fe3943e8a P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:30.369694 1151 tablet_bootstrap.cc:654] T 3239e58531d445fa80165cf678bc277c P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:30.372601 2118 ts_tablet_manager.cc:1918] T a289a66b2dfa44dc8c8547ff83c1716e P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:30.376631 1486 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: default.Table_99f0fb91a6994b2abf5af0c8f701159b [id=5d14706774d345c48ca3575feabb17b0] alter complete (version 3)
I20250901 01:52:30.373378 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 123) took 5164 ms. Trace:
I20250901 01:52:30.379506 2113 rpcz_store.cc:276] 0901 01:52:25.209240 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.209300 (+ 60us) service_pool.cc:225] Handling call
0901 01:52:30.326478 (+5117178us) ts_tablet_manager.cc:1145] Deleting tablet 9473f0ef978c4bfaa41fb7b0729e8863
0901 01:52:30.326489 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:30.337566 (+ 11077us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:30.340464 (+ 2898us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:30.373326 (+ 32862us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5112060}
I20250901 01:52:30.379928 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 9473f0ef978c4bfaa41fb7b0729e8863 (table default.Table_ae4267b6728a4e60a333464621e3dc8f [id=a0fdcd4db982472c9e18aaa4d9fe50c5]) successfully deleted
I20250901 01:52:30.380012 2113 tablet_replica.cc:331] T 154574e17f214cc0ad56e6e6be3af899 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:30.380227 2113 raft_consensus.cc:2241] T 154574e17f214cc0ad56e6e6be3af899 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250901 01:52:30.365739 992 leader_election.cc:343] T 56475ba491a24301a534304811f30140 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 56475ba491a24301a534304811f30140
I20250901 01:52:30.380611 2113 raft_consensus.cc:2270] T 154574e17f214cc0ad56e6e6be3af899 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:30.381151 992 leader_election.cc:304] T a9b7f2e6548043a7865e25387cdd06c9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:30.381493 2613 raft_consensus.cc:2802] T a9b7f2e6548043a7865e25387cdd06c9 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:30.381727 2613 raft_consensus.cc:695] T a9b7f2e6548043a7865e25387cdd06c9 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:30.382007 2613 consensus_queue.cc:237] T a9b7f2e6548043a7865e25387cdd06c9 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.383108 1483 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet f4fd117c395e41c2ab679c684908c5b7
W20250901 01:52:30.383324 1483 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet f4fd117c395e41c2ab679c684908c5b7: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:30.383697 1483 catalog_manager.cc:6652] ReplaceTablet: tablet f4fd117c395e41c2ab679c684908c5b7 deleted and replaced by tablet ff2bd017417c41b5b8225b8f00283e63
I20250901 01:52:30.384980 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet f4fd117c395e41c2ab679c684908c5b7 with delete_type TABLET_DATA_DELETED (replaced by tablet ff2bd017417c41b5b8225b8f00283e63) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:30.390304 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet c763d252598745a088344aa0eb5fa3d3 with delete_type TABLET_DATA_DELETED (replaced by tablet a8483e7212d94daba7e5401256c33aef) from {username='slave'} at 127.0.0.1:35946
W20250901 01:52:30.390089 992 leader_election.cc:336] T ed1b41340ce742a88c3e4fd85cd25ca6 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:30.390868 992 leader_election.cc:336] T 33f637f6a8134215bb1995ec38440073 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:30.391114 992 leader_election.cc:336] T 1495de70f0844f82a0eea99c059f076b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:30.391279 1489 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet f80f13db501c42c7950499df8fcb6fc8
W20250901 01:52:30.391435 1489 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet f80f13db501c42c7950499df8fcb6fc8: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:30.391723 1489 catalog_manager.cc:6652] ReplaceTablet: tablet f80f13db501c42c7950499df8fcb6fc8 deleted and replaced by tablet 0c88c339e59a4c1da21756cba0645bdd
I20250901 01:52:30.392908 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet f80f13db501c42c7950499df8fcb6fc8 with delete_type TABLET_DATA_DELETED (replaced by tablet 0c88c339e59a4c1da21756cba0645bdd) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:30.393540 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 579b9cdf819f4609a532e96bec093fd5 with delete_type TABLET_DATA_DELETED (replaced by tablet 56475ba491a24301a534304811f30140) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:30.395524 2113 ts_tablet_manager.cc:1905] T 154574e17f214cc0ad56e6e6be3af899 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:30.396706 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ab5bd159998b4a1d817df997f78aca82"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.397648 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_bfddd2c6609e45fbaeb4e3d56eb9741b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.398365 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9153e222ae8a4064ba7a34277170d6a2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.399001 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ca4899609719461c956165b41c16706e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.399659 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_445501cbfecd497db0d071d401805238"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.406100 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f5c829272e564d3a90f21d38ea32a659"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.406920 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2ec3aa4dab2d4ab498682b31597c21ae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.407588 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a54e1b563b7a4292a94be582f3738eb4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.408205 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8a5d549fc2b24ba4bfa4721ae1cf7660"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.410101 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8b2eb60c8db3475184382b592e178324"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.410751 1054 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "088ef4861dc14f81ae470c3aaee5b4e1" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:30.410794 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5b7faa7753134cfdbb2a76564d06d15a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.410908 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2b172757e50404aae78e818921a7d9d" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:30.411233 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4f9bfade22d246438912a9a77a29760f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:30.411821 992 leader_election.cc:336] T 56475ba491a24301a534304811f30140 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:30.411842 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ec1c1d5cc6144702bafbca78c85dd6ba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.411942 992 leader_election.cc:304] T 56475ba491a24301a534304811f30140 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
W20250901 01:52:30.412249 992 leader_election.cc:336] T a9b7f2e6548043a7865e25387cdd06c9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:30.412299 1761 raft_consensus.cc:2747] T 56475ba491a24301a534304811f30140 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:30.412921 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_0f41c7865f5a4397b1232b785c5f50f2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.413429 1055 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c0abbfc8d904dce989ce51961e4237d" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:30.413616 1055 raft_consensus.cc:2466] T 6c0abbfc8d904dce989ce51961e4237d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00c282f779c1477d920f24dc0dc16628 in term 0.
I20250901 01:52:30.373027 1150 tablet_bootstrap.cc:492] T 50472a7aa2f04e0fbd65ac6c86c34b3e P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:30.414674 1150 ts_tablet_manager.cc:1397] T 50472a7aa2f04e0fbd65ac6c86c34b3e P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.085s user 0.005s sys 0.001s
I20250901 01:52:30.384227 887 tablet_service.cc:1515] Processing DeleteTablet for tablet f4fd117c395e41c2ab679c684908c5b7 with delete_type TABLET_DATA_DELETED (replaced by tablet ff2bd017417c41b5b8225b8f00283e63) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:30.392462 885 tablet_service.cc:1515] Processing DeleteTablet for tablet f80f13db501c42c7950499df8fcb6fc8 with delete_type TABLET_DATA_DELETED (replaced by tablet 0c88c339e59a4c1da21756cba0645bdd) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:30.392743 922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ed1b41340ce742a88c3e4fd85cd25ca6" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:30.415571 1150 raft_consensus.cc:357] T 50472a7aa2f04e0fbd65ac6c86c34b3e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.415709 922 raft_consensus.cc:3058] T ed1b41340ce742a88c3e4fd85cd25ca6 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:30.415735 1150 raft_consensus.cc:383] T 50472a7aa2f04e0fbd65ac6c86c34b3e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:30.415795 1150 raft_consensus.cc:738] T 50472a7aa2f04e0fbd65ac6c86c34b3e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:30.415987 1150 consensus_queue.cc:260] T 50472a7aa2f04e0fbd65ac6c86c34b3e P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.416509 1150 ts_tablet_manager.cc:1428] T 50472a7aa2f04e0fbd65ac6c86c34b3e P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250901 01:52:30.416795 1150 tablet_bootstrap.cc:492] T f8d9bbf0c5be462a8ddc539238cbdf97 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:30.408927 2608 raft_consensus.cc:491] T 44af4764d4f64e44aec84ccb84be6f5f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:30.418439 2608 raft_consensus.cc:513] T 44af4764d4f64e44aec84ccb84be6f5f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.418555 922 raft_consensus.cc:2466] T ed1b41340ce742a88c3e4fd85cd25ca6 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:30.418756 1150 tablet_bootstrap.cc:654] T f8d9bbf0c5be462a8ddc539238cbdf97 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:30.419142 2608 leader_election.cc:290] T 44af4764d4f64e44aec84ccb84be6f5f P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:30.420113 2118 log.cc:1199] T a289a66b2dfa44dc8c8547ff83c1716e P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/a289a66b2dfa44dc8c8547ff83c1716e
I20250901 01:52:30.413515 2590 tablet.cc:1722] T ad45c90ba0da43c9860745e534664a98 P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250901 01:52:30.414036 2582 tablet.cc:1722] T f13b972e25214f168421439fe3943e8a P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:30.420545 2118 ts_tablet_manager.cc:1939] T a289a66b2dfa44dc8c8547ff83c1716e P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:30.421252 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 134) took 3774 ms. Trace:
I20250901 01:52:30.421324 2118 rpcz_store.cc:276] 0901 01:52:26.646715 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.646763 (+ 48us) service_pool.cc:225] Handling call
0901 01:52:30.313893 (+3667130us) ts_tablet_manager.cc:1145] Deleting tablet a289a66b2dfa44dc8c8547ff83c1716e
0901 01:52:30.313901 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:30.343576 (+ 29675us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:30.372505 (+ 28929us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:30.421234 (+ 48729us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3666948}
I20250901 01:52:30.421643 2118 tablet_replica.cc:331] T 36e10328c34242ebb4dbd8f294162696 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:30.421824 2118 raft_consensus.cc:2241] T 36e10328c34242ebb4dbd8f294162696 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:30.421955 2118 raft_consensus.cc:2270] T 36e10328c34242ebb4dbd8f294162696 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:30.423230 2118 ts_tablet_manager.cc:1905] T 36e10328c34242ebb4dbd8f294162696 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:30.425103 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet a289a66b2dfa44dc8c8547ff83c1716e (table default.Table_4f2be95a1b824f06ba16e56abcdebc8a [id=434cfb105b894499b15bf360a8cffda4]) successfully deleted
I20250901 01:52:30.428292 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_fac30139413743708a4684a761b01f8e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:30.429104 858 leader_election.cc:336] T 44af4764d4f64e44aec84ccb84be6f5f P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:30.429167 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1381482a219144e0b9d032daa0825ee3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.429297 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_982218fe38bb4be7a9c7a10f3e9d8ab4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.429982 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_68c037f3d994423fa0144625e5a82835"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.430711 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8e24046438304442aa5e741cd227917f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.431047 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9bb9ba2be2d342cbad1d874ec3664fef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.430752 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8d81e7985dd34ca89fc8315dce83a111"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.429982 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1f01a6da0dea4585a5415f7ca27878cc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.434975 1489 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_37e451883f5642f094a03b5ca48a0b25" } new_table_name: "default.Table_7aa28ad9ee8c49859af40f9ccb61af33" modify_external_catalogs: true
I20250901 01:52:30.435439 1482 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_ca5bada2f51949d790e9a38e25fa1bac" } new_table_name: "default.Table_7d149b0571834312849553b6e8c8f2ea" modify_external_catalogs: true
W20250901 01:52:30.440935 859 leader_election.cc:343] T 088ef4861dc14f81ae470c3aaee5b4e1 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:30.441066 859 leader_election.cc:304] T 088ef4861dc14f81ae470c3aaee5b4e1 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:30.441463 859 leader_election.cc:343] T b2b172757e50404aae78e818921a7d9d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:30.441571 859 leader_election.cc:304] T b2b172757e50404aae78e818921a7d9d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:30.441656 992 consensus_peers.cc:489] T f62d96c37cca4143a4b098b7c69979e6 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Couldn't send request to peer 00c282f779c1477d920f24dc0dc16628. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f62d96c37cca4143a4b098b7c69979e6. This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:30.442003 2408 raft_consensus.cc:2747] T 088ef4861dc14f81ae470c3aaee5b4e1 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:30.442163 2408 raft_consensus.cc:2747] T b2b172757e50404aae78e818921a7d9d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:30.442691 992 leader_election.cc:304] T ed1b41340ce742a88c3e4fd85cd25ca6 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:30.442924 1761 raft_consensus.cc:2802] T ed1b41340ce742a88c3e4fd85cd25ca6 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:30.443049 1761 raft_consensus.cc:695] T ed1b41340ce742a88c3e4fd85cd25ca6 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:30.443286 1761 consensus_queue.cc:237] T ed1b41340ce742a88c3e4fd85cd25ca6 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.443640 1490 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_8b1086c9a4c44239851c586ea7267e3c" } modify_external_catalogs: true
I20250901 01:52:30.443851 1490 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_8b1086c9a4c44239851c586ea7267e3c" } modify_external_catalogs: true
I20250901 01:52:30.444446 1483 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_c714bf77d32b42aeb225bbc5b9a6f43b" } modify_external_catalogs: true
I20250901 01:52:30.444658 1483 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_c714bf77d32b42aeb225bbc5b9a6f43b" } modify_external_catalogs: true
I20250901 01:52:30.445161 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4b5efe489a184b49b685220cb0ab9572"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.446010 1488 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_f840789a58984799bd8057128f6ea439" } modify_external_catalogs: true
I20250901 01:52:30.446174 1488 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_f840789a58984799bd8057128f6ea439" } modify_external_catalogs: true
I20250901 01:52:30.447151 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_88efeb06b78b4b688cc9726df3a2a961"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.447922 1485 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_7c12d66048a6470e8c227dc1d9f45fa4" } new_table_name: "default.Table_3349df70e80e4caabfd33a2f303cf723" modify_external_catalogs: true
W20250901 01:52:30.467633 1486 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.3:51295 (request call id 183) took 2132 ms (client timeout 1000 ms). Trace:
W20250901 01:52:30.473891 1482 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5ddbc7e6820247a987f4d54757d1191d consensus state has no leader
I20250901 01:52:30.477638 1482 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_83f2c0acedbc4c6982caf5313646e90b" } new_table_name: "default.Table_f9e2a9182f6340bfa551fa2fc2eb2429" modify_external_catalogs: true
I20250901 01:52:30.489923 2552 raft_consensus.cc:2747] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250901 01:52:30.494163 2105 ts_manager.cc:194] Registered new tserver with Master: fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:30.498896 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet a59ac57aebe24860aa02f53ce0485910 with delete_type TABLET_DATA_DELETED (replaced by tablet 93bcac1fd95d44dc927abfca81692921) from {username='slave'} at 127.0.0.1:35946
W20250901 01:52:30.498996 1485 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 787be682c0e44248a80d7037fb61dc6c has no consensus state
I20250901 01:52:30.499181 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 824229183e024feaa975c7f0387c9c1a with delete_type TABLET_DATA_DELETED (replaced by tablet d4c9aa723aee4dc4b6237cb5c3722b52) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:30.499341 1034 tablet_service.cc:1515] Processing DeleteTablet for tablet c9a4c80bec1d41458974a0c449aecaf7 with delete_type TABLET_DATA_DELETED (replaced by tablet 121803e79ab64ec0b948b448646b9d0a) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:30.499680 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 40d74d7b5a3b4d1dae09eebd9e07a0c4 with delete_type TABLET_DATA_DELETED (replaced by tablet 6cf09e8f299548dba2cce16d7b565bad) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:30.499728 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_76ef7617ef6744408032797b27bd8e48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.500000 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet 12c04e0c1f804179ae23509cdb24d395 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:13 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:30.500166 2608 raft_consensus.cc:491] T b13afacd37474781877b9f2cd20b313a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:30.500208 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 50e3da53433f4c14b57db8a733fa53bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:18 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:30.500298 2608 raft_consensus.cc:513] T b13afacd37474781877b9f2cd20b313a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:30.500419 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 1286c0ab85704837a99b6e5a62184d77 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:16 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:30.500607 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c6c24820956d48ea8b9fd1528945e5f4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.500960 2608 leader_election.cc:290] T b13afacd37474781877b9f2cd20b313a P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:30.501967 992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:40561, user_credentials={real_user=slave}} blocked reactor thread for 49931.8us
I20250901 01:52:30.502460 859 leader_election.cc:304] T 6c0abbfc8d904dce989ce51961e4237d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:30.502815 1483 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 55bd2daa07294358999956cb553d973f
I20250901 01:52:30.502923 2608 raft_consensus.cc:2802] T 6c0abbfc8d904dce989ce51961e4237d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:30.502995 2608 raft_consensus.cc:491] T 6c0abbfc8d904dce989ce51961e4237d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250901 01:52:30.502996 1483 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 55bd2daa07294358999956cb553d973f: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:30.503057 2608 raft_consensus.cc:3058] T 6c0abbfc8d904dce989ce51961e4237d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:30.503777 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4eff20325d0045e18b1f9e1fcbe9c499"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.505421 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ecbc739e9dac4187a7c13a70b4dc7859"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.505851 2608 raft_consensus.cc:513] T 6c0abbfc8d904dce989ce51961e4237d P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.506254 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_19f1a3f39b294bb69123f3ca52539cf3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.507035 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_cea337b64dfe46b0b5646383840e5e99"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:30.506721 858 leader_election.cc:336] T b13afacd37474781877b9f2cd20b313a P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:30.507607 2118 ts_tablet_manager.cc:1918] T 36e10328c34242ebb4dbd8f294162696 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:30.507711 2118 log.cc:1199] T 36e10328c34242ebb4dbd8f294162696 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/36e10328c34242ebb4dbd8f294162696
I20250901 01:52:30.507932 2608 leader_election.cc:290] T 6c0abbfc8d904dce989ce51961e4237d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:30.508064 1483 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_2d7b072b55354009b9cce1aef9408e35" } modify_external_catalogs: true
I20250901 01:52:30.508297 1483 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_2d7b072b55354009b9cce1aef9408e35" } modify_external_catalogs: true
W20250901 01:52:30.509693 858 leader_election.cc:336] T 6c0abbfc8d904dce989ce51961e4237d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:30.501351 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e35720c43efc437abc691adaf20f0ef2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.511260 1151 tablet_bootstrap.cc:492] T 3239e58531d445fa80165cf678bc277c P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:30.508024 2118 ts_tablet_manager.cc:1939] T 36e10328c34242ebb4dbd8f294162696 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:30.508503 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 55bd2daa07294358999956cb553d973f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:30 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:30.515033 885 tablet_service.cc:1515] Processing DeleteTablet for tablet e8e7d2a72f704f52a8f8d46911a8a9e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:30.512210 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 822382a1781b4304bce29355ce83730e with delete_type TABLET_DATA_DELETED (replaced by tablet 0958b0a4068344abb0f9db601d7dd2ad) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:30.512342 1035 tablet_service.cc:1515] Processing DeleteTablet for tablet abe27e749fa0485195e0064a49d5431e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:15 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:30.512356 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 121803e79ab64ec0b948b448646b9d0a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:14 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:30.512487 1036 tablet_service.cc:1515] Processing DeleteTablet for tablet 82d163681fda47f38d79863bd28f576d with delete_type TABLET_DATA_DELETED (replaced by tablet bf8f0237233a49219cd1c41cf808bdba) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:30.512573 1033 tablet_service.cc:1515] Processing DeleteTablet for tablet 4a87e68850c4459eb2c8eafac89c6aa9 with delete_type TABLET_DATA_DELETED (replaced by tablet 615b2306fca84003b264da9d5f95ddab) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:30.512650 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 670d90a37b934579b353bb4e4820127b with delete_type TABLET_DATA_DELETED (replaced by tablet 75a703c8e20a4a2fa0ab6ec58d41ef92) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:30.512722 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet f8a5886ddb644a86b76df44aab309057 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:14 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:30.512811 1038 tablet_service.cc:1515] Processing DeleteTablet for tablet 9473f0ef978c4bfaa41fb7b0729e8863 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:17 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:30.512889 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet 55bd2daa07294358999956cb553d973f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:30 UTC) from {username='slave'} at 127.0.0.1:35946
W20250901 01:52:30.512964 992 consensus_peers.cc:489] T fcb7b5b7d63d4f0b906fea97cb548de2 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250901 01:52:30.515933 1151 ts_tablet_manager.cc:1397] T 3239e58531d445fa80165cf678bc277c P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.152s user 0.002s sys 0.006s
W20250901 01:52:30.516211 992 leader_election.cc:336] T ed1b41340ce742a88c3e4fd85cd25ca6 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:30.513042 2113 ts_tablet_manager.cc:1918] T 154574e17f214cc0ad56e6e6be3af899 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:30.516585 2113 log.cc:1199] T 154574e17f214cc0ad56e6e6be3af899 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/154574e17f214cc0ad56e6e6be3af899
I20250901 01:52:30.516963 2113 ts_tablet_manager.cc:1939] T 154574e17f214cc0ad56e6e6be3af899 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:30.518126 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 125) took 5270 ms. Trace:
I20250901 01:52:30.518402 1151 raft_consensus.cc:357] T 3239e58531d445fa80165cf678bc277c P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.520278 1055 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "44af4764d4f64e44aec84ccb84be6f5f" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:30.520476 1151 raft_consensus.cc:383] T 3239e58531d445fa80165cf678bc277c P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:30.520614 1151 raft_consensus.cc:738] T 3239e58531d445fa80165cf678bc277c P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:30.520639 885 tablet_service.cc:1515] Processing DeleteTablet for tablet becbaae4fd7e44faa679e9479465a7c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:23 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:30.520882 1151 consensus_queue.cc:260] T 3239e58531d445fa80165cf678bc277c P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.521416 1151 ts_tablet_manager.cc:1428] T 3239e58531d445fa80165cf678bc277c P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250901 01:52:30.519876 2613 raft_consensus.cc:491] T 31614db4c53a4f3c96fd190d326ce601 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:30.521632 2613 raft_consensus.cc:513] T 31614db4c53a4f3c96fd190d326ce601 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.522208 2613 leader_election.cc:290] T 31614db4c53a4f3c96fd190d326ce601 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:30.520287 2113 rpcz_store.cc:276] 0901 01:52:25.247312 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.247428 (+ 116us) service_pool.cc:225] Handling call
0901 01:52:30.379928 (+5132500us) ts_tablet_manager.cc:1145] Deleting tablet 154574e17f214cc0ad56e6e6be3af899
0901 01:52:30.379938 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:30.398666 (+ 18728us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:30.504607 (+105941us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:30.518107 (+ 13500us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5119665}
I20250901 01:52:30.523030 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b13afacd37474781877b9f2cd20b313a" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:30.523062 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 154574e17f214cc0ad56e6e6be3af899 (table default.Table_cb6a9252a90d4184b917db4c7edc06b1 [id=b550b0e6553e41a2a20ea3bf23e1c7a0]) successfully deleted
I20250901 01:52:30.523347 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 879fa66e55544118aafe504587393c66 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:23 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:30.523700 1151 tablet_bootstrap.cc:492] T 879fa66e55544118aafe504587393c66 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:30.523813 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 410d44e5f1974ed69e5dee65e98856a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:25 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:30.523993 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 31614db4c53a4f3c96fd190d326ce601 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:30.524245 1150 tablet_bootstrap.cc:492] T f8d9bbf0c5be462a8ddc539238cbdf97 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:30.524010 2113 tablet_replica.cc:331] T 9d886fa63f8f4996b087997b542e8bb1 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:30.524446 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c0abbfc8d904dce989ce51961e4237d" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58"
I20250901 01:52:30.525386 1052 raft_consensus.cc:3058] T 6c0abbfc8d904dce989ce51961e4237d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
W20250901 01:52:30.467809 1486 rpcz_store.cc:269] 0901 01:52:28.334883 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.438780 (+103897us) service_pool.cc:225] Handling call
0901 01:52:30.467588 (+2028808us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0901 01:52:29.032755 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:29.032845 (+ 90us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:29.032854 (+ 9us) write_op.cc:435] Acquired schema lock
0901 01:52:29.032857 (+ 3us) tablet.cc:660] Decoding operations
0901 01:52:29.033482 (+ 625us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:29.033497 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:29.033504 (+ 7us) tablet.cc:683] Acquiring locks for 3 operations
0901 01:52:29.033643 (+ 139us) tablet.cc:699] Row locks acquired
0901 01:52:29.033647 (+ 4us) write_op.cc:260] PREPARE: finished
0901 01:52:29.033687 (+ 40us) write_op.cc:270] Start()
0901 01:52:29.033730 (+ 43us) write_op.cc:276] Timestamp: P: 1756691549033621 usec, L: 0
0901 01:52:29.033735 (+ 5us) op_driver.cc:348] REPLICATION: starting
0901 01:52:29.033970 (+ 235us) log.cc:844] Serialized 1275 byte log entry
0901 01:52:29.041503 (+ 7533us) op_driver.cc:464] REPLICATION: finished
0901 01:52:29.044587 (+ 3084us) write_op.cc:301] APPLY: starting
0901 01:52:29.044607 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:29.048557 (+ 3950us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:29.048590 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:29.048668 (+ 78us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:29.048710 (+ 42us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0901 01:52:29.048718 (+ 8us) write_op.cc:312] APPLY: finished
0901 01:52:29.048829 (+ 111us) log.cc:844] Serialized 35 byte log entry
0901 01:52:29.048907 (+ 78us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:29.048935 (+ 28us) write_op.cc:454] Released schema lock
0901 01:52:29.048945 (+ 10us) write_op.cc:341] FINISH: Updating metrics
0901 01:52:29.049032 (+ 87us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:29.049038 (+ 6us) write_op.cc:454] Released schema lock
Related trace 'op':
0901 01:52:30.274441 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:30.274556 (+ 115us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:30.274564 (+ 8us) write_op.cc:435] Acquired schema lock
0901 01:52:30.274566 (+ 2us) tablet.cc:660] Decoding operations
0901 01:52:30.274642 (+ 76us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:30.274657 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:30.274661 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0901 01:52:30.274704 (+ 43us) tablet.cc:699] Row locks acquired
0901 01:52:30.274706 (+ 2us) write_op.cc:260] PREPARE: finished
0901 01:52:30.274748 (+ 42us) write_op.cc:270] Start()
0901 01:52:30.274771 (+ 23us) write_op.cc:276] Timestamp: P: 1756691550274702 usec, L: 0
0901 01:52:30.274774 (+ 3us) op_driver.cc:348] REPLICATION: starting
0901 01:52:30.274915 (+ 141us) log.cc:844] Serialized 356 byte log entry
0901 01:52:30.375254 (+100339us) op_driver.cc:464] REPLICATION: finished
0901 01:52:30.375395 (+ 141us) write_op.cc:301] APPLY: starting
0901 01:52:30.375410 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:30.375451 (+ 41us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:30.375482 (+ 31us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:30.375538 (+ 56us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:30.375571 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0901 01:52:30.375578 (+ 7us) write_op.cc:312] APPLY: finished
0901 01:52:30.375657 (+ 79us) log.cc:844] Serialized 23 byte log entry
0901 01:52:30.375717 (+ 60us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:30.375735 (+ 18us) write_op.cc:454] Released schema lock
0901 01:52:30.375741 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0901 01:52:30.375820 (+ 79us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:30.375824 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":59522,"reported_tablets":208,"child_traces":[["op",{"apply.queue_time_us":3003,"apply.run_cpu_time_us":598,"apply.run_wall_time_us":4441,"num_ops":3,"prepare.queue_time_us":1725,"prepare.run_cpu_time_us":883,"prepare.run_wall_time_us":1327,"raft.queue_time_us":73,"raft.run_cpu_time_us":236,"raft.run_wall_time_us":235,"replication_time_us":7736}],["op",{"apply.queue_time_us":91,"apply.run_cpu_time_us":416,"apply.run_wall_time_us":416,"num_ops":1,"prepare.queue_time_us":15916,"prepare.run_cpu_time_us":518,"prepare.run_wall_time_us":517,"replication_time_us":100451}]]}
I20250901 01:52:30.527989 1052 raft_consensus.cc:2466] T 6c0abbfc8d904dce989ce51961e4237d P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c282f779c1477d920f24dc0dc16628 in term 1.
I20250901 01:52:30.528149 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_cc30d785753e47d49ef84c052241796f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.529188 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b4adf6ef2bd94e8782d516bab2489e08"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.530243 2113 raft_consensus.cc:2241] T 9d886fa63f8f4996b087997b542e8bb1 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:30.530663 1150 ts_tablet_manager.cc:1397] T f8d9bbf0c5be462a8ddc539238cbdf97 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.114s user 0.007s sys 0.004s
I20250901 01:52:30.530871 2113 raft_consensus.cc:2270] T 9d886fa63f8f4996b087997b542e8bb1 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:30.530936 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1e7602ff20114e79bbeab6a93fdba7c9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.531798 1485 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_af5338b09bbc4c879c7f4f573c32fd8d" } new_table_name: "default.Table_7570c3012f8a4ae7bcfc7346eadac92a" modify_external_catalogs: true
I20250901 01:52:30.533032 2113 ts_tablet_manager.cc:1905] T 9d886fa63f8f4996b087997b542e8bb1 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:30.534150 1150 raft_consensus.cc:357] T f8d9bbf0c5be462a8ddc539238cbdf97 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:30.530481 992 consensus_peers.cc:489] T 186ec810d02e4748aad15228b5866ac1 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250901 01:52:30.530368 1103 heartbeater.cc:499] Master 127.0.104.60:44231 was elected leader, sending a full tablet report...
I20250901 01:52:30.543203 1491 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 81ce448b148447ddbc41875d863e0518 (table default.Table_6e461d9cb0ae4dd095975bbc21533794 [id=49673fa73f8c436e9b8026efc4f3cbbc]). Expected version 1 got 0
I20250901 01:52:30.545569 1483 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet cf806854a8074b0798b3f9f3df587c63
I20250901 01:52:30.545799 2613 raft_consensus.cc:491] T 4cde5f4f02044b7bbd7d43b49aac4ae8 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:30.545938 2613 raft_consensus.cc:513] T 4cde5f4f02044b7bbd7d43b49aac4ae8 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:30.546335 1483 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet cf806854a8074b0798b3f9f3df587c63: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:30.546679 2613 leader_election.cc:290] T 4cde5f4f02044b7bbd7d43b49aac4ae8 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:30.547137 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet cf806854a8074b0798b3f9f3df587c63 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:30 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:30.547700 1483 master_service.cc:948] ReplaceTablet: received request to replace tablet 15e4d972e8254cf899c3e0b71b6e2bd9 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:30.548202 1491 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet d1f2d98712ff4ea6aec665512de068bd (table default.Table_82d91a1ba2eb4265896824028dec8d3a [id=3f7624d2901445e39155c12fa68e34e9]). Expected version 1 got 0
I20250901 01:52:30.550180 1491 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet ed1b41340ce742a88c3e4fd85cd25ca6 (table default.Table_ce2f3b515c8f4c1bb557c5ed84b48dba [id=0baa4993b72e4a8d94ee3059c9d2db6c]). Expected version 2 got 0
I20250901 01:52:30.550428 1491 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet c83ce95fd6fa41478de0e2168be201b1 (table default.Table_a525265208584713b1b3465d63130cd9 [id=bfd87b9b708449d3b16e3e0639561a49]). Expected version 1 got 0
I20250901 01:52:30.550843 1491 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet f13b972e25214f168421439fe3943e8a (table default.Table_63ef1dcd92db46d298446117a62cfba2 [id=c1483e9d0db64d468b12344500f9cac8]). Expected version 1 got 0
I20250901 01:52:30.551470 1491 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet c6f7c525f59b49938b697de98d4b5835 (table default.Table_db61e055dbd7450898e82c4c0c4763be [id=aabca67f6f754820977ac9c4c20f3e4f]). Expected version 1 got 0
I20250901 01:52:30.536055 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 8ed6616ce54548ba83ba299c4ecf3c38 with delete_type TABLET_DATA_DELETED (replaced by tablet 8a13346dcb3d4b6ab35c261829d50ae5) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:30.552481 1150 raft_consensus.cc:383] T f8d9bbf0c5be462a8ddc539238cbdf97 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250901 01:52:30.545533 859 leader_election.cc:343] T 44af4764d4f64e44aec84ccb84be6f5f P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:30.552570 1150 raft_consensus.cc:738] T f8d9bbf0c5be462a8ddc539238cbdf97 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:30.552631 859 leader_election.cc:304] T 44af4764d4f64e44aec84ccb84be6f5f P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:30.552809 1150 consensus_queue.cc:260] T f8d9bbf0c5be462a8ddc539238cbdf97 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.547555 887 tablet_service.cc:1515] Processing DeleteTablet for tablet cf806854a8074b0798b3f9f3df587c63 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:30 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:30.553064 2323 raft_consensus.cc:2747] T 44af4764d4f64e44aec84ccb84be6f5f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:30.553644 992 consensus_peers.cc:489] T a289a66b2dfa44dc8c8547ff83c1716e P fd27ab6665304aaea53adbd102dd4f58 -> Peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Couldn't send request to peer 00c282f779c1477d920f24dc0dc16628. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a289a66b2dfa44dc8c8547ff83c1716e. This is attempt 1: this message will repeat every 5th retry.
W20250901 01:52:30.554500 992 leader_election.cc:343] T 31614db4c53a4f3c96fd190d326ce601 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 31614db4c53a4f3c96fd190d326ce601
I20250901 01:52:30.533829 1151 tablet_bootstrap.cc:654] T 879fa66e55544118aafe504587393c66 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:30.557615 1150 ts_tablet_manager.cc:1428] T f8d9bbf0c5be462a8ddc539238cbdf97 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.026s user 0.001s sys 0.001s
I20250901 01:52:30.557884 1150 tablet_bootstrap.cc:492] T d4c9aa723aee4dc4b6237cb5c3722b52 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:30.558708 2323 raft_consensus.cc:491] T e1c8fe96a6ba4edba4f09c00f7add75e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:30.558832 2323 raft_consensus.cc:513] T e1c8fe96a6ba4edba4f09c00f7add75e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.559360 2323 leader_election.cc:290] T e1c8fe96a6ba4edba4f09c00f7add75e P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:30.560388 1150 tablet_bootstrap.cc:654] T d4c9aa723aee4dc4b6237cb5c3722b52 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:30.560904 858 leader_election.cc:336] T e1c8fe96a6ba4edba4f09c00f7add75e P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:30.562781 901 tablet_service.cc:1515] Processing DeleteTablet for tablet d58f69fad01f403eab74a47d3695ed83 with delete_type TABLET_DATA_DELETED (replaced by tablet 74b709cb5ec64006b2f12bead8217d80) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:30.563455 901 tablet_service.cc:1515] Processing DeleteTablet for tablet a289a66b2dfa44dc8c8547ff83c1716e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:26 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:30.564019 901 tablet_service.cc:1515] Processing DeleteTablet for tablet e2aa32450a584395ae07b49f76a7b26d with delete_type TABLET_DATA_DELETED (replaced by tablet 7304937048ca48a7bd153f3cdcfaeff0) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:30.564513 898 tablet_service.cc:1515] Processing DeleteTablet for tablet 7cc1efcb6bff4aba8015d2d3b7f4c923 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:25 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:30.564594 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 135) took 3893 ms. Trace:
I20250901 01:52:30.564679 2118 rpcz_store.cc:276] 0901 01:52:26.670663 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.670710 (+ 47us) service_pool.cc:225] Handling call
0901 01:52:30.421599 (+3750889us) ts_tablet_manager.cc:1145] Deleting tablet 36e10328c34242ebb4dbd8f294162696
0901 01:52:30.421608 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:30.426141 (+ 4533us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:30.507495 (+ 81354us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:30.564577 (+ 57082us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3750692}
I20250901 01:52:30.564924 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 72ba689f9230411995dc96e2b45601df with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:26 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:30.565086 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 2599f98abfdc4871b573edfc2ad97d22 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:25 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:30.565209 2118 tablet_replica.cc:331] T 3363fd67b1ad44849ecb7b4090b61a72 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:30.565311 898 tablet_service.cc:1515] Processing DeleteTablet for tablet f9b293bf6a17453b9ea9892a6970d0b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:25 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:30.565579 887 tablet_service.cc:1515] Processing DeleteTablet for tablet ddbc15685fca4c708329420109ad0c57 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:26 UTC) from {username='slave'} at 127.0.0.1:34472
W20250901 01:52:30.566109 859 leader_election.cc:343] T b13afacd37474781877b9f2cd20b313a P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:30.566215 859 leader_election.cc:304] T b13afacd37474781877b9f2cd20b313a P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:30.566581 859 leader_election.cc:304] T 6c0abbfc8d904dce989ce51961e4237d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:30.566720 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e1c8fe96a6ba4edba4f09c00f7add75e" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:30.566880 2608 raft_consensus.cc:2747] T b13afacd37474781877b9f2cd20b313a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:30.567029 2608 raft_consensus.cc:2802] T 6c0abbfc8d904dce989ce51961e4237d P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:30.567126 2608 raft_consensus.cc:695] T 6c0abbfc8d904dce989ce51961e4237d P 00c282f779c1477d920f24dc0dc16628 [term 1 LEADER]: Becoming Leader. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Running, Role: LEADER
I20250901 01:52:30.567368 2608 consensus_queue.cc:237] T 6c0abbfc8d904dce989ce51961e4237d P 00c282f779c1477d920f24dc0dc16628 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:30.568022 992 leader_election.cc:336] T 31614db4c53a4f3c96fd190d326ce601 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:30.568120 992 leader_election.cc:304] T 31614db4c53a4f3c96fd190d326ce601 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:30.568301 2613 raft_consensus.cc:2747] T 31614db4c53a4f3c96fd190d326ce601 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:30.568786 921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4cde5f4f02044b7bbd7d43b49aac4ae8" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:30.568964 921 raft_consensus.cc:2466] T 4cde5f4f02044b7bbd7d43b49aac4ae8 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:30.569200 898 tablet_service.cc:1515] Processing DeleteTablet for tablet 617cc33d0d23439cb5f1e0c31bf11281 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:24 UTC) from {username='slave'} at 127.0.0.1:34472
W20250901 01:52:30.570042 859 leader_election.cc:343] T e1c8fe96a6ba4edba4f09c00f7add75e P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:30.570139 859 leader_election.cc:304] T e1c8fe96a6ba4edba4f09c00f7add75e P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:30.570605 2408 raft_consensus.cc:491] T f23df6f253a943fa8e146bb8c811ef51 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:30.570714 2408 raft_consensus.cc:513] T f23df6f253a943fa8e146bb8c811ef51 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.571260 2408 leader_election.cc:290] T f23df6f253a943fa8e146bb8c811ef51 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:30.571455 2408 raft_consensus.cc:2747] T e1c8fe96a6ba4edba4f09c00f7add75e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:30.571834 1151 tablet_bootstrap.cc:492] T 879fa66e55544118aafe504587393c66 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:30.571959 1151 ts_tablet_manager.cc:1397] T 879fa66e55544118aafe504587393c66 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.048s user 0.001s sys 0.001s
I20250901 01:52:30.573244 1151 raft_consensus.cc:357] T 879fa66e55544118aafe504587393c66 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.573464 1151 raft_consensus.cc:383] T 879fa66e55544118aafe504587393c66 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:30.573572 1151 raft_consensus.cc:738] T 879fa66e55544118aafe504587393c66 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:30.573825 1151 consensus_queue.cc:260] T 879fa66e55544118aafe504587393c66 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.574411 1151 ts_tablet_manager.cc:1428] T 879fa66e55544118aafe504587393c66 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250901 01:52:30.574745 1151 tablet_bootstrap.cc:492] T 5f52698fb56d472d9e91fb10cb10c69a P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:30.575960 898 tablet_service.cc:1515] Processing DeleteTablet for tablet 88ceba7f48b04d7885cc0164668700a5 with delete_type TABLET_DATA_DELETED (replaced by tablet 9074543dea9949e7a085717b40913348) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:30.576122 1151 tablet_bootstrap.cc:654] T 5f52698fb56d472d9e91fb10cb10c69a P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:30.576272 898 tablet_service.cc:1515] Processing DeleteTablet for tablet d4c9aa723aee4dc4b6237cb5c3722b52 with delete_type TABLET_DATA_DELETED (replaced by tablet f8f923ddcd2740bcb7e521f4f57c4034) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:30.577004 2118 raft_consensus.cc:2241] T 3363fd67b1ad44849ecb7b4090b61a72 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:30.577332 2118 raft_consensus.cc:2270] T 3363fd67b1ad44849ecb7b4090b61a72 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:30.578447 2118 ts_tablet_manager.cc:1905] T 3363fd67b1ad44849ecb7b4090b61a72 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:30.578835 898 tablet_service.cc:1515] Processing DeleteTablet for tablet eee34044c39d4013baf78d3705489626 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:25 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:30.579150 898 tablet_service.cc:1515] Processing DeleteTablet for tablet fa7fb88ded21432a8757f37ec3db5e71 with delete_type TABLET_DATA_DELETED (replaced by tablet 81319e373c9a4475abceeeedb507aabc) from {username='slave'} at 127.0.0.1:34472
W20250901 01:52:30.580691 858 leader_election.cc:336] T f23df6f253a943fa8e146bb8c811ef51 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:30.581549 1491 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet ad45c90ba0da43c9860745e534664a98 (table default.Table_a920a3ffbabf4d83a6563c3423cde12a [id=6fd739b0e725424ca844dbb652ea8028]). Expected version 3 got 2
I20250901 01:52:30.553164 1486 master_service.cc:948] ReplaceTablet: received request to replace tablet 3618a3eb4f174c43805ce7dc0e5315cb from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:30.582366 898 tablet_service.cc:1515] Processing DeleteTablet for tablet da5f69c5b0ca4257ad2ff7918ee2af2e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:26 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:30.583990 992 leader_election.cc:304] T 4cde5f4f02044b7bbd7d43b49aac4ae8 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:30.582036 1491 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 5f5396045979450193d461ba1334a5c9 (table default.Table_2b946a63bd574ac0971fa3d3b9c9a6a4 [id=9c3930e9fcfe4b49ba4bea67a2b0ad07]). Expected version 2 got 0
I20250901 01:52:30.584233 2613 raft_consensus.cc:2802] T 4cde5f4f02044b7bbd7d43b49aac4ae8 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:30.584311 2613 raft_consensus.cc:491] T 4cde5f4f02044b7bbd7d43b49aac4ae8 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:30.584379 2613 raft_consensus.cc:3058] T 4cde5f4f02044b7bbd7d43b49aac4ae8 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:30.584578 898 tablet_service.cc:1515] Processing DeleteTablet for tablet 36e10328c34242ebb4dbd8f294162696 with delete_type TABLET_DATA_DELETED (replaced by tablet 4abf9725be5d42f3893f07f0a9c7614c) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:30.584415 1491 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 2973b59bb1df41d18d9a5b6a29ae336f (table default.Table_7790af96855e45c0a316ba3a4a87c9ed [id=746845bd84c6444b8a1e5683eaba4a68]). Expected version 1 got 0
I20250901 01:52:30.584821 1491 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet d000ba05e12843b4925e622ab55471c0 (table default.Table_e0902dfbdd984f85b3fdac8d283731f7 [id=b43a92ed46164d199b445287ef039bc6]). Expected version 1 got 0
I20250901 01:52:30.584996 1491 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 5d68ea97c71942c7b55b967ba870e0d7 (table default.Table_f610cb75906d4c5cb3b57cfdb80ecefc [id=36915a3e843340ba8ef3578ac35a9af4]). Expected version 1 got 0
I20250901 01:52:30.585222 1491 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet ae21b9ae2e3b472eb67fb5f96388a50f (table default.Table_046289f393ff432aae0b033bdec1fb17 [id=db57b18a9ce24e068b362f5fbe940531]). Expected version 1 got 0
I20250901 01:52:30.585645 898 tablet_service.cc:1515] Processing DeleteTablet for tablet 75a703c8e20a4a2fa0ab6ec58d41ef92 with delete_type TABLET_DATA_DELETED (replaced by tablet 9745582cebfe47b58c21d0191ab4cb9b) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:30.585522 1491 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet e9f1f892658b4736859ff507536920a1 (table default.Table_8825dd28afbe464193eb1b668cc9326a [id=9aa62cb634314a0a849612c54c1220d9]). Expected version 1 got 0
I20250901 01:52:30.585983 1491 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 93bcac1fd95d44dc927abfca81692921 (table default.Table_b8f8ee74648c406eaf589d3ab80934db [id=b8f69313439543be9c11eea88dfa69c5]). Expected version 1 got 0
I20250901 01:52:30.588862 1491 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 61ddcf86c9f34726939cc116434f0822 (table default.Table_63d0f2ead9d946d39ce40162ebd769d6 [id=f9dcf4937b1143248bfa9d88053fce17]). Expected version 3 got 0
I20250901 01:52:30.589179 1491 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 44af4764d4f64e44aec84ccb84be6f5f (table default.Table_9896b980158948339a0b91f5882ed56b [id=a541b55be0dc420a8353ef4793b925b5]). Expected version 1 got 0
I20250901 01:52:30.589531 1491 catalog_manager.cc:5582] T 14b466b2a5a249aa841f1b8b06301edc P 00c282f779c1477d920f24dc0dc16628 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00c282f779c1477d920f24dc0dc16628 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "00c282f779c1477d920f24dc0dc16628" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: HEALTHY } } }
I20250901 01:52:30.590031 1491 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 14b466b2a5a249aa841f1b8b06301edc (table default.Table_a2a8011bb7624d9ea71e7b3169762e43 [id=78cd71321e9a480ca2d7d88327f157ee]). Expected version 3 got 0
I20250901 01:52:30.590466 1491 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 0e4009963cfe42d6ba164fc828897f09 (table default.Table_1adc70cccd2f4cb3a79df96861cb4181 [id=24fffc7ebf5a47bdb37211910301acf7]). Expected version 1 got 0
I20250901 01:52:30.590886 1491 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 6c0abbfc8d904dce989ce51961e4237d (table default.Table_3f9422b08b0349b9a5466218f0fe1b06 [id=f0dfb3811e744bd4ba6d59195cebde6f]). Expected version 2 got 0
I20250901 01:52:30.591198 2613 raft_consensus.cc:513] T 4cde5f4f02044b7bbd7d43b49aac4ae8 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.591166 1491 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 508406601c994e42a5c5b1f665e74957 (table default.Table_345b7f9fe8214aa6878f20a166125364 [id=4aabc328f6374ec5b1dea26f778208b2]). Expected version 1 got 0
I20250901 01:52:30.591661 1491 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet aec951f2cc84499b9162b9fa24d5440e (table default.Table_156d1feafd5246ff8ff1bb7b9ae444a5 [id=55c31cd42fac43e6b8eddc048f565d0a]). Expected version 1 got 0
I20250901 01:52:30.591749 2613 leader_election.cc:290] T 4cde5f4f02044b7bbd7d43b49aac4ae8 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:30.592677 1150 tablet_bootstrap.cc:492] T d4c9aa723aee4dc4b6237cb5c3722b52 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:30.592794 1150 ts_tablet_manager.cc:1397] T d4c9aa723aee4dc4b6237cb5c3722b52 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.035s user 0.000s sys 0.003s
I20250901 01:52:30.592927 898 tablet_service.cc:1515] Processing DeleteTablet for tablet 06d5eeb70e664f73bf643922d445bc7a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:25 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:30.594622 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f23df6f253a943fa8e146bb8c811ef51" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:30.594820 1491 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet b164b4a67a9143ceaff69b0ce49d63fe (table default.Table_da2c70a8f2fe49a0a8184e88c0a9e0ea [id=c1b1e9e3876244f68d31c63ce7bdebf1]). Expected version 1 got 0
W20250901 01:52:30.595055 992 leader_election.cc:336] T 4cde5f4f02044b7bbd7d43b49aac4ae8 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:30.595467 859 leader_election.cc:343] T f23df6f253a943fa8e146bb8c811ef51 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:30.595808 1491 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 858218c4427c41e980b2b5c969a5af40 (table default.Table_6d47266af0ed433eb1f2abbc81d18e69 [id=264cc469a73841e79538843479a165ab]). Expected version 1 got 0
I20250901 01:52:30.595974 2574 tablet.cc:1722] T 7954393c8c1c4457ab8bb723c03a9c8b P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250901 01:52:30.596282 1491 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 4cde5f4f02044b7bbd7d43b49aac4ae8 (table default.Table_d53a456dbca64daf92aeb5fb4517b417 [id=77ae3376281349e087a3b3cee41f2a09]). Expected version 1 got 0
I20250901 01:52:30.596607 1491 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 7954393c8c1c4457ab8bb723c03a9c8b (table default.Table_7050639ebd0147959ac4a251f534eb96 [id=543f2074598d4c29972e564be1123fca]). Expected version 5 got 0
I20250901 01:52:30.597211 1491 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 615b2306fca84003b264da9d5f95ddab (table default.Table_302f6e7cf97d46b9a621eb852e6aefe0 [id=cecdd8d6ca6d445c9fd82decd2a95366]). Expected version 2 got 0
I20250901 01:52:30.597504 1491 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet d0c3c363dda444519a17dca63ad9caa7 (table default.Table_42422c44ca254d8ca989134c2433391f [id=8c1b263824ec4c15a5ec424364c78895]). Expected version 3 got 0
W20250901 01:52:30.597780 1484 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:54603 (request call id 184) took 1366 ms (client timeout 1000 ms). Trace:
I20250901 01:52:30.597875 1491 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 33f637f6a8134215bb1995ec38440073 (table default.Table_3d5afc22c4e14d6990aea31564dfde20 [id=557afe29976d40f6bfb6ae4ca348f887]). Expected version 2 got 0
I20250901 01:52:30.598491 1491 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 45f6e8ba99b74898a2922b36768c6c36 (table default.Table_938421db3cf74cf8b878089bac931818 [id=9ce26dcf307149b595a43b6ac4562fba]). Expected version 1 got 0
W20250901 01:52:30.598162 1484 rpcz_store.cc:269] 0901 01:52:29.231709 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:29.239125 (+ 7416us) service_pool.cc:225] Handling call
0901 01:52:30.597715 (+1358590us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0901 01:52:29.903862 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:29.904021 (+ 159us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:29.904030 (+ 9us) write_op.cc:435] Acquired schema lock
0901 01:52:29.904035 (+ 5us) tablet.cc:660] Decoding operations
0901 01:52:29.904168 (+ 133us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:29.904183 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:29.904191 (+ 8us) tablet.cc:683] Acquiring locks for 1 operations
0901 01:52:29.904263 (+ 72us) tablet.cc:699] Row locks acquired
0901 01:52:29.904267 (+ 4us) write_op.cc:260] PREPARE: finished
0901 01:52:29.904314 (+ 47us) write_op.cc:270] Start()
0901 01:52:29.904356 (+ 42us) write_op.cc:276] Timestamp: P: 1756691549904257 usec, L: 0
0901 01:52:29.904362 (+ 6us) op_driver.cc:348] REPLICATION: starting
0901 01:52:29.904617 (+ 255us) log.cc:844] Serialized 527 byte log entry
0901 01:52:29.948915 (+ 44298us) op_driver.cc:464] REPLICATION: finished
0901 01:52:29.949052 (+ 137us) write_op.cc:301] APPLY: starting
0901 01:52:29.949072 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:29.949118 (+ 46us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:29.949180 (+ 62us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:29.949244 (+ 64us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:29.949286 (+ 42us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0901 01:52:29.949296 (+ 10us) write_op.cc:312] APPLY: finished
0901 01:52:29.949404 (+ 108us) log.cc:844] Serialized 23 byte log entry
0901 01:52:29.949475 (+ 71us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:29.949501 (+ 26us) write_op.cc:454] Released schema lock
0901 01:52:29.949511 (+ 10us) write_op.cc:341] FINISH: Updating metrics
0901 01:52:29.949603 (+ 92us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:29.949610 (+ 7us) write_op.cc:454] Released schema lock
Related trace 'op':
0901 01:52:30.230213 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:30.230296 (+ 83us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:30.230301 (+ 5us) write_op.cc:435] Acquired schema lock
0901 01:52:30.230303 (+ 2us) tablet.cc:660] Decoding operations
0901 01:52:30.230380 (+ 77us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:30.230389 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:30.230392 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0901 01:52:30.230438 (+ 46us) tablet.cc:699] Row locks acquired
0901 01:52:30.230440 (+ 2us) write_op.cc:260] PREPARE: finished
0901 01:52:30.230471 (+ 31us) write_op.cc:270] Start()
0901 01:52:30.230516 (+ 45us) write_op.cc:276] Timestamp: P: 1756691550230445 usec, L: 0
0901 01:52:30.230519 (+ 3us) op_driver.cc:348] REPLICATION: starting
0901 01:52:30.230659 (+ 140us) log.cc:844] Serialized 356 byte log entry
0901 01:52:30.243160 (+ 12501us) op_driver.cc:464] REPLICATION: finished
0901 01:52:30.243260 (+ 100us) write_op.cc:301] APPLY: starting
0901 01:52:30.243274 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:30.243308 (+ 34us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:30.243330 (+ 22us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:30.243376 (+ 46us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:30.243408 (+ 32us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0901 01:52:30.243418 (+ 10us) write_op.cc:312] APPLY: finished
0901 01:52:30.243485 (+ 67us) log.cc:844] Serialized 23 byte log entry
0901 01:52:30.243569 (+ 84us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:30.243581 (+ 12us) write_op.cc:454] Released schema lock
0901 01:52:30.243587 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0901 01:52:30.243649 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:30.243653 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0901 01:52:30.245408 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:30.245510 (+ 102us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:30.245515 (+ 5us) write_op.cc:435] Acquired schema lock
0901 01:52:30.245517 (+ 2us) tablet.cc:660] Decoding operations
0901 01:52:30.245591 (+ 74us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:30.245598 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:30.245603 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0901 01:52:30.245645 (+ 42us) tablet.cc:699] Row locks acquired
0901 01:52:30.245647 (+ 2us) write_op.cc:260] PREPARE: finished
0901 01:52:30.245673 (+ 26us) write_op.cc:270] Start()
0901 01:52:30.245695 (+ 22us) write_op.cc:276] Timestamp: P: 1756691550245646 usec, L: 0
0901 01:52:30.245698 (+ 3us) op_driver.cc:348] REPLICATION: starting
0901 01:52:30.245828 (+ 130us) log.cc:844] Serialized 356 byte log entry
0901 01:52:30.289901 (+ 44073us) op_driver.cc:464] REPLICATION: finished
0901 01:52:30.306100 (+ 16199us) write_op.cc:301] APPLY: starting
0901 01:52:30.306123 (+ 23us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:30.306170 (+ 47us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:30.320126 (+ 13956us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:30.336126 (+ 16000us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:30.336165 (+ 39us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0901 01:52:30.336174 (+ 9us) write_op.cc:312] APPLY: finished
0901 01:52:30.336304 (+ 130us) log.cc:844] Serialized 23 byte log entry
0901 01:52:30.336549 (+ 245us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:30.336564 (+ 15us) write_op.cc:454] Released schema lock
0901 01:52:30.336571 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0901 01:52:30.340791 (+ 4220us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:30.340798 (+ 7us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":164,"child_traces":[["op",{"apply.queue_time_us":80,"apply.run_cpu_time_us":539,"apply.run_wall_time_us":539,"num_ops":1,"prepare.queue_time_us":13508,"prepare.run_cpu_time_us":880,"prepare.run_wall_time_us":888,"raft.queue_time_us":39,"raft.run_cpu_time_us":284,"raft.run_wall_time_us":285,"replication_time_us":44513}],["op",{"apply.queue_time_us":55,"apply.run_cpu_time_us":382,"apply.run_wall_time_us":382,"num_ops":1,"prepare.queue_time_us":4083,"prepare.run_cpu_time_us":511,"prepare.run_wall_time_us":510,"replication_time_us":12611}],["op",{"apply.queue_time_us":26,"apply.run_cpu_time_us":785,"apply.run_wall_time_us":34689,"num_ops":1,"prepare.queue_time_us":50,"prepare.run_cpu_time_us":490,"prepare.run_wall_time_us":490,"replication_time_us":44175}]]}
I20250901 01:52:30.595593 859 leader_election.cc:304] T f23df6f253a943fa8e146bb8c811ef51 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:30.608991 922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4cde5f4f02044b7bbd7d43b49aac4ae8" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:30.609201 922 raft_consensus.cc:3058] T 4cde5f4f02044b7bbd7d43b49aac4ae8 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:30.611946 922 raft_consensus.cc:2466] T 4cde5f4f02044b7bbd7d43b49aac4ae8 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:30.612273 2608 raft_consensus.cc:2747] T f23df6f253a943fa8e146bb8c811ef51 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:30.612447 2608 raft_consensus.cc:491] T 508406601c994e42a5c5b1f665e74957 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:30.612548 2608 raft_consensus.cc:513] T 508406601c994e42a5c5b1f665e74957 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.615423 898 tablet_service.cc:1515] Processing DeleteTablet for tablet 438ae684af234c868a419fbab43b0b98 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:24 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:30.615625 2608 leader_election.cc:290] T 508406601c994e42a5c5b1f665e74957 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:30.616139 898 tablet_service.cc:1515] Processing DeleteTablet for tablet 606fac5c6efe48d1877104b01c23ba5b with delete_type TABLET_DATA_DELETED (replaced by tablet 787be682c0e44248a80d7037fb61dc6c) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:30.616333 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 6cf09e8f299548dba2cce16d7b565bad with delete_type TABLET_DATA_DELETED (replaced by tablet 0e4009963cfe42d6ba164fc828897f09) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:30.616453 2113 ts_tablet_manager.cc:1918] T 9d886fa63f8f4996b087997b542e8bb1 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:30.616523 885 tablet_service.cc:1515] Processing DeleteTablet for tablet f62d96c37cca4143a4b098b7c69979e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:25 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:30.616577 2113 log.cc:1199] T 9d886fa63f8f4996b087997b542e8bb1 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/9d886fa63f8f4996b087997b542e8bb1
I20250901 01:52:30.616875 2113 ts_tablet_manager.cc:1939] T 9d886fa63f8f4996b087997b542e8bb1 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:30.617671 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 126) took 5359 ms. Trace:
I20250901 01:52:30.617761 2113 rpcz_store.cc:276] 0901 01:52:25.258088 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.258170 (+ 82us) service_pool.cc:225] Handling call
0901 01:52:30.523850 (+5265680us) ts_tablet_manager.cc:1145] Deleting tablet 9d886fa63f8f4996b087997b542e8bb1
0901 01:52:30.523860 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:30.550075 (+ 26215us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:30.616318 (+ 66243us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:30.617649 (+ 1331us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5263021}
I20250901 01:52:30.618218 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 9d886fa63f8f4996b087997b542e8bb1 (table default.Table_14cf61059c5145b0a02159aae2a3e457 [id=a33b9b981bd64d7981829d663b0b48b5]) successfully deleted
I20250901 01:52:30.618284 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 131) took 5356 ms. Trace:
I20250901 01:52:30.618351 2113 rpcz_store.cc:276] 0901 01:52:25.261595 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.263089 (+ 1494us) service_pool.cc:225] Handling call
0901 01:52:30.618139 (+5355050us) ts_tablet_manager.cc:1145] Deleting tablet 26c8ed083cea4dc0acca80d00b11b29b
0901 01:52:30.618149 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:30.618270 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5354865}
I20250901 01:52:30.619947 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 246b33338de647689d7404c70676e76f with delete_type TABLET_DATA_DELETED (replaced by tablet ae4844183a3c4a04a775e6fd878391a3) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:30.620107 898 tablet_service.cc:1515] Processing DeleteTablet for tablet 6c05eff658924ca4a9a1e7a37f917d67 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:25 UTC) from {username='slave'} at 127.0.0.1:34472
W20250901 01:52:30.621251 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 26c8ed083cea4dc0acca80d00b11b29b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 26c8ed083cea4dc0acca80d00b11b29b already in progress: creating tablet
I20250901 01:52:30.621516 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2fc7324c386b424d824d62130ead434f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:30.622085 1485 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f5e3316adeae47cd848833dd84b972bb has no consensus state
W20250901 01:52:30.622586 1489 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 315c15cf0afc44d4a79cb29294281d60 consensus state has no leader
I20250901 01:52:30.622522 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c14b2df9a6bb446f844cb778a2390515"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.622946 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4eb7f0d35bc74d4dac6bae4ff06c3345"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.623299 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d4393a00230d427da47e5cdbe29c68f6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.623680 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_847367e6f2c547b4bf551c51d5a6bca7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.624089 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1a50076b40f74f8bac90d05023e02af6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.621580 1492 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: default.Table_f840789a58984799bd8057128f6ea439 [id=4453959e273c473fadd737f56f7db3dd] alter complete (version 1)
I20250901 01:52:30.621830 1490 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet f195b040a6ae4b7b83a8308ab0a12b37
I20250901 01:52:30.624848 1761 raft_consensus.cc:491] T 879fa66e55544118aafe504587393c66 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250901 01:52:30.624877 1490 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet f195b040a6ae4b7b83a8308ab0a12b37: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:30.624981 1761 raft_consensus.cc:513] T 879fa66e55544118aafe504587393c66 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.624943 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_72dbb93b77ca4d578377ff3ff8d77130"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.625627 1761 leader_election.cc:290] T 879fa66e55544118aafe504587393c66 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:30.625597 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_19a2791d10e749bd90c5f1485313fea9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.626124 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_18b87747a9fc479d8853a63286ad660b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.626406 1490 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_fea398de1f1948809aebf95c387f4b17" } modify_external_catalogs: true
I20250901 01:52:30.626616 1490 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_fea398de1f1948809aebf95c387f4b17" } modify_external_catalogs: true
I20250901 01:52:30.626715 1485 master_service.cc:948] ReplaceTablet: received request to replace tablet d1f2d98712ff4ea6aec665512de068bd from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:30.627507 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet f195b040a6ae4b7b83a8308ab0a12b37 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:30 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:30.627702 1761 raft_consensus.cc:491] T 0dbdcb76eeba4fefa56e7835c23c0316 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:30.627828 898 tablet_service.cc:1515] Processing DeleteTablet for tablet f195b040a6ae4b7b83a8308ab0a12b37 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:30 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:30.627861 1761 raft_consensus.cc:513] T 0dbdcb76eeba4fefa56e7835c23c0316 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.628594 1761 leader_election.cc:290] T 0dbdcb76eeba4fefa56e7835c23c0316 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:30.624430 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9ca6f68e71c948f3ae198f2edf369784"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.630414 1150 raft_consensus.cc:357] T d4c9aa723aee4dc4b6237cb5c3722b52 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:30.632522 1150 raft_consensus.cc:383] T d4c9aa723aee4dc4b6237cb5c3722b52 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:30.632613 1150 raft_consensus.cc:738] T d4c9aa723aee4dc4b6237cb5c3722b52 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:30.632925 1150 consensus_queue.cc:260] T d4c9aa723aee4dc4b6237cb5c3722b52 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:30.633603 1150 ts_tablet_manager.cc:1428] T d4c9aa723aee4dc4b6237cb5c3722b52 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.041s user 0.000s sys 0.002s
I20250901 01:52:30.634347 1150 tablet_bootstrap.cc:492] T 3697821cd29446bb908b020599df0db9 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:30.635075 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d3e2ab4af14d463ba3a8fdc36b200b17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:30.636344 858 leader_election.cc:336] T 508406601c994e42a5c5b1f665e74957 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:30.636323 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_bf0a225aba9b4693acf71e51195aff54"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.637102 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5cb3192beb9447c0a9a2c78f24cb405e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.638739 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_56402b10b3b7432596d832f5d6ec15c8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.639430 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_71c0df2b61a844e8a7499cdda47cc785"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.641149 1150 tablet_bootstrap.cc:654] T 3697821cd29446bb908b020599df0db9 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:30.641296 1151 tablet_bootstrap.cc:492] T 5f52698fb56d472d9e91fb10cb10c69a P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:30.641433 1151 ts_tablet_manager.cc:1397] T 5f52698fb56d472d9e91fb10cb10c69a P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.067s user 0.000s sys 0.002s
I20250901 01:52:30.642163 1151 raft_consensus.cc:357] T 5f52698fb56d472d9e91fb10cb10c69a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.642319 1151 raft_consensus.cc:383] T 5f52698fb56d472d9e91fb10cb10c69a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:30.642370 1151 raft_consensus.cc:738] T 5f52698fb56d472d9e91fb10cb10c69a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:30.642603 1151 consensus_queue.cc:260] T 5f52698fb56d472d9e91fb10cb10c69a P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.643208 1151 ts_tablet_manager.cc:1428] T 5f52698fb56d472d9e91fb10cb10c69a P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250901 01:52:30.646175 2582 tablet.cc:1722] T 7954393c8c1c4457ab8bb723c03a9c8b P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250901 01:52:30.648607 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "508406601c994e42a5c5b1f665e74957" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:30.649259 992 leader_election.cc:304] T 4cde5f4f02044b7bbd7d43b49aac4ae8 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:30.649490 2613 raft_consensus.cc:2802] T 4cde5f4f02044b7bbd7d43b49aac4ae8 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:30.649683 2613 raft_consensus.cc:695] T 4cde5f4f02044b7bbd7d43b49aac4ae8 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:30.649907 2613 consensus_queue.cc:237] T 4cde5f4f02044b7bbd7d43b49aac4ae8 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:30.656443 992 leader_election.cc:343] T 879fa66e55544118aafe504587393c66 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 879fa66e55544118aafe504587393c66
W20250901 01:52:30.657037 992 leader_election.cc:343] T 0dbdcb76eeba4fefa56e7835c23c0316 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 0dbdcb76eeba4fefa56e7835c23c0316
I20250901 01:52:30.657018 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4510c0939c3e40ce9a4c7222b659997b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.618670 2113 tablet_replica.cc:331] T 98cca86164db4252b1b49de88747822a P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:30.661125 2574 tablet.cc:1722] T 3618a3eb4f174c43805ce7dc0e5315cb P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250901 01:52:30.663897 2608 raft_consensus.cc:491] T 69f61ec283484f9d8b3eebcea27f741b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:30.664034 2608 raft_consensus.cc:513] T 69f61ec283484f9d8b3eebcea27f741b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:30.664610 2608 leader_election.cc:290] T 69f61ec283484f9d8b3eebcea27f741b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:30.665797 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9f83f3ac399e42bfb4fd6abbf6186635"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.666914 2113 raft_consensus.cc:2241] T 98cca86164db4252b1b49de88747822a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:30.667074 2113 raft_consensus.cc:2270] T 98cca86164db4252b1b49de88747822a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:30.667670 2113 ts_tablet_manager.cc:1905] T 98cca86164db4252b1b49de88747822a P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:30.668000 2412 raft_consensus.cc:491] T 7a2f6e2c7ee2402ea310c058e046416c P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00c282f779c1477d920f24dc0dc16628)
I20250901 01:52:30.668151 2412 raft_consensus.cc:513] T 7a2f6e2c7ee2402ea310c058e046416c P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:30.668198 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a9eed44041d14f0a8e6227b96434d8b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.668825 2412 leader_election.cc:290] T 7a2f6e2c7ee2402ea310c058e046416c P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:30.668725 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ef58b2a54ab440b19c46bf6f5695932a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.670039 1151 tablet_bootstrap.cc:492] T 9fbd44641d9e43c3b592e388e4adf350 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:30.671518 1151 tablet_bootstrap.cc:654] T 9fbd44641d9e43c3b592e388e4adf350 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:30.671418 992 leader_election.cc:336] T 4cde5f4f02044b7bbd7d43b49aac4ae8 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:30.673755 2608 raft_consensus.cc:491] T ae21b9ae2e3b472eb67fb5f96388a50f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:30.673907 2608 raft_consensus.cc:513] T ae21b9ae2e3b472eb67fb5f96388a50f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.674398 2608 leader_election.cc:290] T ae21b9ae2e3b472eb67fb5f96388a50f P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:30.674693 858 leader_election.cc:336] T 69f61ec283484f9d8b3eebcea27f741b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:30.681653 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69f61ec283484f9d8b3eebcea27f741b" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:30.682085 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ae21b9ae2e3b472eb67fb5f96388a50f" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:30.682724 992 leader_election.cc:336] T 879fa66e55544118aafe504587393c66 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:30.682837 992 leader_election.cc:304] T 879fa66e55544118aafe504587393c66 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:30.683033 2613 raft_consensus.cc:2747] T 879fa66e55544118aafe504587393c66 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:30.683650 2118 ts_tablet_manager.cc:1918] T 3363fd67b1ad44849ecb7b4090b61a72 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:30.683790 2118 log.cc:1199] T 3363fd67b1ad44849ecb7b4090b61a72 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/3363fd67b1ad44849ecb7b4090b61a72
I20250901 01:52:30.684098 2118 ts_tablet_manager.cc:1939] T 3363fd67b1ad44849ecb7b4090b61a72 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:30.684556 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 136) took 4012 ms. Trace:
I20250901 01:52:30.684638 2582 tablet.cc:1722] T 3618a3eb4f174c43805ce7dc0e5315cb P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250901 01:52:30.684830 859 leader_election.cc:343] T 508406601c994e42a5c5b1f665e74957 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:30.684938 859 leader_election.cc:304] T 508406601c994e42a5c5b1f665e74957 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:30.685297 2549 raft_consensus.cc:2747] T 508406601c994e42a5c5b1f665e74957 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:30.685395 2608 raft_consensus.cc:491] T 6ba88af177874544b764cc32420e7ac9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:30.685497 2608 raft_consensus.cc:513] T 6ba88af177874544b764cc32420e7ac9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.686003 2608 leader_election.cc:290] T 6ba88af177874544b764cc32420e7ac9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:30.688367 1474 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet f195b040a6ae4b7b83a8308ab0a12b37 on TS 0cf599f74322478d91475df234e58965 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
W20250901 01:52:30.688489 858 leader_election.cc:336] T ae21b9ae2e3b472eb67fb5f96388a50f P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:30.689582 1490 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet abf6e3be60f74872b2308dc95bf9cd37
W20250901 01:52:30.689778 1490 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet abf6e3be60f74872b2308dc95bf9cd37: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:30.690412 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_fb08cd95c602445c94eb17d6f63e3336"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.691295 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_dc3caf6b649f48a79ed94d72d3bedf2e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.691881 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet abf6e3be60f74872b2308dc95bf9cd37 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:30 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:30.692512 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d06ef08e69374575b110e6c4ef4f7d2a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.693428 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_218ea68d110a448589f16c1a91f4daac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.697062 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_de7f6e5ffb59478fbd0b11bc1921c947"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.698686 898 tablet_service.cc:1515] Processing DeleteTablet for tablet abf6e3be60f74872b2308dc95bf9cd37 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:30 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:30.684670 2118 rpcz_store.cc:276] 0901 01:52:26.671857 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.672167 (+ 310us) service_pool.cc:225] Handling call
0901 01:52:30.565159 (+3892992us) ts_tablet_manager.cc:1145] Deleting tablet 3363fd67b1ad44849ecb7b4090b61a72
0901 01:52:30.565168 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:30.582373 (+ 17205us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:30.683542 (+101169us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:30.684528 (+ 986us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11589,"tablet-delete.queue_time_us":3892755}
I20250901 01:52:30.700906 2118 tablet_replica.cc:331] T 500e3c33598b45a4afa983da7cb0845a P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:30.701117 2118 raft_consensus.cc:2241] T 500e3c33598b45a4afa983da7cb0845a P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:30.701289 2118 raft_consensus.cc:2270] T 500e3c33598b45a4afa983da7cb0845a P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250901 01:52:30.701305 858 leader_election.cc:336] T 6ba88af177874544b764cc32420e7ac9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:30.701862 2118 ts_tablet_manager.cc:1905] T 500e3c33598b45a4afa983da7cb0845a P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:30.702419 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_0556b353ff234b7982d5886617e80a1b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.703648 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_752a320f945248b5a6628890ab90ec34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.704375 1486 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 3618a3eb4f174c43805ce7dc0e5315cb
W20250901 01:52:30.704581 1486 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 3618a3eb4f174c43805ce7dc0e5315cb: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
W20250901 01:52:30.704703 992 leader_election.cc:336] T 0dbdcb76eeba4fefa56e7835c23c0316 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:30.704831 992 leader_election.cc:304] T 0dbdcb76eeba4fefa56e7835c23c0316 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:30.705269 2613 raft_consensus.cc:2747] T 0dbdcb76eeba4fefa56e7835c23c0316 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:30.705291 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet 3618a3eb4f174c43805ce7dc0e5315cb with delete_type TABLET_DATA_DELETED (replaced by tablet 5a7fa68d29644ae8afd2711e39dde59a) from {username='slave'} at 127.0.0.1:35946
W20250901 01:52:30.706763 859 leader_election.cc:343] T 69f61ec283484f9d8b3eebcea27f741b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:30.706880 859 leader_election.cc:304] T 69f61ec283484f9d8b3eebcea27f741b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:30.707684 2608 raft_consensus.cc:2747] T 69f61ec283484f9d8b3eebcea27f741b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:30.709502 1151 tablet_bootstrap.cc:492] T 9fbd44641d9e43c3b592e388e4adf350 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:30.709632 1151 ts_tablet_manager.cc:1397] T 9fbd44641d9e43c3b592e388e4adf350 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.040s user 0.004s sys 0.001s
I20250901 01:52:30.710672 1486 catalog_manager.cc:6652] ReplaceTablet: tablet 3618a3eb4f174c43805ce7dc0e5315cb deleted and replaced by tablet 5a7fa68d29644ae8afd2711e39dde59a
I20250901 01:52:30.710712 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5f85a068a0fe40738858849c1177e128"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.714032 1151 raft_consensus.cc:357] T 9fbd44641d9e43c3b592e388e4adf350 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.711062 1486 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_2eba037c989c4095bc355667be2fb109" } new_table_name: "default.Table_608d053d56f7424dbba42177d5e72fe3" modify_external_catalogs: true
I20250901 01:52:30.715180 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_99235fa926994007be458c94c7630b1b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.716392 1151 raft_consensus.cc:383] T 9fbd44641d9e43c3b592e388e4adf350 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:30.716641 1151 raft_consensus.cc:738] T 9fbd44641d9e43c3b592e388e4adf350 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
W20250901 01:52:30.717204 859 leader_election.cc:343] T ae21b9ae2e3b472eb67fb5f96388a50f P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:30.717339 859 leader_election.cc:304] T ae21b9ae2e3b472eb67fb5f96388a50f P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:30.717550 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f0e9004f3e384b78abe2f1f15529a5b5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.717720 2608 raft_consensus.cc:2747] T ae21b9ae2e3b472eb67fb5f96388a50f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:30.718557 992 leader_election.cc:343] T 7a2f6e2c7ee2402ea310c058e046416c P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 7a2f6e2c7ee2402ea310c058e046416c
I20250901 01:52:30.719022 1485 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet d1f2d98712ff4ea6aec665512de068bd
I20250901 01:52:30.716945 1151 consensus_queue.cc:260] T 9fbd44641d9e43c3b592e388e4adf350 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.723335 2574 tablet.cc:1722] T aec951f2cc84499b9162b9fa24d5440e P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:30.729106 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7538c26b16b141eba8574bd5e96d0836"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.731344 1483 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 15e4d972e8254cf899c3e0b71b6e2bd9
W20250901 01:52:30.731547 1483 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 15e4d972e8254cf899c3e0b71b6e2bd9: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:30.732354 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a7bab81bb0b841958b7c47754f116971"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.732795 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet 15e4d972e8254cf899c3e0b71b6e2bd9 with delete_type TABLET_DATA_DELETED (replaced by tablet e4d8330a4e454d1b8d643e17a6996741) from {username='slave'} at 127.0.0.1:35946
W20250901 01:52:30.735515 992 leader_election.cc:336] T 7a2f6e2c7ee2402ea310c058e046416c P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:30.735666 992 leader_election.cc:304] T 7a2f6e2c7ee2402ea310c058e046416c P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:30.735656 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_95a8274165464d3abd1207a1b8b562f1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.735872 2613 raft_consensus.cc:2747] T 7a2f6e2c7ee2402ea310c058e046416c P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250901 01:52:30.736320 1485 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet d1f2d98712ff4ea6aec665512de068bd: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:30.737283 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet d1f2d98712ff4ea6aec665512de068bd with delete_type TABLET_DATA_DELETED (replaced by tablet d602c8986c75492ead9936ac04e2622d) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:30.738173 1485 catalog_manager.cc:6652] ReplaceTablet: tablet d1f2d98712ff4ea6aec665512de068bd deleted and replaced by tablet d602c8986c75492ead9936ac04e2622d
I20250901 01:52:30.738334 1483 catalog_manager.cc:6652] ReplaceTablet: tablet 15e4d972e8254cf899c3e0b71b6e2bd9 deleted and replaced by tablet e4d8330a4e454d1b8d643e17a6996741
I20250901 01:52:30.738533 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_67290ce04370475da3eed3fce1b8ea88"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.741241 1483 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_dfb6e260190a40df8db77152b962cb8e" } modify_external_catalogs: true
I20250901 01:52:30.741369 2113 ts_tablet_manager.cc:1918] T 98cca86164db4252b1b49de88747822a P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:30.741474 2113 log.cc:1199] T 98cca86164db4252b1b49de88747822a P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/98cca86164db4252b1b49de88747822a
I20250901 01:52:30.741417 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3be42d7c48654fb988c081863f8e6940"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.741777 2113 ts_tablet_manager.cc:1939] T 98cca86164db4252b1b49de88747822a P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:30.742816 1485 master_service.cc:948] ReplaceTablet: received request to replace tablet 5d794471dbed4477b9d52fd411fa4d10 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:30.748466 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9ce5cdaa30a14fa6a9a4c1afe75814dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.749166 1485 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 5d794471dbed4477b9d52fd411fa4d10
W20250901 01:52:30.749341 1485 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 5d794471dbed4477b9d52fd411fa4d10: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:30.749657 1485 catalog_manager.cc:6652] ReplaceTablet: tablet 5d794471dbed4477b9d52fd411fa4d10 deleted and replaced by tablet 182d99cb66024b66b16f1a4e76084260
W20250901 01:52:30.750077 992 consensus_peers.cc:489] T 49aaaec611784e6bac131efd58dbca51 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250901 01:52:30.750012 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_cf07c760f5064cf193e4f57c47058a63"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:30.750182 992 consensus_peers.cc:489] T 7442c6ff699141e1aff208ebdc9d192a P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250901 01:52:30.750542 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_093b0c1701534d4fb5975590da0d8c2c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.750710 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet 5d794471dbed4477b9d52fd411fa4d10 with delete_type TABLET_DATA_DELETED (replaced by tablet 182d99cb66024b66b16f1a4e76084260) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:30.751453 2582 tablet.cc:1722] T aec951f2cc84499b9162b9fa24d5440e P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:30.751581 1490 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_302f6e7cf97d46b9a621eb852e6aefe0" } new_table_name: "default.Table_76438e5616384140b57dafdc896f11c5" modify_external_catalogs: true
I20250901 01:52:30.752650 1483 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_dfb6e260190a40df8db77152b962cb8e" } modify_external_catalogs: true
I20250901 01:52:30.753077 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e36887a36c9248d986ca50ce41acd31a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.753989 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 132) took 5492 ms. Trace:
I20250901 01:52:30.754070 2113 rpcz_store.cc:276] 0901 01:52:25.261755 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.263306 (+ 1551us) service_pool.cc:225] Handling call
0901 01:52:30.618626 (+5355320us) ts_tablet_manager.cc:1145] Deleting tablet 98cca86164db4252b1b49de88747822a
0901 01:52:30.618635 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:30.697595 (+ 78960us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:30.741258 (+ 43663us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:30.753969 (+ 12711us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9397,"tablet-delete.queue_time_us":5355202}
I20250901 01:52:30.754515 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ba88af177874544b764cc32420e7ac9" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
E20250901 01:52:30.756646 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 0c88c339e59a4c1da21756cba0645bdd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ce86974fd8974de9a5a737954fac2ad7'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:30.756816 1696 catalog_manager.cc:6205] Error selecting replicas for tablet a656adbf1b29431687e0989f0db3459c, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2d3d2227e6a24bd2a86f457e553e698b'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:30.756906 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 494b6488aa0d4355b4caa46004267465, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_eecfab7239cc40c5a2f1fb6ac6d35dc9'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:30.757025 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 0ee3f35392954817b350e5f48653be78, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_442e11b9211c4b209c68f653d7bfc313'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:30.757099 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae4844183a3c4a04a775e6fd878391a3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_aee74198b51d4f089538c4f2bf3ce4dc'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:30.757187 1696 catalog_manager.cc:6205] Error selecting replicas for tablet be8dfbc466564842a2fdb4953eee939b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_75ba3ba70ca24a94920e7bce96190ff2'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:30.757257 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9745582cebfe47b58c21d0191ab4cb9b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77728af9eee44f29a1e2131d5b75b7ca'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:30.757339 1696 catalog_manager.cc:6205] Error selecting replicas for tablet e2ed32b3292141278af28026528e96c2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b5077f55951d48be92a67407ffe26a08'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:30.757417 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ff2bd017417c41b5b8225b8f00283e63, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_97526c9a6c9f4acbb6dedd78475571e0'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:30.757544 1696 catalog_manager.cc:6205] Error selecting replicas for tablet dcc60bb000f440f1aca9133928b5ce20, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2fbca8463d6546c9b8f5b018fc60251d'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:30.757627 1696 catalog_manager.cc:6205] Error selecting replicas for tablet a2ca69fabadd4ad69e1670cae7a0c503, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_32c4716e3aa24a87a6b1472984b255c4'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:30.757714 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 65e5cdc10e9642b7b0df775cb02861fa, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_1b5729d9768546fe99eaafc4e2604f42'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:30.757798 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f8f923ddcd2740bcb7e521f4f57c4034, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_5e65c5f068b84995a2d7a76fa65dda0f'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:30.757892 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 0813cc68d4b0488ba60a523e4479971b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d1302673d68f424ebb54b220911ce385'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:30.757978 1696 catalog_manager.cc:6205] Error selecting replicas for tablet b1ba0e19dc0b462e80c537bf2d183f5b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_66b111271c15410aab23682f8f6081ac'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:30.758093 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 8e10a01e77d14cf28c2c5e0205e595b9, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_83cf2d86a1284544ba2e7aa9ebb66bb7'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:30.758170 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 05a547a8866149cb95d23e960a737237, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_583e53c2e86544e9be68eb27ab6068a3'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:30.758261 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5f0e054f62de4cf6bc6f69ad6664b192, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ef6832d0845044ed9f165ca3541b4c3f'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:30.758383 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 787be682c0e44248a80d7037fb61dc6c, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_3349df70e80e4caabfd33a2f303cf723'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:30.758469 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae9a4ad4fdfb4771ba23a91f7007447e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_5e84ae9a9c2a4084a1f9c5ffb4594de2'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:30.758545 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9074543dea9949e7a085717b40913348, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_53e30205c5fd488eb3b48c685a159d04'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:30.758641 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 3a62034e688348ed9e2ac9de009ed3a2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_409dad98d7eb455a8e16edfb4acedb34'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:30.758723 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 18498c9cb09141d08c1c5c0854b63ebd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77f4810bc83c4d948531da2a45a3792e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:30.758823 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f5e3316adeae47cd848833dd84b972bb, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7570c3012f8a4ae7bcfc7346eadac92a'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:30.758821 1150 tablet_bootstrap.cc:492] T 3697821cd29446bb908b020599df0db9 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
E20250901 01:52:30.758913 1696 catalog_manager.cc:6205] Error selecting replicas for tablet a6b539e160194da3a849d9e2f1ec9859, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_03fb48eaab9b42858ab7cefb722e43a8'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:30.758946 1150 ts_tablet_manager.cc:1397] T 3697821cd29446bb908b020599df0db9 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.125s user 0.005s sys 0.016s
E20250901 01:52:30.759013 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 66719d8e9c614657820440ebe8ce7c1e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b3e694397be14321a668e46c09c51a71'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:30.759089 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 6ce9521b925b4f97bb4a810ce77e147e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_9e04f976a93f4d30baccf90fa3622dfe'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:30.759183 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2f61edb50d7a42198bfd70639c7bf4de, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_a5c783cbdb694af69047b31d99ff8def'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:30.768324 1150 raft_consensus.cc:357] T 3697821cd29446bb908b020599df0db9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.769158 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_15d2e41c8b1549319c86309577feb000"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.770210 2113 tablet_replica.cc:331] T bb0cf5a6a27045a08df6845d7473184d P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:30.770438 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_48afc95ec0264bbc9f2e6f74edd14304"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.770901 1489 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_32c4716e3aa24a87a6b1472984b255c4" } modify_external_catalogs: true
I20250901 01:52:30.771086 1489 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_32c4716e3aa24a87a6b1472984b255c4" } modify_external_catalogs: true
W20250901 01:52:30.771317 859 leader_election.cc:343] T 6ba88af177874544b764cc32420e7ac9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:30.771607 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_08c5bbeab251477ea62e35054e3aa06a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.771744 859 leader_election.cc:304] T 6ba88af177874544b764cc32420e7ac9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:30.770576 2613 raft_consensus.cc:491] T 4902062977ea42149e3466e999117322 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:30.771981 2613 raft_consensus.cc:513] T 4902062977ea42149e3466e999117322 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.772166 2608 raft_consensus.cc:2747] T 6ba88af177874544b764cc32420e7ac9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:30.772171 1150 raft_consensus.cc:383] T 3697821cd29446bb908b020599df0db9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:30.772523 1150 raft_consensus.cc:738] T 3697821cd29446bb908b020599df0db9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:30.772590 2613 leader_election.cc:290] T 4902062977ea42149e3466e999117322 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:30.770422 2113 raft_consensus.cc:2241] T bb0cf5a6a27045a08df6845d7473184d P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:30.772820 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c92132e1b0ff4a29958115c8abfd7d49"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.772940 1761 consensus_queue.cc:1035] T ff013ea889d44cba8797834e4611b489 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:30.773058 2113 raft_consensus.cc:2270] T bb0cf5a6a27045a08df6845d7473184d P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250901 01:52:30.773128 858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:40731, user_credentials={real_user=slave}} blocked reactor thread for 71837us
I20250901 01:52:30.772722 1150 consensus_queue.cc:260] T 3697821cd29446bb908b020599df0db9 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.773835 1150 ts_tablet_manager.cc:1428] T 3697821cd29446bb908b020599df0db9 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20250901 01:52:30.774226 1150 tablet_bootstrap.cc:492] T 34ebb441c27747ec860ff2e25f9a0adc P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:30.774379 2113 ts_tablet_manager.cc:1905] T bb0cf5a6a27045a08df6845d7473184d P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:30.776120 2118 ts_tablet_manager.cc:1918] T 500e3c33598b45a4afa983da7cb0845a P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:30.772040 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ee7d48f7cb1242dd90dd22950be4ca52"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.774797 1151 ts_tablet_manager.cc:1428] T 9fbd44641d9e43c3b592e388e4adf350 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.065s user 0.002s sys 0.000s
I20250901 01:52:30.779582 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3c50afe92fb845a69e071c89b8b12e61"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.779824 1151 tablet_bootstrap.cc:492] T 5afae316ff164413be0d25d4bc0f58dc P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:30.780515 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f518d92b63d44e80b9e8d0464a18ca92"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.781180 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8a6498b26eed434a9d303968c444aa1f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.781641 1151 tablet_bootstrap.cc:654] T 5afae316ff164413be0d25d4bc0f58dc P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:30.781901 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a6469136d64549ef8ca29c13408f496b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.776522 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 3363fd67b1ad44849ecb7b4090b61a72 (table default.Table_09ffc2a7b4b14b25adbc5471363c02af [id=eaa8142db4844b908cf36991898aca3c]) successfully deleted
I20250901 01:52:30.783412 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_71a4d6c6f0934b85a3ff5ea6d3629e72"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.787364 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3eba254193574b17828b829f6df4a652"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.789994 2118 log.cc:1199] T 500e3c33598b45a4afa983da7cb0845a P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/500e3c33598b45a4afa983da7cb0845a
I20250901 01:52:30.784817 1484 master_service.cc:948] ReplaceTablet: received request to replace tablet 9fbd44641d9e43c3b592e388e4adf350 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:30.790439 2118 ts_tablet_manager.cc:1939] T 500e3c33598b45a4afa983da7cb0845a P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:30.790889 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d4e92831aff4402c95f5745ab8f8a473"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.791250 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 137) took 4106 ms. Trace:
I20250901 01:52:30.791368 2118 rpcz_store.cc:276] 0901 01:52:26.684530 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.684580 (+ 50us) service_pool.cc:225] Handling call
0901 01:52:30.700854 (+4016274us) ts_tablet_manager.cc:1145] Deleting tablet 500e3c33598b45a4afa983da7cb0845a
0901 01:52:30.700864 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:30.749120 (+ 48256us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:30.776020 (+ 26900us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:30.791231 (+ 15211us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4016084}
I20250901 01:52:30.791779 2549 raft_consensus.cc:491] T e9f1f892658b4736859ff507536920a1 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:30.783756 1150 tablet_bootstrap.cc:654] T 34ebb441c27747ec860ff2e25f9a0adc P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:30.791911 2549 raft_consensus.cc:513] T e9f1f892658b4736859ff507536920a1 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.792471 2549 leader_election.cc:290] T e9f1f892658b4736859ff507536920a1 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:30.798666 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_443a3711b02e4473a14290e3c7e06e56"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.799001 1150 tablet_bootstrap.cc:492] T 34ebb441c27747ec860ff2e25f9a0adc P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:30.799124 1150 ts_tablet_manager.cc:1397] T 34ebb441c27747ec860ff2e25f9a0adc P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.000s
I20250901 01:52:30.799628 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1afc5ef59e524550af9ea20aa3264dde"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.799855 1150 raft_consensus.cc:357] T 34ebb441c27747ec860ff2e25f9a0adc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.800011 1150 raft_consensus.cc:383] T 34ebb441c27747ec860ff2e25f9a0adc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:30.800065 1150 raft_consensus.cc:738] T 34ebb441c27747ec860ff2e25f9a0adc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:30.800231 1150 consensus_queue.cc:260] T 34ebb441c27747ec860ff2e25f9a0adc P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.800509 2357 raft_consensus.cc:491] T d1f2d98712ff4ea6aec665512de068bd P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:30.800540 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d3ce17d125cb4b59bc67583156635137"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.800616 2357 raft_consensus.cc:513] T d1f2d98712ff4ea6aec665512de068bd P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.800676 2549 consensus_queue.cc:1035] T 14b466b2a5a249aa841f1b8b06301edc P 00c282f779c1477d920f24dc0dc16628 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:30.801098 2357 leader_election.cc:290] T d1f2d98712ff4ea6aec665512de068bd P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:30.801530 1150 ts_tablet_manager.cc:1428] T 34ebb441c27747ec860ff2e25f9a0adc P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250901 01:52:30.802171 2613 raft_consensus.cc:491] T 9fbd44641d9e43c3b592e388e4adf350 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:30.790005 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_34c88ff045d647a097786373eafc0a04"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.802325 2613 raft_consensus.cc:513] T 9fbd44641d9e43c3b592e388e4adf350 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.802870 2613 leader_election.cc:290] T 9fbd44641d9e43c3b592e388e4adf350 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:30.803290 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_620b2648865442c4a6cfa9527eede837"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:30.803865 992 leader_election.cc:343] T 4902062977ea42149e3466e999117322 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 4902062977ea42149e3466e999117322
I20250901 01:52:30.804888 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c3bbec277da64c09919861a2889f85ec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.806596 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_706baa67b01b4ee983d5be5ba932d308"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.807518 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_23d0b446d00e4fe496c458cc065a463d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.791819 2118 tablet_replica.cc:331] T 42c1156e8dd84c6fa1396b46de13eb6f P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:30.816913 2118 raft_consensus.cc:2241] T 42c1156e8dd84c6fa1396b46de13eb6f P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:30.818893 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9f1f892658b4736859ff507536920a1" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:30.821385 2608 raft_consensus.cc:491] T 9fbd44641d9e43c3b592e388e4adf350 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:30.821549 2608 raft_consensus.cc:513] T 9fbd44641d9e43c3b592e388e4adf350 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.822286 2608 leader_election.cc:290] T 9fbd44641d9e43c3b592e388e4adf350 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:30.822667 919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9fbd44641d9e43c3b592e388e4adf350" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:30.822872 919 raft_consensus.cc:2466] T 9fbd44641d9e43c3b592e388e4adf350 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:30.831714 1055 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1f2d98712ff4ea6aec665512de068bd" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:30.831969 1489 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet a2ca69fabadd4ad69e1670cae7a0c503
I20250901 01:52:30.831969 2608 raft_consensus.cc:491] T d0c3c363dda444519a17dca63ad9caa7 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:30.832120 1054 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9fbd44641d9e43c3b592e388e4adf350" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:30.832095 2608 raft_consensus.cc:513] T d0c3c363dda444519a17dca63ad9caa7 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:30.819638 2118 raft_consensus.cc:2270] T 42c1156e8dd84c6fa1396b46de13eb6f P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:30.832284 1054 raft_consensus.cc:2466] T 9fbd44641d9e43c3b592e388e4adf350 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00c282f779c1477d920f24dc0dc16628 in term 0.
I20250901 01:52:30.832652 2608 leader_election.cc:290] T d0c3c363dda444519a17dca63ad9caa7 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:30.833734 1696 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet a2ca69fabadd4ad69e1670cae7a0c503
W20250901 01:52:30.835994 992 leader_election.cc:336] T 4902062977ea42149e3466e999117322 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:30.836139 992 leader_election.cc:304] T 4902062977ea42149e3466e999117322 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:30.837117 2409 raft_consensus.cc:491] T d0c3c363dda444519a17dca63ad9caa7 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:30.837258 2409 raft_consensus.cc:513] T d0c3c363dda444519a17dca63ad9caa7 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:30.837808 2409 leader_election.cc:290] T d0c3c363dda444519a17dca63ad9caa7 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:30.838228 1484 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_142ccc8cc7414621bc0c31326d5aa0e9" } new_table_name: "default.Table_a4f42e5055e6416caf95b93122822c6f" modify_external_catalogs: true
I20250901 01:52:30.838183 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7ed3c0fc6f3c4f6f9e87e790193a86ae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.839525 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_fcc58c45c4414f83af8a5cc6ff97065a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.840004 2118 ts_tablet_manager.cc:1905] T 42c1156e8dd84c6fa1396b46de13eb6f P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:30.840607 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_20dbd807ef7b437c94a42d9b4c90b93b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.841624 898 tablet_service.cc:1515] Processing DeleteTablet for tablet 3618a3eb4f174c43805ce7dc0e5315cb with delete_type TABLET_DATA_DELETED (replaced by tablet 5a7fa68d29644ae8afd2711e39dde59a) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:30.841863 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 15e4d972e8254cf899c3e0b71b6e2bd9 with delete_type TABLET_DATA_DELETED (replaced by tablet e4d8330a4e454d1b8d643e17a6996741) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:30.842065 885 tablet_service.cc:1515] Processing DeleteTablet for tablet d1f2d98712ff4ea6aec665512de068bd with delete_type TABLET_DATA_DELETED (replaced by tablet d602c8986c75492ead9936ac04e2622d) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:30.842095 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 500e3c33598b45a4afa983da7cb0845a (table default.Table_8ac222c93e36436c9a3ecb2c489f6d74 [id=44b75816b8b24af0a9530d4a12c2416e]) successfully deleted
I20250901 01:52:30.842233 898 tablet_service.cc:1515] Processing DeleteTablet for tablet 5d794471dbed4477b9d52fd411fa4d10 with delete_type TABLET_DATA_DELETED (replaced by tablet 182d99cb66024b66b16f1a4e76084260) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:30.844827 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8d9e2cf80ca6474a8d73288171a3e334"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.845463 1150 tablet_bootstrap.cc:492] T b4278a013ae84082bc7a8497b04ac147 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:30.846948 2113 ts_tablet_manager.cc:1918] T bb0cf5a6a27045a08df6845d7473184d P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:30.847070 2113 log.cc:1199] T bb0cf5a6a27045a08df6845d7473184d P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/bb0cf5a6a27045a08df6845d7473184d
I20250901 01:52:30.847335 1150 tablet_bootstrap.cc:654] T b4278a013ae84082bc7a8497b04ac147 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:30.847395 2113 ts_tablet_manager.cc:1939] T bb0cf5a6a27045a08df6845d7473184d P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:30.847635 2409 raft_consensus.cc:2747] T 4902062977ea42149e3466e999117322 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:30.850015 859 leader_election.cc:343] T e9f1f892658b4736859ff507536920a1 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:30.850030 992 leader_election.cc:304] T 9fbd44641d9e43c3b592e388e4adf350 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:30.851142 2409 raft_consensus.cc:2802] T 9fbd44641d9e43c3b592e388e4adf350 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:30.851239 2409 raft_consensus.cc:491] T 9fbd44641d9e43c3b592e388e4adf350 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:30.851315 2409 raft_consensus.cc:3058] T 9fbd44641d9e43c3b592e388e4adf350 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:30.853870 2409 raft_consensus.cc:513] T 9fbd44641d9e43c3b592e388e4adf350 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.854378 2409 leader_election.cc:290] T 9fbd44641d9e43c3b592e388e4adf350 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:30.856420 859 leader_election.cc:343] T d1f2d98712ff4ea6aec665512de068bd P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:30.856601 859 leader_election.cc:304] T 9fbd44641d9e43c3b592e388e4adf350 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:30.856819 2608 raft_consensus.cc:2802] T 9fbd44641d9e43c3b592e388e4adf350 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:30.856878 2608 raft_consensus.cc:491] T 9fbd44641d9e43c3b592e388e4adf350 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:30.856932 2608 raft_consensus.cc:3058] T 9fbd44641d9e43c3b592e388e4adf350 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:30.858448 919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0c3c363dda444519a17dca63ad9caa7" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:30.858623 919 raft_consensus.cc:2466] T d0c3c363dda444519a17dca63ad9caa7 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
W20250901 01:52:30.859092 992 leader_election.cc:336] T 9fbd44641d9e43c3b592e388e4adf350 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:30.859402 992 consensus_peers.cc:489] T ff013ea889d44cba8797834e4611b489 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:30.859359 2608 raft_consensus.cc:513] T 9fbd44641d9e43c3b592e388e4adf350 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.859941 2608 leader_election.cc:290] T 9fbd44641d9e43c3b592e388e4adf350 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:30.864039 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_421bee72eefd4dc59bfeb50db4b4a83f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.864298 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 133) took 5600 ms. Trace:
I20250901 01:52:30.865052 917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9fbd44641d9e43c3b592e388e4adf350" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:30.865300 917 raft_consensus.cc:2391] T 9fbd44641d9e43c3b592e388e4adf350 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate fd27ab6665304aaea53adbd102dd4f58 in current term 1: Already voted for candidate 00c282f779c1477d920f24dc0dc16628 in this term.
I20250901 01:52:30.865691 2113 rpcz_store.cc:276] 0901 01:52:25.261862 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.263451 (+ 1589us) service_pool.cc:225] Handling call
0901 01:52:30.770152 (+5506701us) ts_tablet_manager.cc:1145] Deleting tablet bb0cf5a6a27045a08df6845d7473184d
0901 01:52:30.770164 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:30.787173 (+ 17009us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:30.846850 (+ 59677us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:30.862263 (+ 15413us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":110,"tablet-delete.queue_time_us":5506579}
I20250901 01:52:30.867882 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet bb0cf5a6a27045a08df6845d7473184d (table default.Table_b23a4778615d4597950ce1709ab00d37 [id=bf59a9514b7d49c3b3c1e9c6064ef369]) successfully deleted
I20250901 01:52:30.868209 1054 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9fbd44641d9e43c3b592e388e4adf350" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58"
I20250901 01:52:30.868536 1054 raft_consensus.cc:2391] T 9fbd44641d9e43c3b592e388e4adf350 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 00c282f779c1477d920f24dc0dc16628 in current term 1: Already voted for candidate fd27ab6665304aaea53adbd102dd4f58 in this term.
I20250901 01:52:30.868700 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0c3c363dda444519a17dca63ad9caa7" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:30.868904 1052 raft_consensus.cc:2466] T d0c3c363dda444519a17dca63ad9caa7 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00c282f779c1477d920f24dc0dc16628 in term 1.
W20250901 01:52:30.870697 992 leader_election.cc:336] T d0c3c363dda444519a17dca63ad9caa7 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:30.874572 992 leader_election.cc:304] T d0c3c363dda444519a17dca63ad9caa7 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:30.876155 2118 ts_tablet_manager.cc:1918] T 42c1156e8dd84c6fa1396b46de13eb6f P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 01:52:30.874810 1151 tablet_bootstrap.cc:492] T 5afae316ff164413be0d25d4bc0f58dc P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:30.877192 1151 ts_tablet_manager.cc:1397] T 5afae316ff164413be0d25d4bc0f58dc P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.097s user 0.005s sys 0.003s
I20250901 01:52:30.877290 2409 raft_consensus.cc:2802] T d0c3c363dda444519a17dca63ad9caa7 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250901 01:52:30.878077 2409 raft_consensus.cc:491] T d0c3c363dda444519a17dca63ad9caa7 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:30.878168 2409 raft_consensus.cc:3058] T d0c3c363dda444519a17dca63ad9caa7 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Advancing to term 2
I20250901 01:52:30.878053 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_0344e4b19e364ad39a705a3ec723e9b6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:30.880019 1489 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_fd17a2ad225546aa821489ce94a788a2" } new_table_name: "default.Table_685b7ca5d6e14076ad261d156b30858b" modify_external_catalogs: true
I20250901 01:52:30.880988 2409 raft_consensus.cc:513] T d0c3c363dda444519a17dca63ad9caa7 P fd27ab6665304aaea53adbd102dd4f58 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:30.881435 1761 raft_consensus.cc:491] T 3239e58531d445fa80165cf678bc277c P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:30.881604 1761 raft_consensus.cc:513] T 3239e58531d445fa80165cf678bc277c P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.881820 2409 leader_election.cc:290] T d0c3c363dda444519a17dca63ad9caa7 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 2 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:30.882131 1761 leader_election.cc:290] T 3239e58531d445fa80165cf678bc277c P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:30.882891 2118 log.cc:1199] T 42c1156e8dd84c6fa1396b46de13eb6f P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/42c1156e8dd84c6fa1396b46de13eb6f
I20250901 01:52:30.882927 1151 raft_consensus.cc:357] T 5afae316ff164413be0d25d4bc0f58dc P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.883111 1151 raft_consensus.cc:383] T 5afae316ff164413be0d25d4bc0f58dc P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:30.883183 1151 raft_consensus.cc:738] T 5afae316ff164413be0d25d4bc0f58dc P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:30.883376 1151 consensus_queue.cc:260] T 5afae316ff164413be0d25d4bc0f58dc P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.879114 859 leader_election.cc:304] T d0c3c363dda444519a17dca63ad9caa7 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:30.883908 1151 ts_tablet_manager.cc:1428] T 5afae316ff164413be0d25d4bc0f58dc P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250901 01:52:30.884086 2118 ts_tablet_manager.cc:1939] T 42c1156e8dd84c6fa1396b46de13eb6f P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:30.884182 1151 tablet_bootstrap.cc:492] T abf6e3be60f74872b2308dc95bf9cd37 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:30.884186 2357 raft_consensus.cc:2802] T d0c3c363dda444519a17dca63ad9caa7 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250901 01:52:30.884253 2357 raft_consensus.cc:491] T d0c3c363dda444519a17dca63ad9caa7 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:30.884310 2357 raft_consensus.cc:3058] T d0c3c363dda444519a17dca63ad9caa7 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Advancing to term 2
I20250901 01:52:30.884598 1485 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_364cf1adeb2c4102aad338f2e228fb0b" } new_table_name: "default.Table_2755f7caf53340afbd230181643366db" modify_external_catalogs: true
I20250901 01:52:30.884886 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 140) took 4035 ms. Trace:
I20250901 01:52:30.884958 2118 rpcz_store.cc:276] 0901 01:52:26.849895 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.849962 (+ 67us) service_pool.cc:225] Handling call
0901 01:52:30.791769 (+3941807us) ts_tablet_manager.cc:1145] Deleting tablet 42c1156e8dd84c6fa1396b46de13eb6f
0901 01:52:30.791779 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:30.843179 (+ 51400us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:30.876048 (+ 32869us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:30.884861 (+ 8813us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3941508}
I20250901 01:52:30.885249 2118 tablet_replica.cc:331] T 7442c6ff699141e1aff208ebdc9d192a P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:30.885469 2118 raft_consensus.cc:2241] T 7442c6ff699141e1aff208ebdc9d192a P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:30.885643 2118 raft_consensus.cc:2270] T 7442c6ff699141e1aff208ebdc9d192a P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:30.886173 2118 ts_tablet_manager.cc:1905] T 7442c6ff699141e1aff208ebdc9d192a P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:30.883430 992 leader_election.cc:336] T 9fbd44641d9e43c3b592e388e4adf350 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:30.887343 2357 raft_consensus.cc:513] T d0c3c363dda444519a17dca63ad9caa7 P 00c282f779c1477d920f24dc0dc16628 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:30.887705 992 leader_election.cc:304] T 9fbd44641d9e43c3b592e388e4adf350 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:30.888023 2357 leader_election.cc:290] T d0c3c363dda444519a17dca63ad9caa7 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 2 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:30.888862 1761 raft_consensus.cc:2747] T 9fbd44641d9e43c3b592e388e4adf350 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250901 01:52:30.891057 918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0c3c363dda444519a17dca63ad9caa7" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:30.891319 918 raft_consensus.cc:2391] T d0c3c363dda444519a17dca63ad9caa7 P 00c282f779c1477d920f24dc0dc16628 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate fd27ab6665304aaea53adbd102dd4f58 in current term 2: Already voted for candidate 00c282f779c1477d920f24dc0dc16628 in this term.
I20250901 01:52:30.892051 918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3239e58531d445fa80165cf678bc277c" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:30.892246 918 raft_consensus.cc:2466] T 3239e58531d445fa80165cf678bc277c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:30.893332 1151 tablet_bootstrap.cc:654] T abf6e3be60f74872b2308dc95bf9cd37 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:30.892640 858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:40731, user_credentials={real_user=slave}} blocked reactor thread for 48291.7us
I20250901 01:52:30.895377 992 leader_election.cc:304] T 3239e58531d445fa80165cf678bc277c P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:30.898834 1761 raft_consensus.cc:2802] T 3239e58531d445fa80165cf678bc277c P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:30.898953 1761 raft_consensus.cc:491] T 3239e58531d445fa80165cf678bc277c P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:30.899044 1761 raft_consensus.cc:3058] T 3239e58531d445fa80165cf678bc277c P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:30.901705 1761 raft_consensus.cc:513] T 3239e58531d445fa80165cf678bc277c P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.902437 1761 leader_election.cc:290] T 3239e58531d445fa80165cf678bc277c P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:30.902801 1761 consensus_queue.cc:1035] T d10c193452954c198bba686169b8025b P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:30.903754 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0c3c363dda444519a17dca63ad9caa7" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58"
I20250901 01:52:30.903975 1052 raft_consensus.cc:2391] T d0c3c363dda444519a17dca63ad9caa7 P fd27ab6665304aaea53adbd102dd4f58 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 00c282f779c1477d920f24dc0dc16628 in current term 2: Already voted for candidate fd27ab6665304aaea53adbd102dd4f58 in this term.
I20250901 01:52:30.906041 922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3239e58531d445fa80165cf678bc277c" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:30.906255 922 raft_consensus.cc:3058] T 3239e58531d445fa80165cf678bc277c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:30.908891 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 42c1156e8dd84c6fa1396b46de13eb6f (table default.Table_b5d681f51fc84395bbabe1e5b4cfb358 [id=40d05d90b0354334840ad17764f5b6bb]) successfully deleted
W20250901 01:52:30.909411 858 leader_election.cc:336] T d1f2d98712ff4ea6aec665512de068bd P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:30.910251 992 leader_election.cc:336] T 3239e58531d445fa80165cf678bc277c P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:30.910508 992 leader_election.cc:336] T d0c3c363dda444519a17dca63ad9caa7 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:30.910638 992 leader_election.cc:304] T d0c3c363dda444519a17dca63ad9caa7 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:30.912111 922 raft_consensus.cc:2466] T 3239e58531d445fa80165cf678bc277c P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:30.912091 858 leader_election.cc:304] T d1f2d98712ff4ea6aec665512de068bd P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:30.912694 1761 raft_consensus.cc:2747] T d0c3c363dda444519a17dca63ad9caa7 P fd27ab6665304aaea53adbd102dd4f58 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250901 01:52:30.912848 2608 raft_consensus.cc:2747] T d1f2d98712ff4ea6aec665512de068bd P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:30.913338 858 leader_election.cc:336] T d0c3c363dda444519a17dca63ad9caa7 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:30.913830 858 leader_election.cc:336] T 9fbd44641d9e43c3b592e388e4adf350 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:30.915174 992 leader_election.cc:304] T 3239e58531d445fa80165cf678bc277c P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
W20250901 01:52:30.915571 858 leader_election.cc:336] T e9f1f892658b4736859ff507536920a1 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:30.916322 858 leader_election.cc:304] T e9f1f892658b4736859ff507536920a1 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:30.872023 2113 tablet_replica.cc:331] T d58f69fad01f403eab74a47d3695ed83 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:30.916788 2608 raft_consensus.cc:2747] T e9f1f892658b4736859ff507536920a1 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:30.916965 1761 raft_consensus.cc:2802] T 3239e58531d445fa80165cf678bc277c P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:30.917117 1761 raft_consensus.cc:695] T 3239e58531d445fa80165cf678bc277c P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:30.917359 1761 consensus_queue.cc:237] T 3239e58531d445fa80165cf678bc277c P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.920563 2608 raft_consensus.cc:491] T 56f2c4d2e6ab4bb4bdd6d025843589b6 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:30.920730 2608 raft_consensus.cc:513] T 56f2c4d2e6ab4bb4bdd6d025843589b6 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:30.921272 992 leader_election.cc:336] T 3239e58531d445fa80165cf678bc277c P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:30.921315 2608 leader_election.cc:290] T 56f2c4d2e6ab4bb4bdd6d025843589b6 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:30.922442 2323 raft_consensus.cc:491] T b164b4a67a9143ceaff69b0ce49d63fe P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:30.922598 2323 raft_consensus.cc:513] T b164b4a67a9143ceaff69b0ce49d63fe P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:30.923326 2323 leader_election.cc:290] T b164b4a67a9143ceaff69b0ce49d63fe P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:30.925220 2113 raft_consensus.cc:2241] T d58f69fad01f403eab74a47d3695ed83 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
W20250901 01:52:30.925539 858 leader_election.cc:336] T 9fbd44641d9e43c3b592e388e4adf350 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:30.925684 858 leader_election.cc:304] T 9fbd44641d9e43c3b592e388e4adf350 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:30.925954 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "56f2c4d2e6ab4bb4bdd6d025843589b6" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:30.926394 859 leader_election.cc:343] T 56f2c4d2e6ab4bb4bdd6d025843589b6 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:30.926656 2323 raft_consensus.cc:2747] T 9fbd44641d9e43c3b592e388e4adf350 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20250901 01:52:30.926991 858 leader_election.cc:336] T d0c3c363dda444519a17dca63ad9caa7 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:30.927150 858 leader_election.cc:304] T d0c3c363dda444519a17dca63ad9caa7 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:30.927758 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b164b4a67a9143ceaff69b0ce49d63fe" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:30.925979 2113 raft_consensus.cc:2270] T d58f69fad01f403eab74a47d3695ed83 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:30.928083 2323 raft_consensus.cc:2747] T d0c3c363dda444519a17dca63ad9caa7 P 00c282f779c1477d920f24dc0dc16628 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20250901 01:52:30.928418 858 consensus_peers.cc:489] T 14b466b2a5a249aa841f1b8b06301edc P 00c282f779c1477d920f24dc0dc16628 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250901 01:52:30.929278 858 leader_election.cc:336] T 56f2c4d2e6ab4bb4bdd6d025843589b6 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:30.929893 858 leader_election.cc:304] T 56f2c4d2e6ab4bb4bdd6d025843589b6 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:30.931856 1482 catalog_manager.cc:5582] T ed1b41340ce742a88c3e4fd85cd25ca6 P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: UNKNOWN } } }
I20250901 01:52:30.932475 2323 raft_consensus.cc:2747] T 56f2c4d2e6ab4bb4bdd6d025843589b6 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:30.933677 2113 ts_tablet_manager.cc:1905] T d58f69fad01f403eab74a47d3695ed83 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:30.933935 858 leader_election.cc:336] T b164b4a67a9143ceaff69b0ce49d63fe P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:30.934175 1151 tablet_bootstrap.cc:492] T abf6e3be60f74872b2308dc95bf9cd37 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:30.935091 1150 tablet_bootstrap.cc:492] T b4278a013ae84082bc7a8497b04ac147 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:30.935233 1150 ts_tablet_manager.cc:1397] T b4278a013ae84082bc7a8497b04ac147 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.090s user 0.006s sys 0.006s
I20250901 01:52:30.935962 1150 raft_consensus.cc:357] T b4278a013ae84082bc7a8497b04ac147 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.936149 1150 raft_consensus.cc:383] T b4278a013ae84082bc7a8497b04ac147 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:30.936231 1150 raft_consensus.cc:738] T b4278a013ae84082bc7a8497b04ac147 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:30.936420 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet ed1b41340ce742a88c3e4fd85cd25ca6 (table default.Table_ce2f3b515c8f4c1bb557c5ed84b48dba [id=0baa4993b72e4a8d94ee3059c9d2db6c]). Expected version 2 got 0
I20250901 01:52:30.936503 1150 consensus_queue.cc:260] T b4278a013ae84082bc7a8497b04ac147 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.937083 1150 ts_tablet_manager.cc:1428] T b4278a013ae84082bc7a8497b04ac147 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250901 01:52:30.937064 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 14b466b2a5a249aa841f1b8b06301edc (table default.Table_a2a8011bb7624d9ea71e7b3169762e43 [id=78cd71321e9a480ca2d7d88327f157ee]). Expected version 3 got 0
I20250901 01:52:30.937451 1151 ts_tablet_manager.cc:1397] T abf6e3be60f74872b2308dc95bf9cd37 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.053s user 0.001s sys 0.005s
I20250901 01:52:30.937897 1150 tablet_bootstrap.cc:492] T 88ceba7f48b04d7885cc0164668700a5 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:30.937500 1482 catalog_manager.cc:5582] T 33f637f6a8134215bb1995ec38440073 P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: UNKNOWN } } }
I20250901 01:52:30.938688 1151 raft_consensus.cc:357] T abf6e3be60f74872b2308dc95bf9cd37 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.938855 1151 raft_consensus.cc:383] T abf6e3be60f74872b2308dc95bf9cd37 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:30.938941 1151 raft_consensus.cc:738] T abf6e3be60f74872b2308dc95bf9cd37 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:30.939103 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 33f637f6a8134215bb1995ec38440073 (table default.Table_3d5afc22c4e14d6990aea31564dfde20 [id=557afe29976d40f6bfb6ae4ca348f887]). Expected version 2 got 0
I20250901 01:52:30.939165 1151 consensus_queue.cc:260] T abf6e3be60f74872b2308dc95bf9cd37 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.939659 1151 ts_tablet_manager.cc:1428] T abf6e3be60f74872b2308dc95bf9cd37 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250901 01:52:30.939932 1151 tablet_bootstrap.cc:492] T cb7d7e1796c0456f8de98fb812b9e374 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:30.940213 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet d0c3c363dda444519a17dca63ad9caa7 (table default.Table_42422c44ca254d8ca989134c2433391f [id=8c1b263824ec4c15a5ec424364c78895]). Expected version 3 got 0
I20250901 01:52:30.940678 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 45f6e8ba99b74898a2922b36768c6c36 (table default.Table_938421db3cf74cf8b878089bac931818 [id=9ce26dcf307149b595a43b6ac4562fba]). Expected version 1 got 0
I20250901 01:52:30.946743 1151 tablet_bootstrap.cc:654] T cb7d7e1796c0456f8de98fb812b9e374 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:30.951692 2113 ts_tablet_manager.cc:1918] T d58f69fad01f403eab74a47d3695ed83 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:30.951910 2113 log.cc:1199] T d58f69fad01f403eab74a47d3695ed83 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/d58f69fad01f403eab74a47d3695ed83
I20250901 01:52:30.952296 2113 ts_tablet_manager.cc:1939] T d58f69fad01f403eab74a47d3695ed83 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:30.954766 1151 tablet_bootstrap.cc:492] T cb7d7e1796c0456f8de98fb812b9e374 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:30.954959 1151 ts_tablet_manager.cc:1397] T cb7d7e1796c0456f8de98fb812b9e374 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.001s
I20250901 01:52:30.955678 1151 raft_consensus.cc:357] T cb7d7e1796c0456f8de98fb812b9e374 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.955897 1151 raft_consensus.cc:383] T cb7d7e1796c0456f8de98fb812b9e374 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:30.956013 1151 raft_consensus.cc:738] T cb7d7e1796c0456f8de98fb812b9e374 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:30.956277 1151 consensus_queue.cc:260] T cb7d7e1796c0456f8de98fb812b9e374 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:30.957036 1151 ts_tablet_manager.cc:1428] T cb7d7e1796c0456f8de98fb812b9e374 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250901 01:52:30.957341 1151 tablet_bootstrap.cc:492] T d1f2d98712ff4ea6aec665512de068bd P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:30.959013 1151 tablet_bootstrap.cc:654] T d1f2d98712ff4ea6aec665512de068bd P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:30.960216 859 leader_election.cc:343] T b164b4a67a9143ceaff69b0ce49d63fe P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:30.960378 859 leader_election.cc:304] T b164b4a67a9143ceaff69b0ce49d63fe P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:30.961037 2608 raft_consensus.cc:2747] T b164b4a67a9143ceaff69b0ce49d63fe P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:30.961771 2582 tablet.cc:1722] T 14b466b2a5a249aa841f1b8b06301edc P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250901 01:52:30.968001 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 7954393c8c1c4457ab8bb723c03a9c8b (table default.Table_7050639ebd0147959ac4a251f534eb96 [id=543f2074598d4c29972e564be1123fca]). Expected version 5 got 0
I20250901 01:52:30.970266 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet e9f1f892658b4736859ff507536920a1 (table default.Table_8825dd28afbe464193eb1b668cc9326a [id=9aa62cb634314a0a849612c54c1220d9]). Expected version 1 got 0
I20250901 01:52:30.970647 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 44af4764d4f64e44aec84ccb84be6f5f (table default.Table_9896b980158948339a0b91f5882ed56b [id=a541b55be0dc420a8353ef4793b925b5]). Expected version 1 got 0
I20250901 01:52:30.971061 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 5f5396045979450193d461ba1334a5c9 (table default.Table_2b946a63bd574ac0971fa3d3b9c9a6a4 [id=9c3930e9fcfe4b49ba4bea67a2b0ad07]). Expected version 2 got 0
I20250901 01:52:30.971393 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 4cde5f4f02044b7bbd7d43b49aac4ae8 (table default.Table_d53a456dbca64daf92aeb5fb4517b417 [id=77ae3376281349e087a3b3cee41f2a09]). Expected version 1 got 0
I20250901 01:52:30.971925 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet aec951f2cc84499b9162b9fa24d5440e (table default.Table_156d1feafd5246ff8ff1bb7b9ae444a5 [id=55c31cd42fac43e6b8eddc048f565d0a]). Expected version 1 got 0
I20250901 01:52:30.972309 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 315c15cf0afc44d4a79cb29294281d60 (table default.Table_7aa28ad9ee8c49859af40f9ccb61af33 [id=aed8d4b4fa6a4fbf850cf5718610108f]). Expected version 1 got 0
I20250901 01:52:30.977907 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet b164b4a67a9143ceaff69b0ce49d63fe (table default.Table_da2c70a8f2fe49a0a8184e88c0a9e0ea [id=c1b1e9e3876244f68d31c63ce7bdebf1]). Expected version 1 got 0
I20250901 01:52:30.978302 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet ae21b9ae2e3b472eb67fb5f96388a50f (table default.Table_046289f393ff432aae0b033bdec1fb17 [id=db57b18a9ce24e068b362f5fbe940531]). Expected version 1 got 0
I20250901 01:52:30.978675 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 6c0abbfc8d904dce989ce51961e4237d (table default.Table_3f9422b08b0349b9a5466218f0fe1b06 [id=f0dfb3811e744bd4ba6d59195cebde6f]). Expected version 2 got 0
I20250901 01:52:30.979341 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 0e4009963cfe42d6ba164fc828897f09 (table default.Table_1adc70cccd2f4cb3a79df96861cb4181 [id=24fffc7ebf5a47bdb37211910301acf7]). Expected version 1 got 0
I20250901 01:52:30.979736 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 93bcac1fd95d44dc927abfca81692921 (table default.Table_b8f8ee74648c406eaf589d3ab80934db [id=b8f69313439543be9c11eea88dfa69c5]). Expected version 1 got 0
I20250901 01:52:30.980187 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 508406601c994e42a5c5b1f665e74957 (table default.Table_345b7f9fe8214aa6878f20a166125364 [id=4aabc328f6374ec5b1dea26f778208b2]). Expected version 1 got 0
I20250901 01:52:30.980688 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 5d68ea97c71942c7b55b967ba870e0d7 (table default.Table_f610cb75906d4c5cb3b57cfdb80ecefc [id=36915a3e843340ba8ef3578ac35a9af4]). Expected version 1 got 0
I20250901 01:52:30.985805 1151 tablet_bootstrap.cc:492] T d1f2d98712ff4ea6aec665512de068bd P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:30.986027 1151 ts_tablet_manager.cc:1397] T d1f2d98712ff4ea6aec665512de068bd P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.007s
I20250901 01:52:30.991307 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet d000ba05e12843b4925e622ab55471c0 (table default.Table_e0902dfbdd984f85b3fdac8d283731f7 [id=b43a92ed46164d199b445287ef039bc6]). Expected version 1 got 0
I20250901 01:52:30.991600 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 2973b59bb1df41d18d9a5b6a29ae336f (table default.Table_7790af96855e45c0a316ba3a4a87c9ed [id=746845bd84c6444b8a1e5683eaba4a68]). Expected version 1 got 0
I20250901 01:52:30.992373 1482 catalog_manager.cc:5582] T 615b2306fca84003b264da9d5f95ddab P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: UNKNOWN } } }
I20250901 01:52:30.992887 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 615b2306fca84003b264da9d5f95ddab (table default.Table_302f6e7cf97d46b9a621eb852e6aefe0 [id=cecdd8d6ca6d445c9fd82decd2a95366]). Expected version 2 got 0
I20250901 01:52:30.993292 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet c6f7c525f59b49938b697de98d4b5835 (table default.Table_db61e055dbd7450898e82c4c0c4763be [id=aabca67f6f754820977ac9c4c20f3e4f]). Expected version 1 got 0
I20250901 01:52:30.993875 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 61ddcf86c9f34726939cc116434f0822 (table default.Table_63d0f2ead9d946d39ce40162ebd769d6 [id=f9dcf4937b1143248bfa9d88053fce17]). Expected version 3 got 0
I20250901 01:52:30.994217 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 858218c4427c41e980b2b5c969a5af40 (table default.Table_6d47266af0ed433eb1f2abbc81d18e69 [id=264cc469a73841e79538843479a165ab]). Expected version 1 got 0
I20250901 01:52:30.994875 1482 catalog_manager.cc:5582] T a9b7f2e6548043a7865e25387cdd06c9 P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: UNKNOWN } } }
I20250901 01:52:30.995725 1150 tablet_bootstrap.cc:654] T 88ceba7f48b04d7885cc0164668700a5 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:30.996109 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 81ce448b148447ddbc41875d863e0518 (table default.Table_6e461d9cb0ae4dd095975bbc21533794 [id=49673fa73f8c436e9b8026efc4f3cbbc]). Expected version 1 got 0
I20250901 01:52:30.996867 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet c83ce95fd6fa41478de0e2168be201b1 (table default.Table_a525265208584713b1b3465d63130cd9 [id=bfd87b9b708449d3b16e3e0639561a49]). Expected version 1 got 0
I20250901 01:52:30.998226 1486 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_6295dbf840ff483a91edb67301e9b439" } modify_external_catalogs: true
I20250901 01:52:30.998457 1486 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_6295dbf840ff483a91edb67301e9b439" } modify_external_catalogs: true
I20250901 01:52:30.998561 1483 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet f8d9bbf0c5be462a8ddc539238cbdf97
W20250901 01:52:30.998765 1483 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet f8d9bbf0c5be462a8ddc539238cbdf97: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:30.999344 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2110d557e58646f99525d8416a05af5e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.000272 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_19d3f4432dab445880520bb4b454e84d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.000689 1488 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet e87d1281f35648b0b115b67c95e244e7
W20250901 01:52:31.000882 1488 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet e87d1281f35648b0b115b67c95e244e7: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:31.001071 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_79b88d9aaa014615b0faa8e1f087616a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.001464 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_0fff880401ac4f8fb03c676348050fc1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.001701 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7c73d3ace00f469ebacf727022e626d3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.002288 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_23d672395f024a3a9709ccaef080e01a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.002373 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e745c531cf414db6b5450a3556b8d91a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.002974 1488 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_2ceaff93c8f44f00aab53e48d8466dfd" } modify_external_catalogs: true
I20250901 01:52:31.003141 1488 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_2ceaff93c8f44f00aab53e48d8466dfd" } modify_external_catalogs: true
I20250901 01:52:31.005064 885 tablet_service.cc:1515] Processing DeleteTablet for tablet f8d9bbf0c5be462a8ddc539238cbdf97 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:30 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:31.005451 887 tablet_service.cc:1515] Processing DeleteTablet for tablet e87d1281f35648b0b115b67c95e244e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:30 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:31.005781 1038 tablet_service.cc:1515] Processing DeleteTablet for tablet f8d9bbf0c5be462a8ddc539238cbdf97 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:30 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:31.005978 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet e87d1281f35648b0b115b67c95e244e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:30 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:30.998466 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_02f591254f2f474d8a1e2506ec68f0be"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.006433 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d9c272a000024e85bcf0436a8edb8755"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.008247 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_23a210410a25477488d4e0cac55633a5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.010258 1489 master_service.cc:948] ReplaceTablet: received request to replace tablet d000ba05e12843b4925e622ab55471c0 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:31.018647 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_6ce94cac627f4d148ad03efd21a4a616"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.019871 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9ad26c6d3fd24b769c5e8d8140ebf201"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.022161 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3422aab85c394057bd6517497647fb73"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.022282 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e7534476fc224cc6bfd0aac8b8154a70"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.022954 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_44bc6aba6bcb4cd7b09c6de87a77bfbe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.023231 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_605e40bd8b184dfb8ccc0d75f60a7cd7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.023869 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b8876743a7284cb7922455f187d1f8b6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.023994 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9f9f5965f5964d769e39c4caafb9f163"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.024744 1483 master_service.cc:948] ReplaceTablet: received request to replace tablet 616ff34e8d314e0aa0e11ce60be11789 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:31.029646 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8cb3568a275547de92c2f2ed4608804d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.030586 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c19c8de2fa3f494aa593c3b311d2e166"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.031407 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2e43aa8e8b4242c79e65bf50315092ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.034394 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8462f447ac9d4134a4e2bf61534067d6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.037012 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8d641c725fc04e6e8b9c3a127b90692b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.038455 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1dc7bb5c62a140648ab2fb9d56c61adc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.043183 1151 raft_consensus.cc:357] T d1f2d98712ff4ea6aec665512de068bd P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:31.043393 1151 raft_consensus.cc:383] T d1f2d98712ff4ea6aec665512de068bd P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:31.039311 1491 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: default.Table_dd92add170f94aaf966f77d81e2c3883 [id=85b64046d2fa4f1d871ab43e5c800fb8] alter complete (version 1)
I20250901 01:52:31.043485 1151 raft_consensus.cc:738] T d1f2d98712ff4ea6aec665512de068bd P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:31.043762 1151 consensus_queue.cc:260] T d1f2d98712ff4ea6aec665512de068bd P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:31.043860 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 134) took 5781 ms. Trace:
I20250901 01:52:31.044045 2113 rpcz_store.cc:276] 0901 01:52:25.261959 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.263596 (+ 1637us) service_pool.cc:225] Handling call
0901 01:52:30.867494 (+5603898us) ts_tablet_manager.cc:1145] Deleting tablet d58f69fad01f403eab74a47d3695ed83
0901 01:52:30.868087 (+ 593us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:30.948677 (+ 80590us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:30.951576 (+ 2899us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:31.043807 (+ 92231us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":79852,"tablet-delete.queue_time_us":5603784}
W20250901 01:52:31.039733 1485 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dabebc77f72e4bb19e264be2f496202a consensus state has no leader
I20250901 01:52:31.044473 2113 tablet_replica.cc:331] T d77aadd7291647a0bae7c421f244c252 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:31.044687 2113 raft_consensus.cc:2241] T d77aadd7291647a0bae7c421f244c252 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:31.044828 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_db40df299e824a9a8c75781212072e67"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.044996 2113 raft_consensus.cc:2270] T d77aadd7291647a0bae7c421f244c252 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:31.045579 2113 ts_tablet_manager.cc:1905] T d77aadd7291647a0bae7c421f244c252 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:31.045684 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet d58f69fad01f403eab74a47d3695ed83 (table default.Table_95884295301d476fb3e03054727f9cf1 [id=1aa18a2513db4e99a72572846527663c]) successfully deleted
I20250901 01:52:31.045653 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f1410c8b264e4a61b3dbb42aa7918ed5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.046294 2412 raft_consensus.cc:491] T cb7d7e1796c0456f8de98fb812b9e374 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:31.046315 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f97f97747d1941fcb8f2aecb6bf750e4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.046437 2412 raft_consensus.cc:513] T cb7d7e1796c0456f8de98fb812b9e374 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:31.046984 2412 leader_election.cc:290] T cb7d7e1796c0456f8de98fb812b9e374 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:31.046959 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_6552a609a1e843e99c2e357a8587c2cb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.047574 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_59f88af6052446d497c9f806853c40d5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.048218 1151 ts_tablet_manager.cc:1428] T d1f2d98712ff4ea6aec665512de068bd P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.062s user 0.000s sys 0.001s
I20250901 01:52:31.048193 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_781fbd7401aa47a2bc48b974888cebb9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:31.043160 1490 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 615b2306fca84003b264da9d5f95ddab consensus state has no leader
I20250901 01:52:31.048535 1151 tablet_bootstrap.cc:492] T df01d5df9fc2424eaf9de4e68e781803 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:31.048745 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_af827b3996d54851a48e6558305a8fca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.049433 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_83eb4889c98b4368874d32284399badb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.050089 1490 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_3349df70e80e4caabfd33a2f303cf723" } modify_external_catalogs: true
I20250901 01:52:31.050253 1490 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_3349df70e80e4caabfd33a2f303cf723" } modify_external_catalogs: true
I20250901 01:52:31.050372 1151 tablet_bootstrap.cc:654] T df01d5df9fc2424eaf9de4e68e781803 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:31.057071 992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:40561, user_credentials={real_user=slave}} blocked reactor thread for 95083.9us
I20250901 01:52:31.067217 1151 tablet_bootstrap.cc:492] T df01d5df9fc2424eaf9de4e68e781803 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:31.067375 1151 ts_tablet_manager.cc:1397] T df01d5df9fc2424eaf9de4e68e781803 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.003s
I20250901 01:52:31.068091 1151 raft_consensus.cc:357] T df01d5df9fc2424eaf9de4e68e781803 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:31.068274 1151 raft_consensus.cc:383] T df01d5df9fc2424eaf9de4e68e781803 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:31.068346 1151 raft_consensus.cc:738] T df01d5df9fc2424eaf9de4e68e781803 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:31.068590 1151 consensus_queue.cc:260] T df01d5df9fc2424eaf9de4e68e781803 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:31.069314 1151 ts_tablet_manager.cc:1428] T df01d5df9fc2424eaf9de4e68e781803 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250901 01:52:31.069620 1151 tablet_bootstrap.cc:492] T 6ba88af177874544b764cc32420e7ac9 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:31.071220 1491 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: default.Table_0f4b4094d2974b7591875375eda6bed0 [id=5ee13d46e06a4fa59b1070547573e523] alter complete (version 2)
I20250901 01:52:31.071296 1151 tablet_bootstrap.cc:654] T 6ba88af177874544b764cc32420e7ac9 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:31.074275 2118 ts_tablet_manager.cc:1918] T 7442c6ff699141e1aff208ebdc9d192a P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 01:52:31.075598 2118 log.cc:1199] T 7442c6ff699141e1aff208ebdc9d192a P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/7442c6ff699141e1aff208ebdc9d192a
I20250901 01:52:31.076187 2118 ts_tablet_manager.cc:1939] T 7442c6ff699141e1aff208ebdc9d192a P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:31.076865 1490 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 787be682c0e44248a80d7037fb61dc6c
I20250901 01:52:31.077397 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_59a8331d860e456387a439e2792707f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.078996 1490 master_service.cc:948] ReplaceTablet: received request to replace tablet 5d794471dbed4477b9d52fd411fa4d10 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:31.080547 921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb7d7e1796c0456f8de98fb812b9e374" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:31.080734 921 raft_consensus.cc:2466] T cb7d7e1796c0456f8de98fb812b9e374 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:31.081599 2574 tablet.cc:1722] T 14b466b2a5a249aa841f1b8b06301edc P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250901 01:52:31.092290 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3497419e62f94d4cab1fcc1f05294673"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.093592 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_794f98b0f7144b859af8f3e2c722f639"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.094426 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4ebd1f2335664512bbcc2a62f95f7dcc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.095194 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_396208f6cbf94456a29643635314aa3f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.095504 1761 raft_consensus.cc:491] T b81a12b38ac64efabd44aeb192d181a0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:31.095706 1761 raft_consensus.cc:513] T b81a12b38ac64efabd44aeb192d181a0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:31.095986 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_53b9a2fb7ddc43a99b9a054ac4454754"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.096781 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_be79646aab9c40b6ac4872ee139c649a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.097543 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ef34494a6ac5449b97f9ee0a9ebeb3aa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.098239 2549 raft_consensus.cc:491] T d4c9aa723aee4dc4b6237cb5c3722b52 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:31.098382 2549 raft_consensus.cc:513] T d4c9aa723aee4dc4b6237cb5c3722b52 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:31.099002 2549 leader_election.cc:290] T d4c9aa723aee4dc4b6237cb5c3722b52 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:31.100809 858 leader_election.cc:336] T d4c9aa723aee4dc4b6237cb5c3722b52 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:31.108260 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_09f98a5a6e3446989a5520d0fb2ddbbc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.108968 2113 ts_tablet_manager.cc:1918] T d77aadd7291647a0bae7c421f244c252 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:31.109216 2113 log.cc:1199] T d77aadd7291647a0bae7c421f244c252 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/d77aadd7291647a0bae7c421f244c252
I20250901 01:52:31.109743 2113 ts_tablet_manager.cc:1939] T d77aadd7291647a0bae7c421f244c252 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:31.111277 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 135) took 5848 ms. Trace:
I20250901 01:52:31.111433 2113 rpcz_store.cc:276] 0901 01:52:25.262729 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.263752 (+ 1023us) service_pool.cc:225] Handling call
0901 01:52:31.044368 (+5780616us) ts_tablet_manager.cc:1145] Deleting tablet d77aadd7291647a0bae7c421f244c252
0901 01:52:31.044379 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:31.063890 (+ 19511us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:31.108781 (+ 44891us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:31.111232 (+ 2451us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5780492}
I20250901 01:52:31.112147 2113 tablet_replica.cc:331] T ad1a6be9fcc045d3bb4e71a2f6bd5ccd P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:31.112450 2113 raft_consensus.cc:2241] T ad1a6be9fcc045d3bb4e71a2f6bd5ccd P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:31.111920 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet d77aadd7291647a0bae7c421f244c252 (table default.Table_eacb38fbc98046dfac27344cfbed0a7c [id=015239c5e105476ca8bf6198560f0feb]) successfully deleted
I20250901 01:52:31.113305 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5955b83dbb244357983151179776277b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.104497 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 141) took 4232 ms. Trace:
I20250901 01:52:31.113940 2118 rpcz_store.cc:276] 0901 01:52:26.872397 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.872681 (+ 284us) service_pool.cc:225] Handling call
0901 01:52:30.885203 (+4012522us) ts_tablet_manager.cc:1145] Deleting tablet 7442c6ff699141e1aff208ebdc9d192a
0901 01:52:30.885215 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:30.908080 (+ 22865us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:31.074137 (+166057us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:31.104473 (+ 30336us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1920,"tablet-delete.queue_time_us":3993342}
I20250901 01:52:31.116143 2113 raft_consensus.cc:2270] T ad1a6be9fcc045d3bb4e71a2f6bd5ccd P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:31.116740 2113 ts_tablet_manager.cc:1905] T ad1a6be9fcc045d3bb4e71a2f6bd5ccd P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:31.117550 1761 leader_election.cc:290] T b81a12b38ac64efabd44aeb192d181a0 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:31.121474 2118 tablet_replica.cc:331] T 061b760d8e6647c6ab0b5d538840a230 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:31.121878 2118 raft_consensus.cc:2241] T 061b760d8e6647c6ab0b5d538840a230 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:31.122148 2118 raft_consensus.cc:2270] T 061b760d8e6647c6ab0b5d538840a230 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:31.128525 992 leader_election.cc:304] T cb7d7e1796c0456f8de98fb812b9e374 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:31.128921 2412 raft_consensus.cc:2802] T cb7d7e1796c0456f8de98fb812b9e374 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:31.129037 2412 raft_consensus.cc:491] T cb7d7e1796c0456f8de98fb812b9e374 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:31.129119 2412 raft_consensus.cc:3058] T cb7d7e1796c0456f8de98fb812b9e374 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
W20250901 01:52:31.129490 992 leader_election.cc:343] T b81a12b38ac64efabd44aeb192d181a0 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: b81a12b38ac64efabd44aeb192d181a0
I20250901 01:52:31.130478 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d4c9aa723aee4dc4b6237cb5c3722b52" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:31.124105 2549 raft_consensus.cc:491] T 75345ea2a3dc40fc8c9f8090d8839d2d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:31.130993 2549 raft_consensus.cc:513] T 75345ea2a3dc40fc8c9f8090d8839d2d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:31.131400 2118 ts_tablet_manager.cc:1905] T 061b760d8e6647c6ab0b5d538840a230 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:31.131644 2549 leader_election.cc:290] T 75345ea2a3dc40fc8c9f8090d8839d2d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:31.131637 2412 raft_consensus.cc:513] T cb7d7e1796c0456f8de98fb812b9e374 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:31.132161 2412 leader_election.cc:290] T cb7d7e1796c0456f8de98fb812b9e374 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:31.133504 858 leader_election.cc:336] T 75345ea2a3dc40fc8c9f8090d8839d2d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:31.134027 2549 raft_consensus.cc:491] T eb19e3a424324a24824f6caaf5848952 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:31.134171 2549 raft_consensus.cc:513] T eb19e3a424324a24824f6caaf5848952 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:31.134718 2549 leader_election.cc:290] T eb19e3a424324a24824f6caaf5848952 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:31.136763 858 leader_election.cc:336] T eb19e3a424324a24824f6caaf5848952 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:31.138012 992 consensus_peers.cc:489] T 061b760d8e6647c6ab0b5d538840a230 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20250901 01:52:31.138135 992 consensus_peers.cc:489] T d10c193452954c198bba686169b8025b P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250901 01:52:31.138222 992 leader_election.cc:336] T cb7d7e1796c0456f8de98fb812b9e374 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:31.144582 2113 ts_tablet_manager.cc:1918] T ad1a6be9fcc045d3bb4e71a2f6bd5ccd P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:31.144776 2113 log.cc:1199] T ad1a6be9fcc045d3bb4e71a2f6bd5ccd P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/ad1a6be9fcc045d3bb4e71a2f6bd5ccd
I20250901 01:52:31.145200 2113 ts_tablet_manager.cc:1939] T ad1a6be9fcc045d3bb4e71a2f6bd5ccd P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:31.146080 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 137) took 5883 ms. Trace:
I20250901 01:52:31.146222 2113 rpcz_store.cc:276] 0901 01:52:25.263005 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.264202 (+ 1197us) service_pool.cc:225] Handling call
0901 01:52:31.112074 (+5847872us) ts_tablet_manager.cc:1145] Deleting tablet ad1a6be9fcc045d3bb4e71a2f6bd5ccd
0901 01:52:31.112084 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:31.120316 (+ 8232us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:31.123211 (+ 2895us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:31.146036 (+ 22825us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5847688}
I20250901 01:52:31.146742 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 129) took 5885 ms. Trace:
W20250901 01:52:31.147706 859 leader_election.cc:343] T d4c9aa723aee4dc4b6237cb5c3722b52 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:31.147841 859 leader_election.cc:304] T d4c9aa723aee4dc4b6237cb5c3722b52 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:31.148034 2323 raft_consensus.cc:2747] T d4c9aa723aee4dc4b6237cb5c3722b52 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:31.148015 918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb7d7e1796c0456f8de98fb812b9e374" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:31.148180 918 raft_consensus.cc:3058] T cb7d7e1796c0456f8de98fb812b9e374 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:31.150817 918 raft_consensus.cc:2466] T cb7d7e1796c0456f8de98fb812b9e374 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:31.156479 2113 rpcz_store.cc:276] 0901 01:52:25.261374 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.264664 (+ 3290us) service_pool.cc:225] Handling call
0901 01:52:31.146599 (+5881935us) ts_tablet_manager.cc:1145] Deleting tablet becbaae4fd7e44faa679e9479465a7c4
0901 01:52:31.146610 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:31.146707 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5881744}
I20250901 01:52:31.157258 2113 tablet_replica.cc:331] T 23af154a6f1c4e2e9352fb17c9024f10 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:31.157568 2113 raft_consensus.cc:2241] T 23af154a6f1c4e2e9352fb17c9024f10 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:31.158322 2113 raft_consensus.cc:2270] T 23af154a6f1c4e2e9352fb17c9024f10 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:31.158324 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75345ea2a3dc40fc8c9f8090d8839d2d" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:31.158967 2113 ts_tablet_manager.cc:1905] T 23af154a6f1c4e2e9352fb17c9024f10 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:31.160022 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eb19e3a424324a24824f6caaf5848952" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:31.160517 992 consensus_peers.cc:489] T e1d18e999b5e42b1873631e0f1445081 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250901 01:52:31.160017 1151 tablet_bootstrap.cc:492] T 6ba88af177874544b764cc32420e7ac9 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:31.162755 1151 ts_tablet_manager.cc:1397] T 6ba88af177874544b764cc32420e7ac9 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.093s user 0.002s sys 0.004s
I20250901 01:52:31.163542 1151 raft_consensus.cc:357] T 6ba88af177874544b764cc32420e7ac9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:31.163779 1151 raft_consensus.cc:383] T 6ba88af177874544b764cc32420e7ac9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:31.163897 1151 raft_consensus.cc:738] T 6ba88af177874544b764cc32420e7ac9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:31.164158 1151 consensus_queue.cc:260] T 6ba88af177874544b764cc32420e7ac9 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:31.164916 1151 ts_tablet_manager.cc:1428] T 6ba88af177874544b764cc32420e7ac9 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250901 01:52:31.165259 1151 tablet_bootstrap.cc:492] T b164b4a67a9143ceaff69b0ce49d63fe P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:31.166922 1151 tablet_bootstrap.cc:654] T b164b4a67a9143ceaff69b0ce49d63fe P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:31.114694 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 7442c6ff699141e1aff208ebdc9d192a (table default.Table_4e2ea69d21394b9db0447f7ec8766795 [id=0bc0d862df03433dacaf6af87a8390cd]) successfully deleted
W20250901 01:52:31.169621 1472 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.2:40561, user_credentials={real_user=slave}} blocked reactor thread for 54969.4us
I20250901 01:52:31.170987 2596 catalog_manager.cc:1261] Loaded cluster ID: d2561e56f09e40169f2e331183eec812
W20250901 01:52:31.167201 992 leader_election.cc:336] T b81a12b38ac64efabd44aeb192d181a0 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:31.172240 992 leader_election.cc:304] T b81a12b38ac64efabd44aeb192d181a0 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:31.172479 2113 ts_tablet_manager.cc:1918] T 23af154a6f1c4e2e9352fb17c9024f10 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:31.172549 2412 raft_consensus.cc:2747] T b81a12b38ac64efabd44aeb192d181a0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:31.172892 2113 log.cc:1199] T 23af154a6f1c4e2e9352fb17c9024f10 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/23af154a6f1c4e2e9352fb17c9024f10
I20250901 01:52:31.173290 2113 ts_tablet_manager.cc:1939] T 23af154a6f1c4e2e9352fb17c9024f10 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:31.174041 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 127) took 5912 ms. Trace:
I20250901 01:52:31.174178 2113 rpcz_store.cc:276] 0901 01:52:25.261124 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.264470 (+ 3346us) service_pool.cc:225] Handling call
0901 01:52:31.157168 (+5892698us) ts_tablet_manager.cc:1145] Deleting tablet 23af154a6f1c4e2e9352fb17c9024f10
0901 01:52:31.157178 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:31.161885 (+ 4707us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:31.171887 (+ 10002us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:31.174004 (+ 2117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5892242}
W20250901 01:52:31.175346 859 leader_election.cc:343] T 75345ea2a3dc40fc8c9f8090d8839d2d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:31.175477 859 leader_election.cc:304] T 75345ea2a3dc40fc8c9f8090d8839d2d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:31.175902 859 leader_election.cc:343] T eb19e3a424324a24824f6caaf5848952 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:31.175999 859 leader_election.cc:304] T eb19e3a424324a24824f6caaf5848952 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:31.176743 2549 raft_consensus.cc:2747] T 75345ea2a3dc40fc8c9f8090d8839d2d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:31.176925 2549 raft_consensus.cc:2747] T eb19e3a424324a24824f6caaf5848952 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:31.179170 2113 tablet_replica.cc:331] T 7a2f6e2c7ee2402ea310c058e046416c P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:31.179320 2596 catalog_manager.cc:1554] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: loading cluster ID for follower catalog manager: success
I20250901 01:52:31.179766 2118 ts_tablet_manager.cc:1918] T 061b760d8e6647c6ab0b5d538840a230 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 01:52:31.179876 2118 log.cc:1199] T 061b760d8e6647c6ab0b5d538840a230 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/061b760d8e6647c6ab0b5d538840a230
I20250901 01:52:31.105855 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_721d18c8c8f14bc4ade109aa8241a73d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.140316 1486 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 2df2240631844defa433270a3829d65b
I20250901 01:52:31.146896 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet ad1a6be9fcc045d3bb4e71a2f6bd5ccd (table default.Table_5e84ae9a9c2a4084a1f9c5ffb4594de2 [id=9bebb2e486ab48888e6b523281ac1e88]) successfully deleted
W20250901 01:52:31.180194 1486 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 2df2240631844defa433270a3829d65b: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:31.180959 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_bd5211691ff4433e82a30fa1c6676d43"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:31.181548 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet becbaae4fd7e44faa679e9479465a7c4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet becbaae4fd7e44faa679e9479465a7c4 already in progress: creating tablet
I20250901 01:52:31.181810 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 23af154a6f1c4e2e9352fb17c9024f10 (table default.Table_54d5a8d2e22840d6a43c4eeb44478d1d [id=ccedb9a9dab94413b88eb7c203a9b2ee]) successfully deleted
I20250901 01:52:31.181916 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b3eaa9db31a14290a30cc7d2333775d0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.183101 1036 tablet_service.cc:1515] Processing DeleteTablet for tablet 2df2240631844defa433270a3829d65b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:30 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:31.182878 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_27a023c5716a4a97800ed7b9bb3d1076"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.183895 2113 raft_consensus.cc:2241] T 7a2f6e2c7ee2402ea310c058e046416c P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:31.184108 2113 raft_consensus.cc:2270] T 7a2f6e2c7ee2402ea310c058e046416c P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:31.184154 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b532caf565834c24b12a8f699ec29a54"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.184839 2113 ts_tablet_manager.cc:1905] T 7a2f6e2c7ee2402ea310c058e046416c P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:31.185286 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_93397b51c3fd4aa7a56d7367923bc288"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.186116 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 2df2240631844defa433270a3829d65b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:30 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:31.186825 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_89c217a7b761467ca2c9a8d7febb7a96"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.187578 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c97ec022bc6d42d795bd24d7e65e3250"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.188316 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_999852ba94de4c14bdf9028b1726766e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.190290 1151 tablet_bootstrap.cc:492] T b164b4a67a9143ceaff69b0ce49d63fe P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:31.190438 1151 ts_tablet_manager.cc:1397] T b164b4a67a9143ceaff69b0ce49d63fe P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.025s user 0.000s sys 0.003s
I20250901 01:52:31.191200 1151 raft_consensus.cc:357] T b164b4a67a9143ceaff69b0ce49d63fe P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:31.191377 1151 raft_consensus.cc:383] T b164b4a67a9143ceaff69b0ce49d63fe P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:31.191666 2113 ts_tablet_manager.cc:1918] T 7a2f6e2c7ee2402ea310c058e046416c P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:31.191772 2113 log.cc:1199] T 7a2f6e2c7ee2402ea310c058e046416c P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/7a2f6e2c7ee2402ea310c058e046416c
I20250901 01:52:31.192231 2113 ts_tablet_manager.cc:1939] T 7a2f6e2c7ee2402ea310c058e046416c P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:31.175376 992 leader_election.cc:304] T cb7d7e1796c0456f8de98fb812b9e374 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:31.192656 2412 raft_consensus.cc:2802] T cb7d7e1796c0456f8de98fb812b9e374 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:31.192809 2412 raft_consensus.cc:695] T cb7d7e1796c0456f8de98fb812b9e374 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:31.196308 1151 raft_consensus.cc:738] T b164b4a67a9143ceaff69b0ce49d63fe P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:31.196720 1151 consensus_queue.cc:260] T b164b4a67a9143ceaff69b0ce49d63fe P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:31.198388 1151 ts_tablet_manager.cc:1428] T b164b4a67a9143ceaff69b0ce49d63fe P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20250901 01:52:31.198668 1151 tablet_bootstrap.cc:492] T b13afacd37474781877b9f2cd20b313a P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:31.200191 1151 tablet_bootstrap.cc:654] T b13afacd37474781877b9f2cd20b313a P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:31.207376 1483 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 616ff34e8d314e0aa0e11ce60be11789
I20250901 01:52:31.207360 2412 consensus_queue.cc:237] T cb7d7e1796c0456f8de98fb812b9e374 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:31.207719 1483 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 616ff34e8d314e0aa0e11ce60be11789: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:31.208169 1483 catalog_manager.cc:6652] ReplaceTablet: tablet 616ff34e8d314e0aa0e11ce60be11789 deleted and replaced by tablet 93aa22afc0ad4d7fa24807ea3c5d5c21
I20250901 01:52:31.208382 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 616ff34e8d314e0aa0e11ce60be11789 with delete_type TABLET_DATA_DELETED (replaced by tablet 93aa22afc0ad4d7fa24807ea3c5d5c21) from {username='slave'} at 127.0.0.1:35946
W20250901 01:52:31.208642 992 leader_election.cc:336] T cb7d7e1796c0456f8de98fb812b9e374 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:31.209146 2574 tablet.cc:1722] T a8483e7212d94daba7e5401256c33aef P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:31.210981 2582 tablet.cc:1722] T a8483e7212d94daba7e5401256c33aef P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:31.211583 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 616ff34e8d314e0aa0e11ce60be11789 with delete_type TABLET_DATA_DELETED (replaced by tablet 93aa22afc0ad4d7fa24807ea3c5d5c21) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:31.215932 2323 raft_consensus.cc:491] T 34ebb441c27747ec860ff2e25f9a0adc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:31.216092 2323 raft_consensus.cc:513] T 34ebb441c27747ec860ff2e25f9a0adc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:31.216813 2323 leader_election.cc:290] T 34ebb441c27747ec860ff2e25f9a0adc P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:31.219302 858 leader_election.cc:336] T 34ebb441c27747ec860ff2e25f9a0adc P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:31.220960 1150 tablet_bootstrap.cc:492] T 88ceba7f48b04d7885cc0164668700a5 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:31.221182 1150 ts_tablet_manager.cc:1397] T 88ceba7f48b04d7885cc0164668700a5 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.283s user 0.003s sys 0.008s
W20250901 01:52:31.222050 992 consensus_peers.cc:489] T 616ff34e8d314e0aa0e11ce60be11789 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250901 01:52:31.222188 2118 ts_tablet_manager.cc:1939] T 061b760d8e6647c6ab0b5d538840a230 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:31.223333 1150 raft_consensus.cc:357] T 88ceba7f48b04d7885cc0164668700a5 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:31.224048 1150 raft_consensus.cc:383] T 88ceba7f48b04d7885cc0164668700a5 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:31.224181 1150 raft_consensus.cc:738] T 88ceba7f48b04d7885cc0164668700a5 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:31.224524 1150 consensus_queue.cc:260] T 88ceba7f48b04d7885cc0164668700a5 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:31.225029 1488 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet fcb7b5b7d63d4f0b906fea97cb548de2
W20250901 01:52:31.225252 1488 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet fcb7b5b7d63d4f0b906fea97cb548de2: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:31.225301 1150 ts_tablet_manager.cc:1428] T 88ceba7f48b04d7885cc0164668700a5 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250901 01:52:31.225940 1489 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet d000ba05e12843b4925e622ab55471c0
W20250901 01:52:31.226084 1489 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet d000ba05e12843b4925e622ab55471c0: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:31.226382 1489 catalog_manager.cc:6652] ReplaceTablet: tablet d000ba05e12843b4925e622ab55471c0 deleted and replaced by tablet eea6bfabe6cc4f83905c24f0f7a5dfc9
I20250901 01:52:31.226514 1150 tablet_bootstrap.cc:492] T eee34044c39d4013baf78d3705489626 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:31.227083 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "34ebb441c27747ec860ff2e25f9a0adc" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:31.228256 1038 tablet_service.cc:1515] Processing DeleteTablet for tablet d000ba05e12843b4925e622ab55471c0 with delete_type TABLET_DATA_DELETED (replaced by tablet eea6bfabe6cc4f83905c24f0f7a5dfc9) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:31.228470 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet fcb7b5b7d63d4f0b906fea97cb548de2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:31 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:31.228648 1150 tablet_bootstrap.cc:654] T eee34044c39d4013baf78d3705489626 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
E20250901 01:52:31.230834 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 0c88c339e59a4c1da21756cba0645bdd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ce86974fd8974de9a5a737954fac2ad7'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:31.230993 1696 catalog_manager.cc:6205] Error selecting replicas for tablet a656adbf1b29431687e0989f0db3459c, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2d3d2227e6a24bd2a86f457e553e698b'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:31.231117 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 494b6488aa0d4355b4caa46004267465, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_eecfab7239cc40c5a2f1fb6ac6d35dc9'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:31.231230 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 0ee3f35392954817b350e5f48653be78, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_442e11b9211c4b209c68f653d7bfc313'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:31.231359 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae4844183a3c4a04a775e6fd878391a3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_aee74198b51d4f089538c4f2bf3ce4dc'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:31.231436 1696 catalog_manager.cc:6205] Error selecting replicas for tablet be8dfbc466564842a2fdb4953eee939b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_75ba3ba70ca24a94920e7bce96190ff2'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:31.231525 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9745582cebfe47b58c21d0191ab4cb9b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77728af9eee44f29a1e2131d5b75b7ca'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:31.231628 1696 catalog_manager.cc:6205] Error selecting replicas for tablet e2ed32b3292141278af28026528e96c2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b5077f55951d48be92a67407ffe26a08'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:31.231755 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5a7fa68d29644ae8afd2711e39dde59a, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_f9e2a9182f6340bfa551fa2fc2eb2429'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:31.231875 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ff2bd017417c41b5b8225b8f00283e63, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_97526c9a6c9f4acbb6dedd78475571e0'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:31.232009 1696 catalog_manager.cc:6205] Error selecting replicas for tablet dcc60bb000f440f1aca9133928b5ce20, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2fbca8463d6546c9b8f5b018fc60251d'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:31.232106 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 182d99cb66024b66b16f1a4e76084260, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_dc856980ff4c4aafb92cf1fd4f29e694'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:31.232231 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 65e5cdc10e9642b7b0df775cb02861fa, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_1b5729d9768546fe99eaafc4e2604f42'. Need at least 3 replicas, but only 2 tablet servers are available
W20250901 01:52:31.227149 992 consensus_peers.cc:489] T 8ed6616ce54548ba83ba299c4ecf3c38 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Couldn't send request to peer 00c282f779c1477d920f24dc0dc16628. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ed6616ce54548ba83ba299c4ecf3c38. This is attempt 1: this message will repeat every 5th retry.
E20250901 01:52:31.232359 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f8f923ddcd2740bcb7e521f4f57c4034, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_5e65c5f068b84995a2d7a76fa65dda0f'. Need at least 3 replicas, but only 2 tablet servers are available
W20250901 01:52:31.232637 859 leader_election.cc:343] T 34ebb441c27747ec860ff2e25f9a0adc P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:31.232767 859 leader_election.cc:304] T 34ebb441c27747ec860ff2e25f9a0adc P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
E20250901 01:52:31.232810 1696 catalog_manager.cc:6205] Error selecting replicas for tablet e4d8330a4e454d1b8d643e17a6996741, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ce16a1b94f714c2699ee6cd916bb0215'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:31.232892 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 0813cc68d4b0488ba60a523e4479971b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d1302673d68f424ebb54b220911ce385'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:31.232988 1696 catalog_manager.cc:6205] Error selecting replicas for tablet b1ba0e19dc0b462e80c537bf2d183f5b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_66b111271c15410aab23682f8f6081ac'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:31.233072 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 8e10a01e77d14cf28c2c5e0205e595b9, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_83cf2d86a1284544ba2e7aa9ebb66bb7'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:31.233144 2323 raft_consensus.cc:2747] T 34ebb441c27747ec860ff2e25f9a0adc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
E20250901 01:52:31.233184 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 05a547a8866149cb95d23e960a737237, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_583e53c2e86544e9be68eb27ab6068a3'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:31.233898 887 tablet_service.cc:1515] Processing DeleteTablet for tablet fcb7b5b7d63d4f0b906fea97cb548de2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:31 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:31.234078 885 tablet_service.cc:1515] Processing DeleteTablet for tablet d000ba05e12843b4925e622ab55471c0 with delete_type TABLET_DATA_DELETED (replaced by tablet eea6bfabe6cc4f83905c24f0f7a5dfc9) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:31.223696 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 142) took 4255 ms. Trace:
I20250901 01:52:31.235513 2118 rpcz_store.cc:276] 0901 01:52:26.968255 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.976467 (+ 8212us) service_pool.cc:225] Handling call
0901 01:52:31.121420 (+4144953us) ts_tablet_manager.cc:1145] Deleting tablet 061b760d8e6647c6ab0b5d538840a230
0901 01:52:31.121429 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:31.161318 (+ 39889us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:31.179670 (+ 18352us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:31.223680 (+ 44010us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4130633}
I20250901 01:52:31.236382 2323 raft_consensus.cc:491] T f8d9bbf0c5be462a8ddc539238cbdf97 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:31.236554 2323 raft_consensus.cc:513] T f8d9bbf0c5be462a8ddc539238cbdf97 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:31.237205 2323 leader_election.cc:290] T f8d9bbf0c5be462a8ddc539238cbdf97 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:31.236230 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 143) took 4194 ms. Trace:
I20250901 01:52:31.237389 2118 rpcz_store.cc:276] 0901 01:52:27.041473 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:27.041547 (+ 74us) service_pool.cc:225] Handling call
0901 01:52:31.236117 (+4194570us) ts_tablet_manager.cc:1145] Deleting tablet 008d23f236c348b6be9aee718f8698af
0901 01:52:31.236127 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:31.236215 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4115207}
I20250901 01:52:31.237903 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f8d9bbf0c5be462a8ddc539238cbdf97" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:31.238407 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 130) took 5976 ms. Trace:
W20250901 01:52:31.238698 859 leader_election.cc:343] T f8d9bbf0c5be462a8ddc539238cbdf97 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:31.238683 2113 rpcz_store.cc:276] 0901 01:52:25.261494 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.264959 (+ 3465us) service_pool.cc:225] Handling call
0901 01:52:31.179081 (+5914122us) ts_tablet_manager.cc:1145] Deleting tablet 7a2f6e2c7ee2402ea310c058e046416c
0901 01:52:31.179091 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:31.188714 (+ 9623us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:31.191536 (+ 2822us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:31.238371 (+ 46835us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4502,"tablet-delete.queue_time_us":5913980}
I20250901 01:52:31.238808 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 144) took 4193 ms. Trace:
I20250901 01:52:31.238875 2118 rpcz_store.cc:276] 0901 01:52:27.045010 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:27.045065 (+ 55us) service_pool.cc:225] Handling call
0901 01:52:31.238698 (+4193633us) ts_tablet_manager.cc:1145] Deleting tablet 3dae00aa38204ee3ac77325eb334be42
0901 01:52:31.238708 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:31.238782 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4117718}
I20250901 01:52:31.239522 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 7a2f6e2c7ee2402ea310c058e046416c (table default.Table_2b946a63bd574ac0971fa3d3b9c9a6a4 [id=9c3930e9fcfe4b49ba4bea67a2b0ad07]) successfully deleted
I20250901 01:52:31.239836 2113 tablet_replica.cc:331] T 831e42de95854541a49ae6b3e039d3e9 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:31.240038 2113 raft_consensus.cc:2241] T 831e42de95854541a49ae6b3e039d3e9 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:31.240358 2113 raft_consensus.cc:2270] T 831e42de95854541a49ae6b3e039d3e9 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:31.240950 2113 ts_tablet_manager.cc:1905] T 831e42de95854541a49ae6b3e039d3e9 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:31.241231 2118 tablet_replica.cc:331] T 435b1cca5a7d4d49959a310cb250b2a0 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:31.244912 2118 raft_consensus.cc:2241] T 435b1cca5a7d4d49959a310cb250b2a0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250901 01:52:31.245182 858 leader_election.cc:336] T f8d9bbf0c5be462a8ddc539238cbdf97 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:31.246537 858 leader_election.cc:304] T f8d9bbf0c5be462a8ddc539238cbdf97 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:31.246866 2323 raft_consensus.cc:2747] T f8d9bbf0c5be462a8ddc539238cbdf97 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
E20250901 01:52:31.248715 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5f0e054f62de4cf6bc6f69ad6664b192, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ef6832d0845044ed9f165ca3541b4c3f'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:31.248900 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae9a4ad4fdfb4771ba23a91f7007447e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_5e84ae9a9c2a4084a1f9c5ffb4594de2'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:31.249027 1696 catalog_manager.cc:6205] Error selecting replicas for tablet d602c8986c75492ead9936ac04e2622d, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_82d91a1ba2eb4265896824028dec8d3a'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:31.249145 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9074543dea9949e7a085717b40913348, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_53e30205c5fd488eb3b48c685a159d04'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:31.249244 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 3a62034e688348ed9e2ac9de009ed3a2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_409dad98d7eb455a8e16edfb4acedb34'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:31.249361 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 18498c9cb09141d08c1c5c0854b63ebd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77f4810bc83c4d948531da2a45a3792e'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:31.249378 2574 tablet.cc:1722] T e1d18e999b5e42b1873631e0f1445081 P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
E20250901 01:52:31.249459 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f5e3316adeae47cd848833dd84b972bb, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7570c3012f8a4ae7bcfc7346eadac92a'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:31.249576 1696 catalog_manager.cc:6205] Error selecting replicas for tablet a6b539e160194da3a849d9e2f1ec9859, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_03fb48eaab9b42858ab7cefb722e43a8'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:31.249668 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 66719d8e9c614657820440ebe8ce7c1e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b3e694397be14321a668e46c09c51a71'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:31.249760 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 6ce9521b925b4f97bb4a810ce77e147e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_9e04f976a93f4d30baccf90fa3622dfe'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:31.249873 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2f61edb50d7a42198bfd70639c7bf4de, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_a5c783cbdb694af69047b31d99ff8def'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:31.250367 2582 tablet.cc:1722] T e1d18e999b5e42b1873631e0f1445081 P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250901 01:52:31.250820 2118 raft_consensus.cc:2270] T 435b1cca5a7d4d49959a310cb250b2a0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:31.251423 2118 ts_tablet_manager.cc:1905] T 435b1cca5a7d4d49959a310cb250b2a0 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:31.255928 1151 tablet_bootstrap.cc:492] T b13afacd37474781877b9f2cd20b313a P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:31.256072 1151 ts_tablet_manager.cc:1397] T b13afacd37474781877b9f2cd20b313a P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.057s user 0.000s sys 0.003s
I20250901 01:52:31.257462 1151 raft_consensus.cc:357] T b13afacd37474781877b9f2cd20b313a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:31.257654 1151 raft_consensus.cc:383] T b13afacd37474781877b9f2cd20b313a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:31.257987 1151 raft_consensus.cc:738] T b13afacd37474781877b9f2cd20b313a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:31.258288 1151 consensus_queue.cc:260] T b13afacd37474781877b9f2cd20b313a P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:31.258996 1151 ts_tablet_manager.cc:1428] T b13afacd37474781877b9f2cd20b313a P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250901 01:52:31.259254 1151 tablet_bootstrap.cc:492] T 06d5eeb70e664f73bf643922d445bc7a P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:31.268015 1036 tablet_service.cc:1515] Processing DeleteTablet for tablet becbaae4fd7e44faa679e9479465a7c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:23 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:31.268440 1036 tablet_service.cc:1515] Processing DeleteTablet for tablet bb0cf5a6a27045a08df6845d7473184d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:31.268678 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 154574e17f214cc0ad56e6e6be3af899 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:13 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:31.268863 1038 tablet_service.cc:1515] Processing DeleteTablet for tablet 9d886fa63f8f4996b087997b542e8bb1 with delete_type TABLET_DATA_DELETED (replaced by tablet eaa521c52a2f4408b59770bd2a05907c) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:31.269023 1033 tablet_service.cc:1515] Processing DeleteTablet for tablet 23af154a6f1c4e2e9352fb17c9024f10 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:15 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:31.269459 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet 7a2f6e2c7ee2402ea310c058e046416c with delete_type TABLET_DATA_DELETED (replaced by tablet 5f5396045979450193d461ba1334a5c9) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:31.269683 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 26c8ed083cea4dc0acca80d00b11b29b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:23 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:31.269474 2113 ts_tablet_manager.cc:1918] T 831e42de95854541a49ae6b3e039d3e9 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:31.269914 1035 tablet_service.cc:1515] Processing DeleteTablet for tablet 98cca86164db4252b1b49de88747822a with delete_type TABLET_DATA_DELETED (replaced by tablet fa7fb88ded21432a8757f37ec3db5e71) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:31.270082 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet d58f69fad01f403eab74a47d3695ed83 with delete_type TABLET_DATA_DELETED (replaced by tablet 74b709cb5ec64006b2f12bead8217d80) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:31.270213 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet d77aadd7291647a0bae7c421f244c252 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:14 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:31.270525 2118 ts_tablet_manager.cc:1918] T 435b1cca5a7d4d49959a310cb250b2a0 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:31.270638 2118 log.cc:1199] T 435b1cca5a7d4d49959a310cb250b2a0 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/435b1cca5a7d4d49959a310cb250b2a0
I20250901 01:52:31.269896 2113 log.cc:1199] T 831e42de95854541a49ae6b3e039d3e9 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/831e42de95854541a49ae6b3e039d3e9
I20250901 01:52:31.271001 2118 ts_tablet_manager.cc:1939] T 435b1cca5a7d4d49959a310cb250b2a0 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:31.271139 2113 ts_tablet_manager.cc:1939] T 831e42de95854541a49ae6b3e039d3e9 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:31.271294 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 145) took 4196 ms. Trace:
I20250901 01:52:31.271364 2118 rpcz_store.cc:276] 0901 01:52:27.075192 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:27.075244 (+ 52us) service_pool.cc:225] Handling call
0901 01:52:31.241177 (+4165933us) ts_tablet_manager.cc:1145] Deleting tablet 435b1cca5a7d4d49959a310cb250b2a0
0901 01:52:31.241190 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:31.260904 (+ 19714us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:31.270426 (+ 9522us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:31.271278 (+ 852us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3709,"tablet-delete.queue_time_us":4116454}
I20250901 01:52:31.271826 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 128) took 6010 ms. Trace:
I20250901 01:52:31.271900 2113 rpcz_store.cc:276] 0901 01:52:25.261258 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.270767 (+ 9509us) service_pool.cc:225] Handling call
0901 01:52:31.239789 (+5969022us) ts_tablet_manager.cc:1145] Deleting tablet 831e42de95854541a49ae6b3e039d3e9
0901 01:52:31.239800 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:31.259089 (+ 19289us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:31.269377 (+ 10288us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:31.271808 (+ 2431us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5961474}
I20250901 01:52:31.272516 2118 tablet_replica.cc:331] T 641a83a3e50d451189c33c7b7b4ca070 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:31.273023 2118 raft_consensus.cc:2241] T 641a83a3e50d451189c33c7b7b4ca070 P 00c282f779c1477d920f24dc0dc16628 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:31.279210 1483 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_e67df63f08fb415293f39cad748a0bd5" } new_table_name: "default.Table_f91f5f37b4fb40bea1190b3eb31f45d1" modify_external_catalogs: true
I20250901 01:52:31.282573 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e8759e9f006f4ad094b7419f87091740"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.283463 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_41fab2f2fed84991b2173afb94b4fc21"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.286140 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 831e42de95854541a49ae6b3e039d3e9 (table default.Table_113664a018e84769a7ff61aa35a38618 [id=68214167814a4ecbab3ec100e83f4a64]) successfully deleted
I20250901 01:52:31.286278 2113 tablet_replica.cc:331] T 6c05eff658924ca4a9a1e7a37f917d67 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:31.286481 2113 raft_consensus.cc:2241] T 6c05eff658924ca4a9a1e7a37f917d67 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:31.283360 1151 tablet_bootstrap.cc:654] T 06d5eeb70e664f73bf643922d445bc7a P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:31.288300 2666 consensus_queue.cc:1035] T a9b7f2e6548043a7865e25387cdd06c9 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:31.289134 2113 raft_consensus.cc:2270] T 6c05eff658924ca4a9a1e7a37f917d67 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:31.289605 2596 catalog_manager.cc:1576] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: acquiring CA information for follower catalog manager: success
W20250901 01:52:31.291420 1483 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 669d26c6551c4fc58283b67a501d9bf1 consensus state has no leader
I20250901 01:52:31.291764 2113 ts_tablet_manager.cc:1905] T 6c05eff658924ca4a9a1e7a37f917d67 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:31.295869 969 heartbeater.cc:499] Master 127.0.104.60:44231 was elected leader, sending a full tablet report...
I20250901 01:52:31.296458 2549 raft_consensus.cc:491] T 50472a7aa2f04e0fbd65ac6c86c34b3e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:31.296592 2549 raft_consensus.cc:513] T 50472a7aa2f04e0fbd65ac6c86c34b3e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:31.297852 2118 raft_consensus.cc:2270] T 641a83a3e50d451189c33c7b7b4ca070 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:31.298420 2118 ts_tablet_manager.cc:1905] T 641a83a3e50d451189c33c7b7b4ca070 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:31.300688 858 consensus_peers.cc:489] T 24ddc13d1bb04db4a7850acbff091577 P 00c282f779c1477d920f24dc0dc16628 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20250901 01:52:31.304527 1483 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_5facf725e0fc44cda7d77f8511030fa5" } modify_external_catalogs: true
I20250901 01:52:31.304716 1483 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_5facf725e0fc44cda7d77f8511030fa5" } modify_external_catalogs: true
W20250901 01:52:31.306625 992 consensus_peers.cc:489] T 615b2306fca84003b264da9d5f95ddab P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:31.306883 2666 consensus_queue.cc:1035] T ed1b41340ce742a88c3e4fd85cd25ca6 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:31.307467 1151 tablet_bootstrap.cc:492] T 06d5eeb70e664f73bf643922d445bc7a P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:31.307643 1151 ts_tablet_manager.cc:1397] T 06d5eeb70e664f73bf643922d445bc7a P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.048s user 0.004s sys 0.001s
I20250901 01:52:31.308297 1151 raft_consensus.cc:357] T 06d5eeb70e664f73bf643922d445bc7a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:31.309271 1151 raft_consensus.cc:383] T 06d5eeb70e664f73bf643922d445bc7a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:31.309376 1151 raft_consensus.cc:738] T 06d5eeb70e664f73bf643922d445bc7a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:31.309649 1151 consensus_queue.cc:260] T 06d5eeb70e664f73bf643922d445bc7a P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:31.310295 1151 ts_tablet_manager.cc:1428] T 06d5eeb70e664f73bf643922d445bc7a P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250901 01:52:31.312076 2113 ts_tablet_manager.cc:1918] T 6c05eff658924ca4a9a1e7a37f917d67 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:31.315601 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e008e48345d84a50bcc5405bdec77d81"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.315707 2357 raft_consensus.cc:491] T 88ceba7f48b04d7885cc0164668700a5 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:31.315829 2357 raft_consensus.cc:513] T 88ceba7f48b04d7885cc0164668700a5 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:31.316376 2357 leader_election.cc:290] T 88ceba7f48b04d7885cc0164668700a5 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:31.316650 1488 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_ce86974fd8974de9a5a737954fac2ad7" } new_table_name: "default.Table_d25639c7008f4e29af6226039e4b976b" modify_external_catalogs: true
I20250901 01:52:31.317090 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_04b833466e534b369d0594794dcbf6ee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.317395 1489 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_985c3c59b6084cd596b5478e83ddb7b9" } modify_external_catalogs: true
I20250901 01:52:31.318531 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ad3cd59ab39c48478fb13d6e087d1d63"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.319671 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2a4f8bdf99c1412f98f38128ec2f3a5a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.318923 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "88ceba7f48b04d7885cc0164668700a5" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:31.320274 992 consensus_peers.cc:489] T f4fd117c395e41c2ab679c684908c5b7 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250901 01:52:31.320575 859 leader_election.cc:343] T 88ceba7f48b04d7885cc0164668700a5 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:31.320860 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_355427cd143143d3a273e4e21df7b4dd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:31.322604 858 leader_election.cc:336] T 88ceba7f48b04d7885cc0164668700a5 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:31.322705 858 leader_election.cc:304] T 88ceba7f48b04d7885cc0164668700a5 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:31.322906 992 consensus_peers.cc:489] T 061b760d8e6647c6ab0b5d538840a230 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Couldn't send request to peer 00c282f779c1477d920f24dc0dc16628. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 061b760d8e6647c6ab0b5d538840a230. This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:31.323320 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 061b760d8e6647c6ab0b5d538840a230 (table default.Table_fbba493de02249499d26467bf15c3d5f [id=748daa63137c48a5b151bd34dc5b13fe]) successfully deleted
W20250901 01:52:31.323550 1472 catalog_manager.cc:4908] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 008d23f236c348b6be9aee718f8698af because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 008d23f236c348b6be9aee718f8698af already in progress: creating tablet
I20250901 01:52:31.323523 2113 log.cc:1199] T 6c05eff658924ca4a9a1e7a37f917d67 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/6c05eff658924ca4a9a1e7a37f917d67
I20250901 01:52:31.323837 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 435b1cca5a7d4d49959a310cb250b2a0 (table default.Table_b989608c88204be2b6f706a9426c47bb [id=126b298fef6c4653952d672adff1db7e]) successfully deleted
I20250901 01:52:31.323863 2357 raft_consensus.cc:2747] T 88ceba7f48b04d7885cc0164668700a5 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:31.322314 1151 tablet_bootstrap.cc:492] T 74b709cb5ec64006b2f12bead8217d80 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:31.326700 2113 ts_tablet_manager.cc:1939] T 6c05eff658924ca4a9a1e7a37f917d67 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:31.330235 2357 raft_consensus.cc:491] T 75a703c8e20a4a2fa0ab6ec58d41ef92 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:31.330370 2357 raft_consensus.cc:513] T 75a703c8e20a4a2fa0ab6ec58d41ef92 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:31.330977 2357 leader_election.cc:290] T 75a703c8e20a4a2fa0ab6ec58d41ef92 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:31.333048 2357 raft_consensus.cc:491] T b85d64e02cc646e380581ac9f59dee9e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:31.333184 2357 raft_consensus.cc:513] T b85d64e02cc646e380581ac9f59dee9e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:31.333292 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75a703c8e20a4a2fa0ab6ec58d41ef92" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:31.333681 2357 leader_election.cc:290] T b85d64e02cc646e380581ac9f59dee9e P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:31.334360 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b85d64e02cc646e380581ac9f59dee9e" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:31.340026 1151 tablet_bootstrap.cc:654] T 74b709cb5ec64006b2f12bead8217d80 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:31.340775 1491 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: default.Table_4dacb3b1d2e94e42ac8937eb6eb58375 [id=8ee0aa1ba24042dd8ded004644913b9c] alter complete (version 1)
I20250901 01:52:31.344973 1483 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 3d90c6ef06d1436d8e2bb1333e7d9985
W20250901 01:52:31.345382 1483 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 3d90c6ef06d1436d8e2bb1333e7d9985: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:31.346169 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 3d90c6ef06d1436d8e2bb1333e7d9985 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:31 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:31.347954 1489 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_985c3c59b6084cd596b5478e83ddb7b9" } modify_external_catalogs: true
W20250901 01:52:31.353435 1488 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0c88c339e59a4c1da21756cba0645bdd has no consensus state
I20250901 01:52:31.341995 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 138) took 5643 ms. Trace:
I20250901 01:52:31.355209 2113 rpcz_store.cc:276] 0901 01:52:25.698578 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.700629 (+ 2051us) service_pool.cc:225] Handling call
0901 01:52:31.286224 (+5585595us) ts_tablet_manager.cc:1145] Deleting tablet 6c05eff658924ca4a9a1e7a37f917d67
0901 01:52:31.286236 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:31.294911 (+ 8675us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:31.311977 (+ 17066us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:31.341724 (+ 29747us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5585387}
I20250901 01:52:31.355701 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 6c05eff658924ca4a9a1e7a37f917d67 (table default.Table_c1959bda12f14bc9995d832476abf670 [id=17355a61629849dd9d8228a469ab744f]) successfully deleted
W20250901 01:52:31.357666 992 consensus_peers.cc:489] T 42c1156e8dd84c6fa1396b46de13eb6f P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250901 01:52:31.360642 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ffee40d7286b498e8879fddb788dd64a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.361546 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 139) took 5662 ms. Trace:
I20250901 01:52:31.361608 922 raft_consensus.cc:1273] T 615b2306fca84003b264da9d5f95ddab P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Refusing update from remote peer fd27ab6665304aaea53adbd102dd4f58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250901 01:52:31.361652 2113 rpcz_store.cc:276] 0901 01:52:25.698680 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.700865 (+ 2185us) service_pool.cc:225] Handling call
0901 01:52:31.357600 (+5656735us) ts_tablet_manager.cc:1145] Deleting tablet eee34044c39d4013baf78d3705489626
0901 01:52:31.361415 (+ 3815us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:31.361524 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5656621}
I20250901 01:52:31.362186 2666 consensus_queue.cc:1035] T 615b2306fca84003b264da9d5f95ddab P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:31.364158 2666 consensus_queue.cc:1035] T 33f637f6a8134215bb1995ec38440073 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250901 01:52:31.365078 1492 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.3:51295 (request call id 184) took 1822 ms (client timeout 1000 ms). Trace:
I20250901 01:52:31.365217 2113 tablet_replica.cc:331] T 7cc1efcb6bff4aba8015d2d3b7f4c923 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
W20250901 01:52:31.365191 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet eee34044c39d4013baf78d3705489626 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eee34044c39d4013baf78d3705489626 already in progress: creating tablet
I20250901 01:52:31.365428 2113 raft_consensus.cc:2241] T 7cc1efcb6bff4aba8015d2d3b7f4c923 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:31.365852 2113 raft_consensus.cc:2270] T 7cc1efcb6bff4aba8015d2d3b7f4c923 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:31.366046 2549 leader_election.cc:290] T 50472a7aa2f04e0fbd65ac6c86c34b3e P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:31.366546 2113 ts_tablet_manager.cc:1905] T 7cc1efcb6bff4aba8015d2d3b7f4c923 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:31.367440 2582 tablet.cc:1722] T 24ddc13d1bb04db4a7850acbff091577 P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
W20250901 01:52:31.334079 859 leader_election.cc:343] T 75a703c8e20a4a2fa0ab6ec58d41ef92 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
W20250901 01:52:31.372612 858 leader_election.cc:336] T 75a703c8e20a4a2fa0ab6ec58d41ef92 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:31.372809 859 leader_election.cc:343] T b85d64e02cc646e380581ac9f59dee9e P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:31.372980 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50472a7aa2f04e0fbd65ac6c86c34b3e" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:31.373912 859 leader_election.cc:343] T 50472a7aa2f04e0fbd65ac6c86c34b3e P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
W20250901 01:52:31.376901 992 consensus_peers.cc:489] T ed1b41340ce742a88c3e4fd85cd25ca6 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:31.389199 858 leader_election.cc:304] T 75a703c8e20a4a2fa0ab6ec58d41ef92 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:31.389639 858 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:40731, user_credentials={real_user=slave}} blocked reactor thread for 55404.4us
I20250901 01:52:31.390957 2549 raft_consensus.cc:2747] T 75a703c8e20a4a2fa0ab6ec58d41ef92 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:31.392658 858 leader_election.cc:336] T b85d64e02cc646e380581ac9f59dee9e P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:31.392840 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 3d90c6ef06d1436d8e2bb1333e7d9985 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:31 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:31.392764 858 leader_election.cc:304] T b85d64e02cc646e380581ac9f59dee9e P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:31.395099 2549 raft_consensus.cc:2747] T b85d64e02cc646e380581ac9f59dee9e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:31.395722 858 leader_election.cc:336] T 50472a7aa2f04e0fbd65ac6c86c34b3e P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:31.395829 858 leader_election.cc:304] T 50472a7aa2f04e0fbd65ac6c86c34b3e P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:31.396294 2549 raft_consensus.cc:2747] T 50472a7aa2f04e0fbd65ac6c86c34b3e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:31.398059 2549 consensus_queue.cc:1035] T 6c0abbfc8d904dce989ce51961e4237d P 00c282f779c1477d920f24dc0dc16628 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:31.417941 2574 tablet.cc:1722] T 24ddc13d1bb04db4a7850acbff091577 P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
W20250901 01:52:31.423221 992 consensus_peers.cc:489] T a9b7f2e6548043a7865e25387cdd06c9 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250901 01:52:31.365196 1492 rpcz_store.cc:269] 0901 01:52:29.542602 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:29.542653 (+ 51us) service_pool.cc:225] Handling call
0901 01:52:29.589179 (+ 46526us) spinlock_profiling.cc:243] Waited 14.2 ms on lock 0x616000040c54. stack: 00007f0d64ec57d6 00007f0d64ec5590 00007f0d63eb83ad 0000000000972d94 0000000000972d5d 00007f0d73c9014d 00007f0d73c90243 00007f0d7094f941 00007f0d784daa5d 00007f0d7865e6d8 00007f0d6f498c86 00007f0d6f4989b5 00007f0d6bc732ce 00007f0d6bc721ae 00007f0d6bc783e5 00007f0d6bc78d0f
0901 01:52:31.365010 (+1775831us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0901 01:52:30.238159 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:30.238254 (+ 95us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:30.238259 (+ 5us) write_op.cc:435] Acquired schema lock
0901 01:52:30.238262 (+ 3us) tablet.cc:660] Decoding operations
0901 01:52:30.238403 (+ 141us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:30.238412 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:30.238417 (+ 5us) tablet.cc:683] Acquiring locks for 4 operations
0901 01:52:30.238519 (+ 102us) tablet.cc:699] Row locks acquired
0901 01:52:30.238521 (+ 2us) write_op.cc:260] PREPARE: finished
0901 01:52:30.238554 (+ 33us) write_op.cc:270] Start()
0901 01:52:30.238581 (+ 27us) write_op.cc:276] Timestamp: P: 1756691550238526 usec, L: 0
0901 01:52:30.238584 (+ 3us) op_driver.cc:348] REPLICATION: starting
0901 01:52:30.238800 (+ 216us) log.cc:844] Serialized 1650 byte log entry
0901 01:52:30.247831 (+ 9031us) op_driver.cc:464] REPLICATION: finished
0901 01:52:30.248036 (+ 205us) write_op.cc:301] APPLY: starting
0901 01:52:30.248047 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:30.248113 (+ 66us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:30.248138 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:30.248205 (+ 67us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:30.248233 (+ 28us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0901 01:52:30.248239 (+ 6us) write_op.cc:312] APPLY: finished
0901 01:52:30.248325 (+ 86us) log.cc:844] Serialized 41 byte log entry
0901 01:52:30.248369 (+ 44us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:30.248384 (+ 15us) write_op.cc:454] Released schema lock
0901 01:52:30.248389 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0901 01:52:30.248471 (+ 82us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:30.248474 (+ 3us) write_op.cc:454] Released schema lock
Related trace 'op':
0901 01:52:30.320001 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:30.320081 (+ 80us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:30.320087 (+ 6us) write_op.cc:435] Acquired schema lock
0901 01:52:30.320089 (+ 2us) tablet.cc:660] Decoding operations
0901 01:52:30.320180 (+ 91us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:30.320193 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:30.320201 (+ 8us) tablet.cc:683] Acquiring locks for 1 operations
0901 01:52:30.320267 (+ 66us) tablet.cc:699] Row locks acquired
0901 01:52:30.320269 (+ 2us) write_op.cc:260] PREPARE: finished
0901 01:52:30.320295 (+ 26us) write_op.cc:270] Start()
0901 01:52:30.320320 (+ 25us) write_op.cc:276] Timestamp: P: 1756691550320261 usec, L: 0
0901 01:52:30.320323 (+ 3us) op_driver.cc:348] REPLICATION: starting
0901 01:52:30.406544 (+ 86221us) log.cc:844] Serialized 356 byte log entry
0901 01:52:30.406728 (+ 184us) spinlock_profiling.cc:243] Waited 86 ms on lock 0x616000040c54. stack: 00007f0d64ec57d6 00007f0d64ec5590 00007f0d63eb83ad 0000000000972d94 0000000000972d5d 000000000097272a 00007f0d70921d62 00007f0d71cca2d5 00007f0d71cc8df9 00007f0d71cd1220 00007f0d71cd102d 0000000000a774dc 00007f0d64f0bb5f 00007f0d64f0ee2b 00007f0d64f0eca8 0000000000a774dc
0901 01:52:30.460563 (+ 53835us) op_driver.cc:464] REPLICATION: finished
0901 01:52:30.460652 (+ 89us) write_op.cc:301] APPLY: starting
0901 01:52:30.460669 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:30.460864 (+ 195us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:30.460867 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:30.460917 (+ 50us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:30.460956 (+ 39us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0901 01:52:30.460964 (+ 8us) write_op.cc:312] APPLY: finished
0901 01:52:30.461033 (+ 69us) log.cc:844] Serialized 23 byte log entry
0901 01:52:30.461087 (+ 54us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:30.461102 (+ 15us) write_op.cc:454] Released schema lock
0901 01:52:30.461108 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":201,"spinlock_wait_cycles":32763136,"child_traces":[["op",{"apply.queue_time_us":164,"apply.run_cpu_time_us":430,"apply.run_wall_time_us":430,"num_ops":4,"prepare.queue_time_us":7189,"prepare.run_cpu_time_us":709,"prepare.run_wall_time_us":708,"replication_time_us":9223}],["op",{"apply.queue_time_us":37,"apply.run_cpu_time_us":380,"apply.run_wall_time_us":519,"num_ops":1,"prepare.queue_time_us":54,"prepare.run_cpu_time_us":1849,"prepare.run_wall_time_us":86789,"raft.queue_time_us":5056,"raft.run_cpu_time_us":301,"raft.run_wall_time_us":301,"replication_time_us":140206,"spinlock_wait_cycles":197825920}]]}
I20250901 01:52:31.430631 2409 raft_consensus.cc:491] T 5f52698fb56d472d9e91fb10cb10c69a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:31.430773 2409 raft_consensus.cc:513] T 5f52698fb56d472d9e91fb10cb10c69a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:31.431269 2409 leader_election.cc:290] T 5f52698fb56d472d9e91fb10cb10c69a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:31.431715 2409 raft_consensus.cc:491] T df01d5df9fc2424eaf9de4e68e781803 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:31.431833 2409 raft_consensus.cc:513] T df01d5df9fc2424eaf9de4e68e781803 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:31.432276 2409 leader_election.cc:290] T df01d5df9fc2424eaf9de4e68e781803 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:31.432921 918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f52698fb56d472d9e91fb10cb10c69a" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:31.433107 918 raft_consensus.cc:2466] T 5f52698fb56d472d9e91fb10cb10c69a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:31.436631 992 leader_election.cc:304] T 5f52698fb56d472d9e91fb10cb10c69a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:31.437359 2409 raft_consensus.cc:2802] T 5f52698fb56d472d9e91fb10cb10c69a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:31.437433 2409 raft_consensus.cc:491] T 5f52698fb56d472d9e91fb10cb10c69a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:31.437490 2409 raft_consensus.cc:3058] T 5f52698fb56d472d9e91fb10cb10c69a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:31.438376 918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "df01d5df9fc2424eaf9de4e68e781803" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:31.438578 918 raft_consensus.cc:2466] T df01d5df9fc2424eaf9de4e68e781803 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:31.440091 2409 raft_consensus.cc:513] T 5f52698fb56d472d9e91fb10cb10c69a P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:31.440930 2409 leader_election.cc:290] T 5f52698fb56d472d9e91fb10cb10c69a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:31.441174 992 consensus_peers.cc:489] T 33f637f6a8134215bb1995ec38440073 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250901 01:52:31.441774 992 leader_election.cc:336] T 5f52698fb56d472d9e91fb10cb10c69a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:31.443497 918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f52698fb56d472d9e91fb10cb10c69a" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:31.443733 918 raft_consensus.cc:3058] T 5f52698fb56d472d9e91fb10cb10c69a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:31.443792 992 leader_election.cc:304] T df01d5df9fc2424eaf9de4e68e781803 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:31.446264 2666 raft_consensus.cc:2802] T df01d5df9fc2424eaf9de4e68e781803 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:31.446343 2666 raft_consensus.cc:491] T df01d5df9fc2424eaf9de4e68e781803 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:31.446411 2666 raft_consensus.cc:3058] T df01d5df9fc2424eaf9de4e68e781803 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:31.447614 918 raft_consensus.cc:2466] T 5f52698fb56d472d9e91fb10cb10c69a P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:31.448933 2666 raft_consensus.cc:513] T df01d5df9fc2424eaf9de4e68e781803 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:31.449443 2666 leader_election.cc:290] T df01d5df9fc2424eaf9de4e68e781803 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:31.451207 992 leader_election.cc:336] T df01d5df9fc2424eaf9de4e68e781803 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:31.453022 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_babc3f5fb3b741f0ae7fbd4ef086bf7e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.453436 918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "df01d5df9fc2424eaf9de4e68e781803" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:31.453478 992 leader_election.cc:304] T 5f52698fb56d472d9e91fb10cb10c69a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:31.453598 918 raft_consensus.cc:3058] T df01d5df9fc2424eaf9de4e68e781803 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:31.453713 2518 raft_consensus.cc:2802] T 5f52698fb56d472d9e91fb10cb10c69a P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
W20250901 01:52:31.453751 992 leader_election.cc:336] T 5f52698fb56d472d9e91fb10cb10c69a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:31.453852 2518 raft_consensus.cc:695] T 5f52698fb56d472d9e91fb10cb10c69a P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:31.454123 2518 consensus_queue.cc:237] T 5f52698fb56d472d9e91fb10cb10c69a P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:31.454470 1488 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_bfb1749ab3454806b139ff43f672583a" } new_table_name: "default.Table_f35bb824217f41ad8b0007945eaac866" modify_external_catalogs: true
I20250901 01:52:31.454972 1483 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_7050639ebd0147959ac4a251f534eb96" } new_table_name: "default.Table_0cb2185875b646dcac4d1f6f2f4cffd5" modify_external_catalogs: true
I20250901 01:52:31.455451 1490 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_0c9bfebabc22494489b1c1251a412c81" } modify_external_catalogs: true
I20250901 01:52:31.455607 1490 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_0c9bfebabc22494489b1c1251a412c81" } modify_external_catalogs: true
I20250901 01:52:31.456135 1485 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_da2c70a8f2fe49a0a8184e88c0a9e0ea" } new_table_name: "default.Table_4505d77e4269443995012fef50c7e7b5" modify_external_catalogs: true
I20250901 01:52:31.456220 918 raft_consensus.cc:2466] T df01d5df9fc2424eaf9de4e68e781803 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:31.457906 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e48b30279d9046fc9d723e29a64efc79"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.458086 992 leader_election.cc:304] T df01d5df9fc2424eaf9de4e68e781803 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:31.459110 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9174ca35b90046c38ec5e490d73e73e7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.459765 2409 raft_consensus.cc:2802] T df01d5df9fc2424eaf9de4e68e781803 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:31.459947 2409 raft_consensus.cc:695] T df01d5df9fc2424eaf9de4e68e781803 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:31.460453 2409 consensus_queue.cc:237] T df01d5df9fc2424eaf9de4e68e781803 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:31.460556 992 leader_election.cc:336] T df01d5df9fc2424eaf9de4e68e781803 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:31.460987 1484 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_7736ed4b5a0d4b1a816500c525e68ad3" } modify_external_catalogs: true
I20250901 01:52:31.461196 1484 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_7736ed4b5a0d4b1a816500c525e68ad3" } modify_external_catalogs: true
W20250901 01:52:31.463665 858 consensus_peers.cc:489] T 6c0abbfc8d904dce989ce51961e4237d P 00c282f779c1477d920f24dc0dc16628 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:31.464524 1151 tablet_bootstrap.cc:492] T 74b709cb5ec64006b2f12bead8217d80 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:31.464699 1151 ts_tablet_manager.cc:1397] T 74b709cb5ec64006b2f12bead8217d80 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.142s user 0.009s sys 0.016s
I20250901 01:52:31.468133 1151 raft_consensus.cc:357] T 74b709cb5ec64006b2f12bead8217d80 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:31.469496 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1deaa057128142ffbb59203341a55a37"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.469791 1151 raft_consensus.cc:383] T 74b709cb5ec64006b2f12bead8217d80 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:31.469904 1151 raft_consensus.cc:738] T 74b709cb5ec64006b2f12bead8217d80 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:31.470171 1151 consensus_queue.cc:260] T 74b709cb5ec64006b2f12bead8217d80 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:31.470881 1151 ts_tablet_manager.cc:1428] T 74b709cb5ec64006b2f12bead8217d80 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250901 01:52:31.472971 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3fac2ef6ab5e451fb0cd883e38bcc896"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.474213 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1e2da0a3b2b64deb9cd81b0cc4dbc0cd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.475314 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9e29d344b6a047a498beca7d1d36d86a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.475425 1150 tablet_bootstrap.cc:492] T eee34044c39d4013baf78d3705489626 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:31.475795 1150 ts_tablet_manager.cc:1397] T eee34044c39d4013baf78d3705489626 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.249s user 0.015s sys 0.030s
I20250901 01:52:31.476518 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_eb40e886de67418599e745e8db93881e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.476653 1150 raft_consensus.cc:357] T eee34044c39d4013baf78d3705489626 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:31.476799 1150 raft_consensus.cc:383] T eee34044c39d4013baf78d3705489626 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:31.476848 1150 raft_consensus.cc:738] T eee34044c39d4013baf78d3705489626 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:31.477195 1150 consensus_queue.cc:260] T eee34044c39d4013baf78d3705489626 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:31.477528 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f35885b0a212499f95e74ea85d33b092"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.477895 1150 ts_tablet_manager.cc:1428] T eee34044c39d4013baf78d3705489626 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250901 01:52:31.478266 1150 tablet_bootstrap.cc:492] T c422881b40f54971a68530abfbd1945b P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:31.479947 1150 tablet_bootstrap.cc:654] T c422881b40f54971a68530abfbd1945b P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:31.480273 1151 tablet_bootstrap.cc:492] T eb19e3a424324a24824f6caaf5848952 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:31.468920 2113 ts_tablet_manager.cc:1918] T 7cc1efcb6bff4aba8015d2d3b7f4c923 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 01:52:31.480916 2113 log.cc:1199] T 7cc1efcb6bff4aba8015d2d3b7f4c923 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/7cc1efcb6bff4aba8015d2d3b7f4c923
I20250901 01:52:31.481738 2113 ts_tablet_manager.cc:1939] T 7cc1efcb6bff4aba8015d2d3b7f4c923 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:31.482074 2574 tablet.cc:1722] T 615b2306fca84003b264da9d5f95ddab P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250901 01:52:31.482589 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 141) took 5749 ms. Trace:
I20250901 01:52:31.482668 2113 rpcz_store.cc:276] 0901 01:52:25.732883 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.732982 (+ 99us) service_pool.cc:225] Handling call
0901 01:52:31.365163 (+5632181us) ts_tablet_manager.cc:1145] Deleting tablet 7cc1efcb6bff4aba8015d2d3b7f4c923
0901 01:52:31.365176 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:31.380864 (+ 15688us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:31.468821 (+ 87957us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:31.482568 (+ 13747us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5631958}
I20250901 01:52:31.483052 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 7cc1efcb6bff4aba8015d2d3b7f4c923 (table default.Table_5610feff8a344dc9be4bc1f0ad8ddd51 [id=78b00856a9b747c19f390b4fdae8ddcf]) successfully deleted
I20250901 01:52:31.483112 2113 tablet_replica.cc:331] T 06d5eeb70e664f73bf643922d445bc7a P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:31.483304 2113 raft_consensus.cc:2241] T 06d5eeb70e664f73bf643922d445bc7a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:31.483434 2113 raft_consensus.cc:2270] T 06d5eeb70e664f73bf643922d445bc7a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:31.486371 1151 tablet_bootstrap.cc:654] T eb19e3a424324a24824f6caaf5848952 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:31.487140 2113 ts_tablet_manager.cc:1905] T 06d5eeb70e664f73bf643922d445bc7a P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:31.487437 2596 catalog_manager.cc:1604] T 00000000000000000000000000000000 P e3add7edb97a40ef9f7c6bc011d13227: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250901 01:52:31.488209 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a97c7218b65b453e9c091ab53cb6c75c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.488528 1761 raft_consensus.cc:491] T 6ba88af177874544b764cc32420e7ac9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:31.488672 1761 raft_consensus.cc:513] T 6ba88af177874544b764cc32420e7ac9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:31.489272 1761 leader_election.cc:290] T 6ba88af177874544b764cc32420e7ac9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:31.491058 2118 ts_tablet_manager.cc:1918] T 641a83a3e50d451189c33c7b7b4ca070 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:31.491207 2118 log.cc:1199] T 641a83a3e50d451189c33c7b7b4ca070 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/641a83a3e50d451189c33c7b7b4ca070
I20250901 01:52:31.491595 2118 ts_tablet_manager.cc:1939] T 641a83a3e50d451189c33c7b7b4ca070 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:31.491925 1491 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: default.Table_ed6fd284be6f4ae3a008e21e7ce17930 [id=303e76c918bc4c39aa47a4d24849a358] alter complete (version 1)
I20250901 01:52:31.496014 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_91b03b4bd00b41e0b28a0d7a87313fed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:31.497152 992 consensus_peers.cc:489] T 85322a864d154f299f2d670fdd2a7bfb P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20250901 01:52:31.497190 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_60532ebbed9540aaa1da337c42d0c273"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.500152 2113 ts_tablet_manager.cc:1918] T 06d5eeb70e664f73bf643922d445bc7a P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:31.500262 2113 log.cc:1199] T 06d5eeb70e664f73bf643922d445bc7a P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/06d5eeb70e664f73bf643922d445bc7a
I20250901 01:52:31.500561 2113 ts_tablet_manager.cc:1939] T 06d5eeb70e664f73bf643922d445bc7a P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
W20250901 01:52:31.500813 992 leader_election.cc:336] T 6ba88af177874544b764cc32420e7ac9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:31.506281 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 142) took 5745 ms. Trace:
I20250901 01:52:31.506366 2113 rpcz_store.cc:276] 0901 01:52:25.760775 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.760842 (+ 67us) service_pool.cc:225] Handling call
0901 01:52:31.483066 (+5722224us) ts_tablet_manager.cc:1145] Deleting tablet 06d5eeb70e664f73bf643922d445bc7a
0901 01:52:31.483075 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:31.492721 (+ 9646us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:31.500050 (+ 7329us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:31.506261 (+ 6211us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5721976}
I20250901 01:52:31.506892 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 143) took 5690 ms. Trace:
I20250901 01:52:31.506960 2113 rpcz_store.cc:276] 0901 01:52:25.816925 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.816987 (+ 62us) service_pool.cc:225] Handling call
0901 01:52:31.506784 (+5689797us) ts_tablet_manager.cc:1145] Deleting tablet 410d44e5f1974ed69e5dee65e98856a4
0901 01:52:31.506793 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:31.506878 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5689595}
I20250901 01:52:31.507244 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 06d5eeb70e664f73bf643922d445bc7a (table default.Table_33c84edd575e4969b264a684accb22e2 [id=86e2bc0201e04503a1f60452605eaf27]) successfully deleted
I20250901 01:52:31.507316 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 144) took 5673 ms. Trace:
W20250901 01:52:31.507520 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 410d44e5f1974ed69e5dee65e98856a4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 410d44e5f1974ed69e5dee65e98856a4 already in progress: creating tablet
I20250901 01:52:31.507377 2113 rpcz_store.cc:276] 0901 01:52:25.833396 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.833459 (+ 63us) service_pool.cc:225] Handling call
0901 01:52:31.507238 (+5673779us) ts_tablet_manager.cc:1145] Deleting tablet d4c9aa723aee4dc4b6237cb5c3722b52
0901 01:52:31.507246 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:31.507305 (+ 59us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5673624}
I20250901 01:52:31.508133 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 145) took 5667 ms. Trace:
I20250901 01:52:31.508208 2113 rpcz_store.cc:276] 0901 01:52:25.840592 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.840652 (+ 60us) service_pool.cc:225] Handling call
0901 01:52:31.508045 (+5667393us) ts_tablet_manager.cc:1145] Deleting tablet fa7fb88ded21432a8757f37ec3db5e71
0901 01:52:31.508054 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:31.508120 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5667227}
W20250901 01:52:31.508311 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet d4c9aa723aee4dc4b6237cb5c3722b52 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d4c9aa723aee4dc4b6237cb5c3722b52 already in progress: creating tablet
I20250901 01:52:31.508551 2113 tablet_replica.cc:331] T 2599f98abfdc4871b573edfc2ad97d22 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:31.508754 2113 raft_consensus.cc:2241] T 2599f98abfdc4871b573edfc2ad97d22 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:31.508909 2113 raft_consensus.cc:2270] T 2599f98abfdc4871b573edfc2ad97d22 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:31.509392 2113 ts_tablet_manager.cc:1905] T 2599f98abfdc4871b573edfc2ad97d22 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:31.518095 2518 raft_consensus.cc:491] T b13afacd37474781877b9f2cd20b313a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:31.518239 2518 raft_consensus.cc:513] T b13afacd37474781877b9f2cd20b313a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:31.519117 2518 leader_election.cc:290] T b13afacd37474781877b9f2cd20b313a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:31.520769 992 leader_election.cc:336] T b13afacd37474781877b9f2cd20b313a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:31.521694 1490 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet bd9b59f0864a497faf281edd6077668d
W20250901 01:52:31.521901 1490 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet bd9b59f0864a497faf281edd6077668d: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:31.522861 2113 ts_tablet_manager.cc:1918] T 2599f98abfdc4871b573edfc2ad97d22 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250901 01:52:31.523384 1488 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8151bad707c04ac687c5bddfea573e3f consensus state has no leader
I20250901 01:52:31.523214 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet bd9b59f0864a497faf281edd6077668d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:31 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:31.522969 2113 log.cc:1199] T 2599f98abfdc4871b573edfc2ad97d22 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/2599f98abfdc4871b573edfc2ad97d22
I20250901 01:52:31.524857 2113 ts_tablet_manager.cc:1939] T 2599f98abfdc4871b573edfc2ad97d22 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:31.525660 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 146) took 5541 ms. Trace:
I20250901 01:52:31.528667 2113 rpcz_store.cc:276] 0901 01:52:25.984566 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.985453 (+ 887us) service_pool.cc:225] Handling call
0901 01:52:31.508501 (+5523048us) ts_tablet_manager.cc:1145] Deleting tablet 2599f98abfdc4871b573edfc2ad97d22
0901 01:52:31.508509 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:31.514316 (+ 5807us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:31.522757 (+ 8441us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:31.525643 (+ 2886us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5522909}
I20250901 01:52:31.529259 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 147) took 5543 ms. Trace:
I20250901 01:52:31.529269 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 2599f98abfdc4871b573edfc2ad97d22 (table default.Table_0e63d21acfcb4c028c9ea6406b6a04e4 [id=81323e6fbe424655b8e185f52657fadc]) successfully deleted
I20250901 01:52:31.529382 2113 rpcz_store.cc:276] 0901 01:52:25.985938 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:25.986351 (+ 413us) service_pool.cc:225] Handling call
0901 01:52:31.529148 (+5542797us) ts_tablet_manager.cc:1145] Deleting tablet 75a703c8e20a4a2fa0ab6ec58d41ef92
0901 01:52:31.529158 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:31.529245 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5542663}
W20250901 01:52:31.529816 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 75a703c8e20a4a2fa0ab6ec58d41ef92 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 75a703c8e20a4a2fa0ab6ec58d41ef92 already in progress: creating tablet
I20250901 01:52:31.532140 2113 tablet_replica.cc:331] T f62d96c37cca4143a4b098b7c69979e6 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
W20250901 01:52:31.532487 1485 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b164b4a67a9143ceaff69b0ce49d63fe consensus state has no leader
I20250901 01:52:31.532610 2113 raft_consensus.cc:2241] T f62d96c37cca4143a4b098b7c69979e6 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:31.532948 2113 raft_consensus.cc:2270] T f62d96c37cca4143a4b098b7c69979e6 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:31.533969 2113 ts_tablet_manager.cc:1905] T f62d96c37cca4143a4b098b7c69979e6 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:31.535950 2518 raft_consensus.cc:491] T 5afae316ff164413be0d25d4bc0f58dc P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:31.536059 2518 raft_consensus.cc:513] T 5afae316ff164413be0d25d4bc0f58dc P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:31.536424 992 consensus_peers.cc:489] T 500e3c33598b45a4afa983da7cb0845a P fd27ab6665304aaea53adbd102dd4f58 -> Peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Couldn't send request to peer 00c282f779c1477d920f24dc0dc16628. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 500e3c33598b45a4afa983da7cb0845a. This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:31.536604 2518 leader_election.cc:290] T 5afae316ff164413be0d25d4bc0f58dc P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:31.538220 992 leader_election.cc:336] T 5afae316ff164413be0d25d4bc0f58dc P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:31.543422 2113 ts_tablet_manager.cc:1918] T f62d96c37cca4143a4b098b7c69979e6 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250901 01:52:31.543525 2113 log.cc:1199] T f62d96c37cca4143a4b098b7c69979e6 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/f62d96c37cca4143a4b098b7c69979e6
I20250901 01:52:31.543844 2113 ts_tablet_manager.cc:1939] T f62d96c37cca4143a4b098b7c69979e6 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:31.544550 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 155) took 5528 ms. Trace:
I20250901 01:52:31.544629 2113 rpcz_store.cc:276] 0901 01:52:26.016503 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.016619 (+ 116us) service_pool.cc:225] Handling call
0901 01:52:31.532094 (+5515475us) ts_tablet_manager.cc:1145] Deleting tablet f62d96c37cca4143a4b098b7c69979e6
0901 01:52:31.532103 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:31.539294 (+ 7191us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:31.543327 (+ 4033us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:31.544531 (+ 1204us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5514514}
I20250901 01:52:31.545047 2113 tablet_replica.cc:331] T f9b293bf6a17453b9ea9892a6970d0b0 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:31.545218 2113 raft_consensus.cc:2241] T f9b293bf6a17453b9ea9892a6970d0b0 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:31.545372 2113 raft_consensus.cc:2270] T f9b293bf6a17453b9ea9892a6970d0b0 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250901 01:52:31.546736 992 consensus_peers.cc:489] T 1495de70f0844f82a0eea99c059f076b P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:31.546916 2113 ts_tablet_manager.cc:1905] T f9b293bf6a17453b9ea9892a6970d0b0 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:31.549474 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet f62d96c37cca4143a4b098b7c69979e6 (table default.Table_2ddf898959684cdc995c1351d84a9661 [id=1f65b3e09c9748f189ecb2b56c5d1132]) successfully deleted
I20250901 01:52:31.550093 1484 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet fae2779fac164b9f8052c3666db803b8
W20250901 01:52:31.550467 1484 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet fae2779fac164b9f8052c3666db803b8: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:31.551993 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet fae2779fac164b9f8052c3666db803b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:31 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:31.553732 2113 ts_tablet_manager.cc:1918] T f9b293bf6a17453b9ea9892a6970d0b0 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:31.553835 2113 log.cc:1199] T f9b293bf6a17453b9ea9892a6970d0b0 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/f9b293bf6a17453b9ea9892a6970d0b0
I20250901 01:52:31.554355 2113 ts_tablet_manager.cc:1939] T f9b293bf6a17453b9ea9892a6970d0b0 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:31.563367 2574 tablet.cc:1722] T 7954393c8c1c4457ab8bb723c03a9c8b P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6
E20250901 01:52:31.564162 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 0c88c339e59a4c1da21756cba0645bdd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d25639c7008f4e29af6226039e4b976b'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:31.564337 1696 catalog_manager.cc:6205] Error selecting replicas for tablet a656adbf1b29431687e0989f0db3459c, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2d3d2227e6a24bd2a86f457e553e698b'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:31.581000 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 156) took 5561 ms. Trace:
I20250901 01:52:31.581123 2113 rpcz_store.cc:276] 0901 01:52:26.019238 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.019295 (+ 57us) service_pool.cc:225] Handling call
0901 01:52:31.545000 (+5525705us) ts_tablet_manager.cc:1145] Deleting tablet f9b293bf6a17453b9ea9892a6970d0b0
0901 01:52:31.545011 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:31.550100 (+ 5089us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:31.553653 (+ 3553us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:31.580975 (+ 27322us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5523486}
I20250901 01:52:31.581434 2113 tablet_replica.cc:331] T 72ba689f9230411995dc96e2b45601df P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:31.581637 2113 raft_consensus.cc:2241] T 72ba689f9230411995dc96e2b45601df P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:31.581964 2113 raft_consensus.cc:2270] T 72ba689f9230411995dc96e2b45601df P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:31.582502 2113 ts_tablet_manager.cc:1905] T 72ba689f9230411995dc96e2b45601df P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
E20250901 01:52:31.587426 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 494b6488aa0d4355b4caa46004267465, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_eecfab7239cc40c5a2f1fb6ac6d35dc9'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:31.587817 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 0ee3f35392954817b350e5f48653be78, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_442e11b9211c4b209c68f653d7bfc313'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:31.588137 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae4844183a3c4a04a775e6fd878391a3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_aee74198b51d4f089538c4f2bf3ce4dc'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:31.569499 1484 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_9896b980158948339a0b91f5882ed56b" } modify_external_catalogs: true
I20250901 01:52:31.588861 1484 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_9896b980158948339a0b91f5882ed56b" } modify_external_catalogs: true
I20250901 01:52:31.570006 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7d42ccb2cb724391b8cbdda7bed2fae9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.576418 1483 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_eecfab7239cc40c5a2f1fb6ac6d35dc9" } new_table_name: "default.Table_3ef97ecacc3042099ec1cc5c3a7a0763" modify_external_catalogs: true
I20250901 01:52:31.584586 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet f9b293bf6a17453b9ea9892a6970d0b0 (table default.Table_a7fa05538bbd4766aa24d557677a692c [id=b67bfe518a094e0781ef931524b55990]) successfully deleted
I20250901 01:52:31.601166 1103 heartbeater.cc:499] Master 127.0.104.60:44231 was elected leader, sending a full tablet report...
W20250901 01:52:31.602190 1483 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 494b6488aa0d4355b4caa46004267465 has no consensus state
W20250901 01:52:31.608724 992 consensus_peers.cc:489] T 5adf6b492640417ca071c94702b48cb9 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250901 01:52:31.614835 2412 consensus_queue.cc:1035] T 1495de70f0844f82a0eea99c059f076b P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:31.568819 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_db2012cb9bc3414b980bea6cdc0bdb61"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.619226 2574 tablet.cc:1722] T ed1b41340ce742a88c3e4fd85cd25ca6 P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250901 01:52:31.493059 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 435b1cca5a7d4d49959a310cb250b2a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:26 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:31.494379 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 061b760d8e6647c6ab0b5d538840a230 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:26 UTC) from {username='slave'} at 127.0.0.1:34472
E20250901 01:52:31.625880 1696 catalog_manager.cc:6205] Error selecting replicas for tablet be8dfbc466564842a2fdb4953eee939b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_75ba3ba70ca24a94920e7bce96190ff2'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:31.626048 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9745582cebfe47b58c21d0191ab4cb9b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77728af9eee44f29a1e2131d5b75b7ca'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:31.626168 1696 catalog_manager.cc:6205] Error selecting replicas for tablet e2ed32b3292141278af28026528e96c2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b5077f55951d48be92a67407ffe26a08'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:31.626294 1696 catalog_manager.cc:6205] Error selecting replicas for tablet eea6bfabe6cc4f83905c24f0f7a5dfc9, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_e0902dfbdd984f85b3fdac8d283731f7'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:31.626400 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5a7fa68d29644ae8afd2711e39dde59a, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_f9e2a9182f6340bfa551fa2fc2eb2429'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:31.626518 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ff2bd017417c41b5b8225b8f00283e63, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_97526c9a6c9f4acbb6dedd78475571e0'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:31.626626 1696 catalog_manager.cc:6205] Error selecting replicas for tablet dcc60bb000f440f1aca9133928b5ce20, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2fbca8463d6546c9b8f5b018fc60251d'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:31.626742 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 182d99cb66024b66b16f1a4e76084260, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_dc856980ff4c4aafb92cf1fd4f29e694'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:31.626852 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 65e5cdc10e9642b7b0df775cb02861fa, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_1b5729d9768546fe99eaafc4e2604f42'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:31.626967 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f8f923ddcd2740bcb7e521f4f57c4034, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_5e65c5f068b84995a2d7a76fa65dda0f'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:31.627082 1696 catalog_manager.cc:6205] Error selecting replicas for tablet e4d8330a4e454d1b8d643e17a6996741, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ce16a1b94f714c2699ee6cd916bb0215'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:31.627202 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 0813cc68d4b0488ba60a523e4479971b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d1302673d68f424ebb54b220911ce385'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:31.627306 1696 catalog_manager.cc:6205] Error selecting replicas for tablet b1ba0e19dc0b462e80c537bf2d183f5b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_66b111271c15410aab23682f8f6081ac'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:31.627420 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 8e10a01e77d14cf28c2c5e0205e595b9, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_83cf2d86a1284544ba2e7aa9ebb66bb7'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:31.627521 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 05a547a8866149cb95d23e960a737237, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_583e53c2e86544e9be68eb27ab6068a3'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:31.496517 917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ba88af177874544b764cc32420e7ac9" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
E20250901 01:52:31.627653 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5f0e054f62de4cf6bc6f69ad6664b192, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ef6832d0845044ed9f165ca3541b4c3f'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:31.497627 2543 mvcc.cc:204] Tried to move back new op lower bound from 7195408594223718400 to 7195408589610164224. Current Snapshot: MvccSnapshot[applied={T|T < 7195408594223718400}]
I20250901 01:52:31.519788 918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b13afacd37474781877b9f2cd20b313a" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:31.627772 917 raft_consensus.cc:2466] T 6ba88af177874544b764cc32420e7ac9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:31.627898 918 raft_consensus.cc:2466] T b13afacd37474781877b9f2cd20b313a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:31.628125 992 leader_election.cc:304] T 6ba88af177874544b764cc32420e7ac9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:31.522961 898 tablet_service.cc:1515] Processing DeleteTablet for tablet bd9b59f0864a497faf281edd6077668d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:31 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:31.537326 919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5afae316ff164413be0d25d4bc0f58dc" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:31.628479 2412 raft_consensus.cc:2802] T 6ba88af177874544b764cc32420e7ac9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:31.628587 2412 raft_consensus.cc:491] T 6ba88af177874544b764cc32420e7ac9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:31.628587 919 raft_consensus.cc:2466] T 5afae316ff164413be0d25d4bc0f58dc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:31.628666 2412 raft_consensus.cc:3058] T 6ba88af177874544b764cc32420e7ac9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:31.540292 2608 raft_consensus.cc:491] T 3697821cd29446bb908b020599df0db9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:31.628722 992 leader_election.cc:304] T b13afacd37474781877b9f2cd20b313a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:31.628793 2608 raft_consensus.cc:513] T 3697821cd29446bb908b020599df0db9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:31.629040 2666 raft_consensus.cc:2802] T b13afacd37474781877b9f2cd20b313a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:31.629102 2666 raft_consensus.cc:491] T b13afacd37474781877b9f2cd20b313a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:31.629151 2666 raft_consensus.cc:3058] T b13afacd37474781877b9f2cd20b313a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:31.629364 2608 leader_election.cc:290] T 3697821cd29446bb908b020599df0db9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:31.551386 901 tablet_service.cc:1515] Processing DeleteTablet for tablet fae2779fac164b9f8052c3666db803b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:31 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:31.564934 2582 tablet.cc:1722] T 7954393c8c1c4457ab8bb723c03a9c8b P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6
I20250901 01:52:31.631222 2412 raft_consensus.cc:513] T 6ba88af177874544b764cc32420e7ac9 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:31.620218 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 3dae00aa38204ee3ac77325eb334be42 with delete_type TABLET_DATA_DELETED (replaced by tablet 76ba642c16734cdf969dbd3130639fda) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:31.631348 2666 raft_consensus.cc:513] T b13afacd37474781877b9f2cd20b313a P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:31.631842 2666 leader_election.cc:290] T b13afacd37474781877b9f2cd20b313a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:31.621340 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 3363fd67b1ad44849ecb7b4090b61a72 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:26 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:31.634754 2582 tablet.cc:1722] T ed1b41340ce742a88c3e4fd85cd25ca6 P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250901 01:52:31.628362 2684 tablet.cc:1722] T 615b2306fca84003b264da9d5f95ddab P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250901 01:52:31.622604 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 42c1156e8dd84c6fa1396b46de13eb6f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:26 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:31.624099 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 008d23f236c348b6be9aee718f8698af with delete_type TABLET_DATA_DELETED (replaced by tablet b1ba0e19dc0b462e80c537bf2d183f5b) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:31.639076 2412 leader_election.cc:290] T 6ba88af177874544b764cc32420e7ac9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:31.640113 2412 raft_consensus.cc:491] T f3962b471bdb49aea6e9fa95f1f9dfe2 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:31.640228 2412 raft_consensus.cc:513] T f3962b471bdb49aea6e9fa95f1f9dfe2 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:31.640751 2412 leader_election.cc:290] T f3962b471bdb49aea6e9fa95f1f9dfe2 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:31.640753 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3fc4acf860e0435aa289674252485c0a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.641777 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3697821cd29446bb908b020599df0db9" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:31.642128 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_78afc895195e436c993de77c7838b0cd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.642619 2113 ts_tablet_manager.cc:1918] T 72ba689f9230411995dc96e2b45601df P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 01:52:31.642726 2113 log.cc:1199] T 72ba689f9230411995dc96e2b45601df P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/72ba689f9230411995dc96e2b45601df
I20250901 01:52:31.643065 2113 ts_tablet_manager.cc:1939] T 72ba689f9230411995dc96e2b45601df P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
W20250901 01:52:31.643052 858 leader_election.cc:336] T 3697821cd29446bb908b020599df0db9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:31.644567 859 leader_election.cc:343] T 3697821cd29446bb908b020599df0db9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:31.644675 859 leader_election.cc:304] T 3697821cd29446bb908b020599df0db9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:31.645041 2608 raft_consensus.cc:2747] T 3697821cd29446bb908b020599df0db9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:31.645165 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 159) took 5581 ms. Trace:
I20250901 01:52:31.645243 2113 rpcz_store.cc:276] 0901 01:52:26.063609 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.063673 (+ 64us) service_pool.cc:225] Handling call
0901 01:52:31.581376 (+5517703us) ts_tablet_manager.cc:1145] Deleting tablet 72ba689f9230411995dc96e2b45601df
0901 01:52:31.581388 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:31.586562 (+ 5174us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:31.642519 (+ 55957us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:31.645146 (+ 2627us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5499522}
I20250901 01:52:31.645619 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 72ba689f9230411995dc96e2b45601df (table default.Table_604f2c5c58d74154bacf2a065a467f32 [id=351a05134f4948039c920cafbe050914]) successfully deleted
I20250901 01:52:31.645782 919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b13afacd37474781877b9f2cd20b313a" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:31.645454 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_0b18882d811746dc98d44d9f579b0c7b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.645951 919 raft_consensus.cc:3058] T b13afacd37474781877b9f2cd20b313a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
W20250901 01:52:31.648510 992 leader_election.cc:336] T 6ba88af177874544b764cc32420e7ac9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:31.648631 992 leader_election.cc:336] T f3962b471bdb49aea6e9fa95f1f9dfe2 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:31.648713 992 leader_election.cc:336] T b13afacd37474781877b9f2cd20b313a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:31.648764 2113 tablet_replica.cc:331] T 606fac5c6efe48d1877104b01c23ba5b P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:31.648905 992 leader_election.cc:304] T 5afae316ff164413be0d25d4bc0f58dc P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:31.648955 2113 raft_consensus.cc:2241] T 606fac5c6efe48d1877104b01c23ba5b P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:31.649262 2113 raft_consensus.cc:2270] T 606fac5c6efe48d1877104b01c23ba5b P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:31.649329 919 raft_consensus.cc:2466] T b13afacd37474781877b9f2cd20b313a P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:31.649935 992 leader_election.cc:304] T b13afacd37474781877b9f2cd20b313a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:31.650198 1761 raft_consensus.cc:2802] T 5afae316ff164413be0d25d4bc0f58dc P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:31.650287 1761 raft_consensus.cc:491] T 5afae316ff164413be0d25d4bc0f58dc P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:31.650341 1761 raft_consensus.cc:3058] T 5afae316ff164413be0d25d4bc0f58dc P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:31.650882 2666 raft_consensus.cc:2802] T b13afacd37474781877b9f2cd20b313a P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:31.651257 2666 raft_consensus.cc:695] T b13afacd37474781877b9f2cd20b313a P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:31.651471 2666 consensus_queue.cc:237] T b13afacd37474781877b9f2cd20b313a P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:31.652491 918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ba88af177874544b764cc32420e7ac9" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:31.652650 918 raft_consensus.cc:3058] T 6ba88af177874544b764cc32420e7ac9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:31.652890 1761 raft_consensus.cc:513] T 5afae316ff164413be0d25d4bc0f58dc P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:31.653327 1761 leader_election.cc:290] T 5afae316ff164413be0d25d4bc0f58dc P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:31.650063 2113 ts_tablet_manager.cc:1905] T 606fac5c6efe48d1877104b01c23ba5b P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:31.655048 918 raft_consensus.cc:2466] T 6ba88af177874544b764cc32420e7ac9 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:31.655512 922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5afae316ff164413be0d25d4bc0f58dc" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:31.655716 922 raft_consensus.cc:3058] T 5afae316ff164413be0d25d4bc0f58dc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:31.656106 992 leader_election.cc:304] T 6ba88af177874544b764cc32420e7ac9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:31.656317 2409 raft_consensus.cc:2802] T 6ba88af177874544b764cc32420e7ac9 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:31.656447 2409 raft_consensus.cc:695] T 6ba88af177874544b764cc32420e7ac9 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
W20250901 01:52:31.656466 992 leader_election.cc:343] T f3962b471bdb49aea6e9fa95f1f9dfe2 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: f3962b471bdb49aea6e9fa95f1f9dfe2
I20250901 01:52:31.656576 992 leader_election.cc:304] T f3962b471bdb49aea6e9fa95f1f9dfe2 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:31.656661 2409 consensus_queue.cc:237] T 6ba88af177874544b764cc32420e7ac9 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:31.656973 2613 raft_consensus.cc:2747] T f3962b471bdb49aea6e9fa95f1f9dfe2 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:31.657920 992 leader_election.cc:336] T 5afae316ff164413be0d25d4bc0f58dc P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:31.658947 922 raft_consensus.cc:2466] T 5afae316ff164413be0d25d4bc0f58dc P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:31.659332 992 leader_election.cc:304] T 5afae316ff164413be0d25d4bc0f58dc P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:31.659741 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 146) took 4553 ms. Trace:
I20250901 01:52:31.659875 2118 rpcz_store.cc:276] 0901 01:52:27.106666 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:27.106718 (+ 52us) service_pool.cc:225] Handling call
0901 01:52:31.272452 (+4165734us) ts_tablet_manager.cc:1145] Deleting tablet 641a83a3e50d451189c33c7b7b4ca070
0901 01:52:31.272466 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:31.418397 (+145931us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:31.490964 (+ 72567us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:31.659724 (+168760us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":24218,"tablet-delete.queue_time_us":4137751}
I20250901 01:52:31.660048 1761 raft_consensus.cc:2802] T 5afae316ff164413be0d25d4bc0f58dc P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:31.660167 1761 raft_consensus.cc:695] T 5afae316ff164413be0d25d4bc0f58dc P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:31.660444 1761 consensus_queue.cc:237] T 5afae316ff164413be0d25d4bc0f58dc P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:31.660542 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 641a83a3e50d451189c33c7b7b4ca070 (table default.Table_ef6832d0845044ed9f165ca3541b4c3f [id=0e69044796ea4292b1dc100db2f26cee]) successfully deleted
I20250901 01:52:31.660979 2118 tablet_replica.cc:331] T f23df6f253a943fa8e146bb8c811ef51 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:31.661273 2118 raft_consensus.cc:2241] T f23df6f253a943fa8e146bb8c811ef51 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:31.661450 2118 raft_consensus.cc:2270] T f23df6f253a943fa8e146bb8c811ef51 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:31.661984 2118 ts_tablet_manager.cc:1905] T f23df6f253a943fa8e146bb8c811ef51 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:31.664535 992 consensus_peers.cc:489] T aa424ac6ef1441e6b1cc2867b1c48186 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250901 01:52:31.667022 1151 tablet_bootstrap.cc:492] T eb19e3a424324a24824f6caaf5848952 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:31.667564 1151 ts_tablet_manager.cc:1397] T eb19e3a424324a24824f6caaf5848952 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.187s user 0.010s sys 0.034s
I20250901 01:52:31.667785 2113 ts_tablet_manager.cc:1918] T 606fac5c6efe48d1877104b01c23ba5b P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 01:52:31.667917 2113 log.cc:1199] T 606fac5c6efe48d1877104b01c23ba5b P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/606fac5c6efe48d1877104b01c23ba5b
I20250901 01:52:31.668324 2113 ts_tablet_manager.cc:1939] T 606fac5c6efe48d1877104b01c23ba5b P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:31.669195 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5ac5c13947504436898043e2ed1f8535"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:31.627741 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae9a4ad4fdfb4771ba23a91f7007447e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_5e84ae9a9c2a4084a1f9c5ffb4594de2'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:31.671115 1696 catalog_manager.cc:6205] Error selecting replicas for tablet d602c8986c75492ead9936ac04e2622d, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_82d91a1ba2eb4265896824028dec8d3a'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:31.674084 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 161) took 5457 ms. Trace:
I20250901 01:52:31.674178 2113 rpcz_store.cc:276] 0901 01:52:26.216518 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.216572 (+ 54us) service_pool.cc:225] Handling call
0901 01:52:31.648714 (+5432142us) ts_tablet_manager.cc:1145] Deleting tablet 606fac5c6efe48d1877104b01c23ba5b
0901 01:52:31.648725 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:31.664112 (+ 15387us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:31.667678 (+ 3566us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:31.674066 (+ 6388us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5431958}
I20250901 01:52:31.674829 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 162) took 5420 ms. Trace:
I20250901 01:52:31.674912 2113 rpcz_store.cc:276] 0901 01:52:26.254092 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.254149 (+ 57us) service_pool.cc:225] Handling call
0901 01:52:31.674734 (+5420585us) ts_tablet_manager.cc:1145] Deleting tablet 87e97a9544a2485897b1053c8929a39a
0901 01:52:31.674744 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:31.674812 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5420422}
I20250901 01:52:31.675474 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 163) took 5405 ms. Trace:
I20250901 01:52:31.675556 2113 rpcz_store.cc:276] 0901 01:52:26.269961 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.270018 (+ 57us) service_pool.cc:225] Handling call
0901 01:52:31.675405 (+5405387us) ts_tablet_manager.cc:1145] Deleting tablet 0f37a24e693847f98b574950379081f3
0901 01:52:31.675413 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:31.675461 (+ 48us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5405179}
I20250901 01:52:31.675938 2113 tablet_replica.cc:331] T ddbc15685fca4c708329420109ad0c57 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:31.676139 2113 raft_consensus.cc:2241] T ddbc15685fca4c708329420109ad0c57 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:31.676478 2113 raft_consensus.cc:2270] T ddbc15685fca4c708329420109ad0c57 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:31.677023 2113 ts_tablet_manager.cc:1905] T ddbc15685fca4c708329420109ad0c57 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:31.683342 1151 raft_consensus.cc:357] T eb19e3a424324a24824f6caaf5848952 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:31.683844 1151 raft_consensus.cc:383] T eb19e3a424324a24824f6caaf5848952 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:31.683974 1151 raft_consensus.cc:738] T eb19e3a424324a24824f6caaf5848952 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:31.683537 2113 ts_tablet_manager.cc:1918] T ddbc15685fca4c708329420109ad0c57 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 01:52:31.684279 2113 log.cc:1199] T ddbc15685fca4c708329420109ad0c57 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/ddbc15685fca4c708329420109ad0c57
I20250901 01:52:31.685065 2113 ts_tablet_manager.cc:1939] T ddbc15685fca4c708329420109ad0c57 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:31.684545 1151 consensus_queue.cc:260] T eb19e3a424324a24824f6caaf5848952 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:31.686372 1151 ts_tablet_manager.cc:1428] T eb19e3a424324a24824f6caaf5848952 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
W20250901 01:52:31.675275 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 87e97a9544a2485897b1053c8929a39a because the tablet was not found. No further retry: Not found: Tablet not found: 87e97a9544a2485897b1053c8929a39a
I20250901 01:52:31.682073 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_dc081928d23f49a88297b00ea332176e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.682690 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_be2a77efd50a46e580fbeb56dafad5d9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:31.687551 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 0f37a24e693847f98b574950379081f3 because the tablet was not found. No further retry: Not found: Tablet not found: 0f37a24e693847f98b574950379081f3
I20250901 01:52:31.687963 1485 master_service.cc:948] ReplaceTablet: received request to replace tablet f80f13db501c42c7950499df8fcb6fc8 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:31.688525 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_eabd47a85ad847cf920db88f69f8c093"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.689252 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e7ac6c480bb2490eb6e29a836fa3415f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.689870 1485 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_a96f01d4ce3c4694a635575ffaa4f02d" } modify_external_catalogs: true
I20250901 01:52:31.690037 1485 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_a96f01d4ce3c4694a635575ffaa4f02d" } modify_external_catalogs: true
I20250901 01:52:31.683175 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_806419864ff34c9fa1f13d7940c36f99"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.691205 1488 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_cb4904990e284a1e8137d11272036154" } new_table_name: "default.Table_50ff85e8f1314f4499e76299441bc110" modify_external_catalogs: true
I20250901 01:52:31.692276 1151 tablet_bootstrap.cc:492] T b85d64e02cc646e380581ac9f59dee9e P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
W20250901 01:52:31.689201 992 consensus_peers.cc:489] T 7442c6ff699141e1aff208ebdc9d192a P fd27ab6665304aaea53adbd102dd4f58 -> Peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Couldn't send request to peer 00c282f779c1477d920f24dc0dc16628. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7442c6ff699141e1aff208ebdc9d192a. This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:31.692965 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 165) took 5134 ms. Trace:
I20250901 01:52:31.693043 2113 rpcz_store.cc:276] 0901 01:52:26.558907 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.559360 (+ 453us) service_pool.cc:225] Handling call
0901 01:52:31.675895 (+5116535us) ts_tablet_manager.cc:1145] Deleting tablet ddbc15685fca4c708329420109ad0c57
0901 01:52:31.675904 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:31.679993 (+ 4089us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:31.683442 (+ 3449us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:31.692948 (+ 9506us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5116366}
I20250901 01:52:31.693464 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet ddbc15685fca4c708329420109ad0c57 (table default.Table_942175dadf4b496195b2fa53f39f6ddc [id=2bd332b452f84f25af4225af8e6c6740]) successfully deleted
I20250901 01:52:31.693809 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 166) took 5113 ms. Trace:
I20250901 01:52:31.693882 2113 rpcz_store.cc:276] 0901 01:52:26.580172 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.580254 (+ 82us) service_pool.cc:225] Handling call
0901 01:52:31.693695 (+5113441us) ts_tablet_manager.cc:1145] Deleting tablet da5f69c5b0ca4257ad2ff7918ee2af2e
0901 01:52:31.693706 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:31.693793 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5113194}
W20250901 01:52:31.694257 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet da5f69c5b0ca4257ad2ff7918ee2af2e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet da5f69c5b0ca4257ad2ff7918ee2af2e already in progress: creating tablet
I20250901 01:52:31.683627 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_26404e6956194e98aa06cc347e3a66ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.684010 1486 master_service.cc:948] ReplaceTablet: received request to replace tablet 7b16ca1daf394a6da65a4af8da82ec33 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:31.694590 2113 tablet_replica.cc:331] T 8ed6616ce54548ba83ba299c4ecf3c38 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:31.696847 2113 raft_consensus.cc:2241] T 8ed6616ce54548ba83ba299c4ecf3c38 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:31.697141 2113 raft_consensus.cc:2270] T 8ed6616ce54548ba83ba299c4ecf3c38 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:31.697674 2113 ts_tablet_manager.cc:1905] T 8ed6616ce54548ba83ba299c4ecf3c38 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:31.703071 1488 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 98c56eb62ad84f79b04f17a43d44c479 consensus state has no leader
E20250901 01:52:31.708679 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 93aa22afc0ad4d7fa24807ea3c5d5c21, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0f4b4094d2974b7591875375eda6bed0'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:31.708987 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9074543dea9949e7a085717b40913348, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_53e30205c5fd488eb3b48c685a159d04'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:31.709132 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 3a62034e688348ed9e2ac9de009ed3a2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_409dad98d7eb455a8e16edfb4acedb34'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:31.709278 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 18498c9cb09141d08c1c5c0854b63ebd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77f4810bc83c4d948531da2a45a3792e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:31.709403 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f5e3316adeae47cd848833dd84b972bb, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7570c3012f8a4ae7bcfc7346eadac92a'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:31.709578 1696 catalog_manager.cc:6205] Error selecting replicas for tablet a6b539e160194da3a849d9e2f1ec9859, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_03fb48eaab9b42858ab7cefb722e43a8'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:31.709751 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 66719d8e9c614657820440ebe8ce7c1e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b3e694397be14321a668e46c09c51a71'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:31.709908 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 6ce9521b925b4f97bb4a810ce77e147e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_9e04f976a93f4d30baccf90fa3622dfe'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:31.710050 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2f61edb50d7a42198bfd70639c7bf4de, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_a5c783cbdb694af69047b31d99ff8def'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:31.710901 2113 ts_tablet_manager.cc:1918] T 8ed6616ce54548ba83ba299c4ecf3c38 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:31.711030 2113 log.cc:1199] T 8ed6616ce54548ba83ba299c4ecf3c38 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/8ed6616ce54548ba83ba299c4ecf3c38
I20250901 01:52:31.711380 2113 ts_tablet_manager.cc:1939] T 8ed6616ce54548ba83ba299c4ecf3c38 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:31.712167 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 167) took 5110 ms. Trace:
I20250901 01:52:31.712252 2113 rpcz_store.cc:276] 0901 01:52:26.601868 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.601917 (+ 49us) service_pool.cc:225] Handling call
0901 01:52:31.694493 (+5092576us) ts_tablet_manager.cc:1145] Deleting tablet 8ed6616ce54548ba83ba299c4ecf3c38
0901 01:52:31.694502 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:31.700679 (+ 6177us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:31.710783 (+ 10104us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:31.712143 (+ 1360us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5092411}
I20250901 01:52:31.714994 2113 tablet_replica.cc:331] T 31614db4c53a4f3c96fd190d326ce601 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:31.715183 2113 raft_consensus.cc:2241] T 31614db4c53a4f3c96fd190d326ce601 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:31.715317 2113 raft_consensus.cc:2270] T 31614db4c53a4f3c96fd190d326ce601 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:31.715826 2113 ts_tablet_manager.cc:1905] T 31614db4c53a4f3c96fd190d326ce601 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:31.718140 992 consensus_peers.cc:489] T 42c1156e8dd84c6fa1396b46de13eb6f P fd27ab6665304aaea53adbd102dd4f58 -> Peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Couldn't send request to peer 00c282f779c1477d920f24dc0dc16628. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 42c1156e8dd84c6fa1396b46de13eb6f. This is attempt 1: this message will repeat every 5th retry.
W20250901 01:52:31.719070 992 consensus_peers.cc:489] T 3239e58531d445fa80165cf678bc277c P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:31.722477 1151 tablet_bootstrap.cc:654] T b85d64e02cc646e380581ac9f59dee9e P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:31.665729 1150 tablet_bootstrap.cc:492] T c422881b40f54971a68530abfbd1945b P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:31.723668 2666 raft_consensus.cc:491] T abf6e3be60f74872b2308dc95bf9cd37 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:31.723680 1150 ts_tablet_manager.cc:1397] T c422881b40f54971a68530abfbd1945b P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.245s user 0.013s sys 0.031s
I20250901 01:52:31.723789 2666 raft_consensus.cc:513] T abf6e3be60f74872b2308dc95bf9cd37 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:31.724360 2666 leader_election.cc:290] T abf6e3be60f74872b2308dc95bf9cd37 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:31.707381 2608 raft_consensus.cc:491] T 3eba54e5ddd14c518f4cbd539268ae37 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:31.724503 2608 raft_consensus.cc:513] T 3eba54e5ddd14c518f4cbd539268ae37 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:31.725044 2608 leader_election.cc:290] T 3eba54e5ddd14c518f4cbd539268ae37 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:31.716856 2357 raft_consensus.cc:491] T c6f7c525f59b49938b697de98d4b5835 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:31.725193 2357 raft_consensus.cc:513] T c6f7c525f59b49938b697de98d4b5835 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:31.725554 921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abf6e3be60f74872b2308dc95bf9cd37" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:31.725665 2357 leader_election.cc:290] T c6f7c525f59b49938b697de98d4b5835 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:31.725701 921 raft_consensus.cc:2466] T abf6e3be60f74872b2308dc95bf9cd37 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:31.726063 992 leader_election.cc:304] T abf6e3be60f74872b2308dc95bf9cd37 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:31.726531 1150 raft_consensus.cc:357] T c422881b40f54971a68530abfbd1945b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:31.726670 1150 raft_consensus.cc:383] T c422881b40f54971a68530abfbd1945b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:31.726713 1150 raft_consensus.cc:738] T c422881b40f54971a68530abfbd1945b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:31.726729 2666 raft_consensus.cc:2802] T abf6e3be60f74872b2308dc95bf9cd37 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:31.726801 2666 raft_consensus.cc:491] T abf6e3be60f74872b2308dc95bf9cd37 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:31.726858 2666 raft_consensus.cc:3058] T abf6e3be60f74872b2308dc95bf9cd37 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:31.726878 1150 consensus_queue.cc:260] T c422881b40f54971a68530abfbd1945b P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:31.727365 1150 ts_tablet_manager.cc:1428] T c422881b40f54971a68530abfbd1945b P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20250901 01:52:31.727651 1150 tablet_bootstrap.cc:492] T 6ba8241644984394ada95b561e76467e P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:31.729259 1150 tablet_bootstrap.cc:654] T 6ba8241644984394ada95b561e76467e P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:31.729350 2666 raft_consensus.cc:513] T abf6e3be60f74872b2308dc95bf9cd37 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:31.729861 2666 leader_election.cc:290] T abf6e3be60f74872b2308dc95bf9cd37 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:31.730118 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3eba54e5ddd14c518f4cbd539268ae37" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:31.730506 858 leader_election.cc:336] T 3eba54e5ddd14c518f4cbd539268ae37 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:31.730581 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c6f7c525f59b49938b697de98d4b5835" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:31.730638 858 leader_election.cc:336] T c6f7c525f59b49938b697de98d4b5835 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:31.732131 859 leader_election.cc:343] T 3eba54e5ddd14c518f4cbd539268ae37 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
W20250901 01:52:31.732172 992 leader_election.cc:336] T abf6e3be60f74872b2308dc95bf9cd37 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:31.732234 859 leader_election.cc:304] T 3eba54e5ddd14c518f4cbd539268ae37 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:31.732606 859 leader_election.cc:343] T c6f7c525f59b49938b697de98d4b5835 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:31.732692 859 leader_election.cc:304] T c6f7c525f59b49938b697de98d4b5835 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:31.733034 2357 raft_consensus.cc:2747] T 3eba54e5ddd14c518f4cbd539268ae37 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:31.733170 2357 raft_consensus.cc:2747] T c6f7c525f59b49938b697de98d4b5835 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:31.733449 992 leader_election.cc:336] T abf6e3be60f74872b2308dc95bf9cd37 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:31.736593 921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abf6e3be60f74872b2308dc95bf9cd37" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:31.736896 921 raft_consensus.cc:3058] T abf6e3be60f74872b2308dc95bf9cd37 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:31.738128 2608 raft_consensus.cc:491] T b4278a013ae84082bc7a8497b04ac147 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:31.738238 2608 raft_consensus.cc:513] T b4278a013ae84082bc7a8497b04ac147 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:31.738765 2608 leader_election.cc:290] T b4278a013ae84082bc7a8497b04ac147 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:31.739475 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b4278a013ae84082bc7a8497b04ac147" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:31.740027 921 raft_consensus.cc:2466] T abf6e3be60f74872b2308dc95bf9cd37 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
W20250901 01:52:31.740351 859 leader_election.cc:343] T b4278a013ae84082bc7a8497b04ac147 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
W20250901 01:52:31.740481 858 leader_election.cc:336] T b4278a013ae84082bc7a8497b04ac147 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:31.740621 858 leader_election.cc:304] T b4278a013ae84082bc7a8497b04ac147 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:31.740662 992 leader_election.cc:304] T abf6e3be60f74872b2308dc95bf9cd37 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:31.740850 2608 raft_consensus.cc:2747] T b4278a013ae84082bc7a8497b04ac147 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:31.740999 2666 raft_consensus.cc:2802] T abf6e3be60f74872b2308dc95bf9cd37 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:31.741384 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 500e3c33598b45a4afa983da7cb0845a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:26 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:31.741492 2666 raft_consensus.cc:695] T abf6e3be60f74872b2308dc95bf9cd37 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:31.741838 2666 consensus_queue.cc:237] T abf6e3be60f74872b2308dc95bf9cd37 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:31.742172 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 7442c6ff699141e1aff208ebdc9d192a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:26 UTC) from {username='slave'} at 127.0.0.1:34472
W20250901 01:52:31.743608 1491 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:54603 (request call id 185) took 1450 ms (client timeout 1000 ms). Trace:
I20250901 01:52:31.743662 2113 ts_tablet_manager.cc:1918] T 31614db4c53a4f3c96fd190d326ce601 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:31.743752 2113 log.cc:1199] T 31614db4c53a4f3c96fd190d326ce601 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/31614db4c53a4f3c96fd190d326ce601
I20250901 01:52:31.744062 2113 ts_tablet_manager.cc:1939] T 31614db4c53a4f3c96fd190d326ce601 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
W20250901 01:52:31.743865 1491 rpcz_store.cc:269] 0901 01:52:30.293380 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.293912 (+ 532us) service_pool.cc:225] Handling call
0901 01:52:30.393654 (+ 99742us) spinlock_profiling.cc:243] Waited 57.5 ms on lock 0x616000040c54. stack: 00007f0d64ec57d6 00007f0d64ec5590 00007f0d63eb83ad 0000000000972d94 0000000000972d5d 00007f0d73c9014d 00007f0d73c90243 00007f0d7094f941 00007f0d784daa5d 00007f0d7865e6d8 00007f0d6f498c86 00007f0d6f4989b5 00007f0d6bc732ce 00007f0d6bc721ae 00007f0d6bc783e5 00007f0d6bc78d0f
0901 01:52:31.491711 (+1098057us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:31.491723 (+ 12us) write_op.cc:454] Released schema lock
0901 01:52:31.743537 (+251814us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0901 01:52:30.630362 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:30.630454 (+ 92us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:30.630461 (+ 7us) write_op.cc:435] Acquired schema lock
0901 01:52:30.630464 (+ 3us) tablet.cc:660] Decoding operations
0901 01:52:30.630565 (+ 101us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:30.630593 (+ 28us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:30.630599 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0901 01:52:30.630683 (+ 84us) tablet.cc:699] Row locks acquired
0901 01:52:30.630687 (+ 4us) write_op.cc:260] PREPARE: finished
0901 01:52:30.630723 (+ 36us) write_op.cc:270] Start()
0901 01:52:30.630758 (+ 35us) write_op.cc:276] Timestamp: P: 1756691550630699 usec, L: 0
0901 01:52:30.630761 (+ 3us) op_driver.cc:348] REPLICATION: starting
0901 01:52:30.630953 (+ 192us) log.cc:844] Serialized 527 byte log entry
0901 01:52:30.640803 (+ 9850us) op_driver.cc:464] REPLICATION: finished
0901 01:52:30.640930 (+ 127us) write_op.cc:301] APPLY: starting
0901 01:52:30.640947 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:30.640988 (+ 41us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:30.656987 (+ 15999us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:30.657069 (+ 82us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:30.657115 (+ 46us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0901 01:52:30.657122 (+ 7us) write_op.cc:312] APPLY: finished
0901 01:52:30.657221 (+ 99us) log.cc:844] Serialized 23 byte log entry
0901 01:52:30.657304 (+ 83us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:30.657331 (+ 27us) write_op.cc:454] Released schema lock
0901 01:52:30.657340 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0901 01:52:30.657423 (+ 83us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:30.657427 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0901 01:52:31.004971 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:31.005072 (+ 101us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:31.005080 (+ 8us) write_op.cc:435] Acquired schema lock
0901 01:52:31.005085 (+ 5us) tablet.cc:660] Decoding operations
0901 01:52:31.005223 (+ 138us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:31.005240 (+ 17us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:31.005246 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0901 01:52:31.005304 (+ 58us) tablet.cc:699] Row locks acquired
0901 01:52:31.005307 (+ 3us) write_op.cc:260] PREPARE: finished
0901 01:52:31.005346 (+ 39us) write_op.cc:270] Start()
0901 01:52:31.005408 (+ 62us) write_op.cc:276] Timestamp: P: 1756691551005282 usec, L: 0
0901 01:52:31.005413 (+ 5us) op_driver.cc:348] REPLICATION: starting
0901 01:52:31.005610 (+ 197us) log.cc:844] Serialized 356 byte log entry
0901 01:52:31.011643 (+ 6033us) op_driver.cc:464] REPLICATION: finished
0901 01:52:31.012600 (+ 957us) write_op.cc:301] APPLY: starting
0901 01:52:31.012617 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:31.012661 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:31.012702 (+ 41us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:31.012763 (+ 61us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:31.012816 (+ 53us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0901 01:52:31.012825 (+ 9us) write_op.cc:312] APPLY: finished
0901 01:52:31.012909 (+ 84us) log.cc:844] Serialized 23 byte log entry
0901 01:52:31.012974 (+ 65us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:31.012994 (+ 20us) write_op.cc:454] Released schema lock
0901 01:52:31.013001 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0901 01:52:31.013071 (+ 70us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:31.013074 (+ 3us) write_op.cc:454] Released schema lock
Related trace 'op':
0901 01:52:31.061641 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:31.061734 (+ 93us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:31.061739 (+ 5us) write_op.cc:435] Acquired schema lock
0901 01:52:31.061741 (+ 2us) tablet.cc:660] Decoding operations
0901 01:52:31.061833 (+ 92us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:31.061843 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:31.061848 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0901 01:52:31.061895 (+ 47us) tablet.cc:699] Row locks acquired
0901 01:52:31.061898 (+ 3us) write_op.cc:260] PREPARE: finished
0901 01:52:31.061925 (+ 27us) write_op.cc:270] Start()
0901 01:52:31.061950 (+ 25us) write_op.cc:276] Timestamp: P: 1756691551061861 usec, L: 0
0901 01:52:31.061954 (+ 4us) op_driver.cc:348] REPLICATION: starting
0901 01:52:31.062086 (+ 132us) log.cc:844] Serialized 356 byte log entry
0901 01:52:31.069858 (+ 7772us) op_driver.cc:464] REPLICATION: finished
0901 01:52:31.069955 (+ 97us) write_op.cc:301] APPLY: starting
0901 01:52:31.069969 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:31.070005 (+ 36us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:31.070031 (+ 26us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:31.070080 (+ 49us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:31.070111 (+ 31us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0901 01:52:31.070118 (+ 7us) write_op.cc:312] APPLY: finished
0901 01:52:31.070214 (+ 96us) log.cc:844] Serialized 23 byte log entry
0901 01:52:31.070271 (+ 57us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:31.070292 (+ 21us) write_op.cc:454] Released schema lock
0901 01:52:31.070300 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0901 01:52:31.070376 (+ 76us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:31.070381 (+ 5us) write_op.cc:454] Released schema lock
Related trace 'op':
0901 01:52:31.331010 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:31.331128 (+ 118us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:31.331134 (+ 6us) write_op.cc:435] Acquired schema lock
0901 01:52:31.331137 (+ 3us) tablet.cc:660] Decoding operations
0901 01:52:31.331237 (+ 100us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:31.331251 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:31.331258 (+ 7us) tablet.cc:683] Acquiring locks for 1 operations
0901 01:52:31.331325 (+ 67us) tablet.cc:699] Row locks acquired
0901 01:52:31.331329 (+ 4us) write_op.cc:260] PREPARE: finished
0901 01:52:31.331367 (+ 38us) write_op.cc:270] Start()
0901 01:52:31.331402 (+ 35us) write_op.cc:276] Timestamp: P: 1756691551331289 usec, L: 0
0901 01:52:31.331408 (+ 6us) op_driver.cc:348] REPLICATION: starting
0901 01:52:31.331572 (+ 164us) log.cc:844] Serialized 356 byte log entry
0901 01:52:31.335513 (+ 3941us) op_driver.cc:464] REPLICATION: finished
0901 01:52:31.335615 (+ 102us) write_op.cc:301] APPLY: starting
0901 01:52:31.335629 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:31.335663 (+ 34us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:31.335688 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:31.335740 (+ 52us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:31.335784 (+ 44us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0901 01:52:31.335792 (+ 8us) write_op.cc:312] APPLY: finished
0901 01:52:31.335873 (+ 81us) log.cc:844] Serialized 23 byte log entry
0901 01:52:31.335932 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:31.335949 (+ 17us) write_op.cc:454] Released schema lock
0901 01:52:31.335955 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0901 01:52:31.336021 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:31.336025 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0901 01:52:31.429552 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:31.429692 (+ 140us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:31.429702 (+ 10us) write_op.cc:435] Acquired schema lock
0901 01:52:31.429707 (+ 5us) tablet.cc:660] Decoding operations
0901 01:52:31.429865 (+ 158us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:31.429885 (+ 20us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:31.429894 (+ 9us) tablet.cc:683] Acquiring locks for 1 operations
0901 01:52:31.429981 (+ 87us) tablet.cc:699] Row locks acquired
0901 01:52:31.429985 (+ 4us) write_op.cc:260] PREPARE: finished
0901 01:52:31.430028 (+ 43us) write_op.cc:270] Start()
0901 01:52:31.430072 (+ 44us) write_op.cc:276] Timestamp: P: 1756691551429938 usec, L: 0
0901 01:52:31.430077 (+ 5us) op_driver.cc:348] REPLICATION: starting
0901 01:52:31.430305 (+ 228us) log.cc:844] Serialized 356 byte log entry
0901 01:52:31.434802 (+ 4497us) op_driver.cc:464] REPLICATION: finished
0901 01:52:31.434902 (+ 100us) write_op.cc:301] APPLY: starting
0901 01:52:31.434916 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:31.434957 (+ 41us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:31.434989 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:31.435043 (+ 54us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:31.435088 (+ 45us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0901 01:52:31.435094 (+ 6us) write_op.cc:312] APPLY: finished
0901 01:52:31.435167 (+ 73us) log.cc:844] Serialized 23 byte log entry
0901 01:52:31.435221 (+ 54us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:31.435239 (+ 18us) write_op.cc:454] Released schema lock
0901 01:52:31.435244 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":94101,"reported_tablets":140,"spinlock_wait_cycles":132276096,"child_traces":[["op",{"apply.queue_time_us":68,"apply.run_cpu_time_us":619,"apply.run_wall_time_us":16482,"num_ops":1,"prepare.queue_time_us":47,"prepare.run_cpu_time_us":700,"prepare.run_wall_time_us":701,"raft.queue_time_us":2155,"raft.run_cpu_time_us":468,"raft.run_wall_time_us":469,"replication_time_us":9624,"spinlock_wait_cycles":172288}],["op",{"apply.queue_time_us":883,"apply.run_cpu_time_us":469,"apply.run_wall_time_us":469,"num_ops":1,"prepare.queue_time_us":681,"prepare.run_cpu_time_us":752,"prepare.run_wall_time_us":754,"raft.queue_time_us":1488,"raft.run_cpu_time_us":622,"raft.run_wall_time_us":622,"replication_time_us":6193}],["op",{"apply.queue_time_us":53,"apply.run_cpu_time_us":407,"apply.run_wall_time_us":407,"num_ops":1,"prepare.queue_time_us":80,"prepare.run_cpu_time_us":525,"prepare.run_wall_time_us":526,"raft.queue_time_us":52,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":174,"replication_time_us":7875}],["op",{"apply.queue_time_us":57,"apply.run_cpu_time_us":397,"apply.run_wall_time_us":396,"num_ops":1,"prepare.queue_time_us":63,"prepare.run_cpu_time_us":664,"prepare.run_wall_time_us":664,"raft.queue_time_us":700,"raft.run_cpu_time_us":231,"raft.run_wall_time_us":231,"replication_time_us":4076}],["op",{"apply.queue_time_us":55,"apply.run_cpu_time_us":389,"apply.run_wall_time_us":389,"num_ops":1,"prepare.queue_time_us":73,"prepare.run_cpu_time_us":885,"prepare.run_wall_time_us":885,"raft.queue_time_us":1343,"raft.run_cpu_time_us":549,"raft.run_wall_time_us":548,"replication_time_us":4695}]]}
I20250901 01:52:31.751848 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 168) took 5126 ms. Trace:
I20250901 01:52:31.751966 2113 rpcz_store.cc:276] 0901 01:52:26.624946 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.625000 (+ 54us) service_pool.cc:225] Handling call
0901 01:52:31.714932 (+5089932us) ts_tablet_manager.cc:1145] Deleting tablet 31614db4c53a4f3c96fd190d326ce601
0901 01:52:31.714943 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:31.737568 (+ 22625us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:31.743556 (+ 5988us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:31.751826 (+ 8270us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5089745}
W20250901 01:52:31.684957 1465 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet d000ba05e12843b4925e622ab55471c0 on TS 0cf599f74322478d91475df234e58965 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:31.752540 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 31614db4c53a4f3c96fd190d326ce601 (table default.Table_e59cac5d3f0a4dd6b9846e7449b60005 [id=fb3be765c8e84e8cb4823153f82b374c]) successfully deleted
I20250901 01:52:31.760604 2113 tablet_replica.cc:331] T 4902062977ea42149e3466e999117322 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:31.760841 2113 raft_consensus.cc:2241] T 4902062977ea42149e3466e999117322 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:31.761865 2113 raft_consensus.cc:2270] T 4902062977ea42149e3466e999117322 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:31.762470 2113 ts_tablet_manager.cc:1905] T 4902062977ea42149e3466e999117322 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:31.765937 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2ec581c7427d4280899977ea0cdf4739"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.767467 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_67356043623247d09fdb786db5c17802"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.768729 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_26409f0b551f4628b9aba78cd4f59dd1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.769784 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b5c9fc67f5eb44558259812659af5bfe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.770689 1488 master_service.cc:948] ReplaceTablet: received request to replace tablet b2e8b2cd10154de08ddd30c214592a4a from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:31.771251 1486 master_service.cc:948] ReplaceTablet: received request to replace tablet 3a2bc5ad1cd243daa3c92fceca8ba414 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:31.771917 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ac692db491d74858bb61a07f49e4f987"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.773192 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2f5d8b61966e4d05ab2941ee00485859"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.774446 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_fee269db08be444f92622237ff98d050"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.777043 1151 tablet_bootstrap.cc:492] T b85d64e02cc646e380581ac9f59dee9e P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:31.777223 1151 ts_tablet_manager.cc:1397] T b85d64e02cc646e380581ac9f59dee9e P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.085s user 0.012s sys 0.006s
I20250901 01:52:31.777976 1151 raft_consensus.cc:357] T b85d64e02cc646e380581ac9f59dee9e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:31.778215 1151 raft_consensus.cc:383] T b85d64e02cc646e380581ac9f59dee9e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:31.778301 1151 raft_consensus.cc:738] T b85d64e02cc646e380581ac9f59dee9e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:31.778556 1151 consensus_queue.cc:260] T b85d64e02cc646e380581ac9f59dee9e P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:31.780972 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_0bda43cebe034f768b7ad10cec2693ca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.783349 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3585a07ba54845b593db8450947a83f2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.779654 2113 ts_tablet_manager.cc:1918] T 4902062977ea42149e3466e999117322 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:31.784312 2113 log.cc:1199] T 4902062977ea42149e3466e999117322 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/4902062977ea42149e3466e999117322
I20250901 01:52:31.784343 1151 ts_tablet_manager.cc:1428] T b85d64e02cc646e380581ac9f59dee9e P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250901 01:52:31.784638 1151 tablet_bootstrap.cc:492] T becbaae4fd7e44faa679e9479465a7c4 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:31.784708 2113 ts_tablet_manager.cc:1939] T 4902062977ea42149e3466e999117322 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:31.785293 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9cbd931bffce451c81c48f962337ca94"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:31.786725 1151 tablet_bootstrap.cc:654] T becbaae4fd7e44faa679e9479465a7c4 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:31.788118 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 169) took 5162 ms. Trace:
I20250901 01:52:31.788208 2113 rpcz_store.cc:276] 0901 01:52:26.625725 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.625767 (+ 42us) service_pool.cc:225] Handling call
0901 01:52:31.760546 (+5134779us) ts_tablet_manager.cc:1145] Deleting tablet 4902062977ea42149e3466e999117322
0901 01:52:31.760560 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:31.769580 (+ 9020us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:31.779539 (+ 9959us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:31.788097 (+ 8558us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5134646}
I20250901 01:52:31.788676 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 4902062977ea42149e3466e999117322 (table default.Table_fb5ee5a5a9514ddaac740c194a820fd2 [id=48b7b856dca841eba100ba8acf02b7e2]) successfully deleted
I20250901 01:52:31.790867 2113 tablet_replica.cc:331] T 48a211e174054866b030cdfb0cf947f4 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:31.792538 1150 tablet_bootstrap.cc:492] T 6ba8241644984394ada95b561e76467e P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:31.791105 2113 raft_consensus.cc:2241] T 48a211e174054866b030cdfb0cf947f4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:31.793044 2113 raft_consensus.cc:2270] T 48a211e174054866b030cdfb0cf947f4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:31.793603 2113 ts_tablet_manager.cc:1905] T 48a211e174054866b030cdfb0cf947f4 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:31.796638 1150 ts_tablet_manager.cc:1397] T 6ba8241644984394ada95b561e76467e P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.069s user 0.006s sys 0.012s
I20250901 01:52:31.798214 1150 raft_consensus.cc:357] T 6ba8241644984394ada95b561e76467e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:31.798542 1150 raft_consensus.cc:383] T 6ba8241644984394ada95b561e76467e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:31.798641 1150 raft_consensus.cc:738] T 6ba8241644984394ada95b561e76467e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:31.799036 1150 consensus_queue.cc:260] T 6ba8241644984394ada95b561e76467e P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:31.800068 1150 ts_tablet_manager.cc:1428] T 6ba8241644984394ada95b561e76467e P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250901 01:52:31.808005 1483 master_service.cc:948] ReplaceTablet: received request to replace tablet 75a703c8e20a4a2fa0ab6ec58d41ef92 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:31.808974 1490 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_b3e694397be14321a668e46c09c51a71" } modify_external_catalogs: true
I20250901 01:52:31.814008 2118 ts_tablet_manager.cc:1918] T f23df6f253a943fa8e146bb8c811ef51 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:31.814548 2118 log.cc:1199] T f23df6f253a943fa8e146bb8c811ef51 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/f23df6f253a943fa8e146bb8c811ef51
I20250901 01:52:31.815148 2666 consensus_queue.cc:1035] T 4cde5f4f02044b7bbd7d43b49aac4ae8 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250901 01:52:31.815538 992 consensus_peers.cc:489] T 4cde5f4f02044b7bbd7d43b49aac4ae8 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:31.818466 1150 tablet_bootstrap.cc:492] T c83ce95fd6fa41478de0e2168be201b1 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:31.827937 2113 ts_tablet_manager.cc:1918] T 48a211e174054866b030cdfb0cf947f4 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:31.829321 1150 tablet_bootstrap.cc:654] T c83ce95fd6fa41478de0e2168be201b1 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:31.833562 2608 raft_consensus.cc:491] T 268439ff8d3a4f7a89f3d6395f2079c5 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:31.833711 2608 raft_consensus.cc:513] T 268439ff8d3a4f7a89f3d6395f2079c5 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:31.834336 2608 leader_election.cc:290] T 268439ff8d3a4f7a89f3d6395f2079c5 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:31.834995 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "268439ff8d3a4f7a89f3d6395f2079c5" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:31.835582 859 leader_election.cc:343] T 268439ff8d3a4f7a89f3d6395f2079c5 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
W20250901 01:52:31.838425 858 leader_election.cc:336] T 268439ff8d3a4f7a89f3d6395f2079c5 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:31.838568 858 leader_election.cc:304] T 268439ff8d3a4f7a89f3d6395f2079c5 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:31.838830 2608 raft_consensus.cc:2747] T 268439ff8d3a4f7a89f3d6395f2079c5 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:31.839591 2113 log.cc:1199] T 48a211e174054866b030cdfb0cf947f4 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/48a211e174054866b030cdfb0cf947f4
I20250901 01:52:31.839962 2113 ts_tablet_manager.cc:1939] T 48a211e174054866b030cdfb0cf947f4 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:31.840804 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 170) took 5214 ms. Trace:
I20250901 01:52:31.846745 2113 rpcz_store.cc:276] 0901 01:52:26.626257 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.626298 (+ 41us) service_pool.cc:225] Handling call
0901 01:52:31.790820 (+5164522us) ts_tablet_manager.cc:1145] Deleting tablet 48a211e174054866b030cdfb0cf947f4
0901 01:52:31.790830 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:31.821644 (+ 30814us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:31.827830 (+ 6186us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:31.840784 (+ 12954us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5164389}
I20250901 01:52:31.847254 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 48a211e174054866b030cdfb0cf947f4 (table default.Table_7c23fc28bbae4efe982c240b87f40956 [id=65d035b79f004c3d9ac3b0cd6a515d45]) successfully deleted
I20250901 01:52:31.848245 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 171) took 5221 ms. Trace:
I20250901 01:52:31.848343 2113 rpcz_store.cc:276] 0901 01:52:26.626952 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.626990 (+ 38us) service_pool.cc:225] Handling call
0901 01:52:31.848133 (+5221143us) ts_tablet_manager.cc:1145] Deleting tablet 533d14f5903e49599a34ea353ffd8c00
0901 01:52:31.848144 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:31.848226 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5221023}
W20250901 01:52:31.848835 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 533d14f5903e49599a34ea353ffd8c00 because the tablet was not found. No further retry: Not found: Tablet not found: 533d14f5903e49599a34ea353ffd8c00
I20250901 01:52:31.850171 2113 tablet_replica.cc:331] T 1a60c10e42104502ac2d3b80b8126ceb P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:31.850368 2113 raft_consensus.cc:2241] T 1a60c10e42104502ac2d3b80b8126ceb P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:31.850553 2113 raft_consensus.cc:2270] T 1a60c10e42104502ac2d3b80b8126ceb P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:31.852867 2113 ts_tablet_manager.cc:1905] T 1a60c10e42104502ac2d3b80b8126ceb P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:31.853857 2666 raft_consensus.cc:491] T 74b709cb5ec64006b2f12bead8217d80 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:31.854034 2666 raft_consensus.cc:513] T 74b709cb5ec64006b2f12bead8217d80 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:31.854826 2666 leader_election.cc:290] T 74b709cb5ec64006b2f12bead8217d80 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:31.856086 992 leader_election.cc:343] T 74b709cb5ec64006b2f12bead8217d80 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 74b709cb5ec64006b2f12bead8217d80
W20250901 01:52:31.856717 992 leader_election.cc:336] T 74b709cb5ec64006b2f12bead8217d80 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:31.856832 992 leader_election.cc:304] T 74b709cb5ec64006b2f12bead8217d80 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:31.857069 2666 raft_consensus.cc:2747] T 74b709cb5ec64006b2f12bead8217d80 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:31.877606 2666 consensus_queue.cc:1035] T 3239e58531d445fa80165cf678bc277c P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:31.903613 2113 ts_tablet_manager.cc:1918] T 1a60c10e42104502ac2d3b80b8126ceb P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:31.903779 2113 log.cc:1199] T 1a60c10e42104502ac2d3b80b8126ceb P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/1a60c10e42104502ac2d3b80b8126ceb
I20250901 01:52:31.904115 2113 ts_tablet_manager.cc:1939] T 1a60c10e42104502ac2d3b80b8126ceb P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:31.909672 1482 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: default.Table_a920a3ffbabf4d83a6563c3423cde12a [id=6fd739b0e725424ca844dbb652ea8028] alter complete (version 3)
I20250901 01:52:31.918799 2666 raft_consensus.cc:491] T d1f2d98712ff4ea6aec665512de068bd P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:31.919023 2666 raft_consensus.cc:513] T d1f2d98712ff4ea6aec665512de068bd P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:31.919839 2666 leader_election.cc:290] T d1f2d98712ff4ea6aec665512de068bd P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:31.920665 922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1f2d98712ff4ea6aec665512de068bd" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:31.920921 922 raft_consensus.cc:2466] T d1f2d98712ff4ea6aec665512de068bd P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:31.921543 992 leader_election.cc:304] T d1f2d98712ff4ea6aec665512de068bd P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:31.921905 2666 raft_consensus.cc:2802] T d1f2d98712ff4ea6aec665512de068bd P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:31.922013 2666 raft_consensus.cc:491] T d1f2d98712ff4ea6aec665512de068bd P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:31.922071 2666 raft_consensus.cc:3058] T d1f2d98712ff4ea6aec665512de068bd P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
W20250901 01:52:31.922279 992 leader_election.cc:336] T d1f2d98712ff4ea6aec665512de068bd P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:31.925892 2666 raft_consensus.cc:513] T d1f2d98712ff4ea6aec665512de068bd P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:31.926843 2666 leader_election.cc:290] T d1f2d98712ff4ea6aec665512de068bd P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:31.927482 922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1f2d98712ff4ea6aec665512de068bd" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:31.927700 922 raft_consensus.cc:3058] T d1f2d98712ff4ea6aec665512de068bd P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:31.930752 922 raft_consensus.cc:2466] T d1f2d98712ff4ea6aec665512de068bd P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:31.932060 992 leader_election.cc:304] T d1f2d98712ff4ea6aec665512de068bd P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:31.932608 2412 raft_consensus.cc:2802] T d1f2d98712ff4ea6aec665512de068bd P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:31.932744 2412 raft_consensus.cc:695] T d1f2d98712ff4ea6aec665512de068bd P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:31.933007 2412 consensus_queue.cc:237] T d1f2d98712ff4ea6aec665512de068bd P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:31.934880 992 leader_election.cc:336] T d1f2d98712ff4ea6aec665512de068bd P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:31.941215 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 172) took 5313 ms. Trace:
I20250901 01:52:31.949546 2118 ts_tablet_manager.cc:1939] T f23df6f253a943fa8e146bb8c811ef51 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:31.948799 2666 raft_consensus.cc:491] T 7a607fac57994001b2304e5d81c9decc P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:31.950899 2666 raft_consensus.cc:513] T 7a607fac57994001b2304e5d81c9decc P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:31.951630 2666 leader_election.cc:290] T 7a607fac57994001b2304e5d81c9decc P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:31.953523 992 leader_election.cc:343] T 7a607fac57994001b2304e5d81c9decc P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 7a607fac57994001b2304e5d81c9decc
W20250901 01:52:31.953689 992 leader_election.cc:336] T 7a607fac57994001b2304e5d81c9decc P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:31.953784 992 leader_election.cc:304] T 7a607fac57994001b2304e5d81c9decc P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:31.953989 2666 raft_consensus.cc:2747] T 7a607fac57994001b2304e5d81c9decc P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:31.955731 2113 rpcz_store.cc:276] 0901 01:52:26.627585 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.627624 (+ 39us) service_pool.cc:225] Handling call
0901 01:52:31.850115 (+5222491us) ts_tablet_manager.cc:1145] Deleting tablet 1a60c10e42104502ac2d3b80b8126ceb
0901 01:52:31.850124 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:31.874176 (+ 24052us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:31.903458 (+ 29282us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:31.941186 (+ 37728us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5222372}
I20250901 01:52:31.956295 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 1a60c10e42104502ac2d3b80b8126ceb (table default.Table_53e30205c5fd488eb3b48c685a159d04 [id=60837b865ee7415ea42a85a953d837a5]) successfully deleted
I20250901 01:52:31.956662 2113 tablet_replica.cc:331] T 56475ba491a24301a534304811f30140 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:31.956892 2113 raft_consensus.cc:2241] T 56475ba491a24301a534304811f30140 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:31.957068 2113 raft_consensus.cc:2270] T 56475ba491a24301a534304811f30140 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:31.957690 2113 ts_tablet_manager.cc:1905] T 56475ba491a24301a534304811f30140 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:31.964426 2608 raft_consensus.cc:491] T 8b9d79a25452410891c09fd3f7c4be6d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:31.964620 2608 raft_consensus.cc:513] T 8b9d79a25452410891c09fd3f7c4be6d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:31.965282 2608 leader_election.cc:290] T 8b9d79a25452410891c09fd3f7c4be6d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:31.967617 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b9d79a25452410891c09fd3f7c4be6d" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:31.968098 859 leader_election.cc:343] T 8b9d79a25452410891c09fd3f7c4be6d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
W20250901 01:52:31.970345 858 leader_election.cc:336] T 8b9d79a25452410891c09fd3f7c4be6d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:31.970458 858 leader_election.cc:304] T 8b9d79a25452410891c09fd3f7c4be6d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:31.971074 2608 raft_consensus.cc:2747] T 8b9d79a25452410891c09fd3f7c4be6d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:31.987387 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 147) took 4879 ms. Trace:
I20250901 01:52:31.987598 2118 rpcz_store.cc:276] 0901 01:52:27.107907 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:27.107956 (+ 49us) service_pool.cc:225] Handling call
0901 01:52:31.660919 (+4552963us) ts_tablet_manager.cc:1145] Deleting tablet f23df6f253a943fa8e146bb8c811ef51
0901 01:52:31.660933 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:31.727419 (+ 66486us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:31.813637 (+ 86218us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:31.987362 (+173725us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4525728}
I20250901 01:52:31.988533 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet f23df6f253a943fa8e146bb8c811ef51 (table default.Table_59039012009944fdbe0179002ea44633 [id=218947d08fb24732b5d0375d4af973e9]) successfully deleted
I20250901 01:52:31.990324 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 148) took 4805 ms. Trace:
I20250901 01:52:32.031436 1761 raft_consensus.cc:491] T b164b4a67a9143ceaff69b0ce49d63fe P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:32.031682 1761 raft_consensus.cc:513] T b164b4a67a9143ceaff69b0ce49d63fe P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:32.032709 1761 leader_election.cc:290] T b164b4a67a9143ceaff69b0ce49d63fe P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:32.040627 992 leader_election.cc:336] T b164b4a67a9143ceaff69b0ce49d63fe P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:32.044900 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b164b4a67a9143ceaff69b0ce49d63fe" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:32.045223 913 raft_consensus.cc:2466] T b164b4a67a9143ceaff69b0ce49d63fe P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:32.045796 992 leader_election.cc:304] T b164b4a67a9143ceaff69b0ce49d63fe P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:32.046165 2613 raft_consensus.cc:2802] T b164b4a67a9143ceaff69b0ce49d63fe P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:32.046258 2613 raft_consensus.cc:491] T b164b4a67a9143ceaff69b0ce49d63fe P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:32.046335 2613 raft_consensus.cc:3058] T b164b4a67a9143ceaff69b0ce49d63fe P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:32.049656 2613 raft_consensus.cc:513] T b164b4a67a9143ceaff69b0ce49d63fe P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:32.050403 2613 leader_election.cc:290] T b164b4a67a9143ceaff69b0ce49d63fe P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:32.051378 922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b164b4a67a9143ceaff69b0ce49d63fe" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:32.051586 922 raft_consensus.cc:3058] T b164b4a67a9143ceaff69b0ce49d63fe P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:32.054646 922 raft_consensus.cc:2466] T b164b4a67a9143ceaff69b0ce49d63fe P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:32.055557 992 leader_election.cc:304] T b164b4a67a9143ceaff69b0ce49d63fe P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
W20250901 01:52:32.056823 992 leader_election.cc:336] T b164b4a67a9143ceaff69b0ce49d63fe P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:32.057080 2666 raft_consensus.cc:2802] T b164b4a67a9143ceaff69b0ce49d63fe P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:32.057257 2666 raft_consensus.cc:695] T b164b4a67a9143ceaff69b0ce49d63fe P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:32.057572 2666 consensus_queue.cc:237] T b164b4a67a9143ceaff69b0ce49d63fe P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:31.991757 2118 rpcz_store.cc:276] 0901 01:52:27.184826 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:27.184882 (+ 56us) service_pool.cc:225] Handling call
0901 01:52:31.989387 (+4804505us) ts_tablet_manager.cc:1145] Deleting tablet 48a211e174054866b030cdfb0cf947f4
0901 01:52:31.989466 (+ 79us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:31.990164 (+ 698us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4804256}
W20250901 01:52:32.059283 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 48a211e174054866b030cdfb0cf947f4 because the tablet was not found. No further retry: Not found: Tablet not found: 48a211e174054866b030cdfb0cf947f4
I20250901 01:52:32.059844 1150 tablet_bootstrap.cc:492] T c83ce95fd6fa41478de0e2168be201b1 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:32.059978 1150 ts_tablet_manager.cc:1397] T c83ce95fd6fa41478de0e2168be201b1 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.242s user 0.009s sys 0.013s
I20250901 01:52:32.060868 1150 raft_consensus.cc:357] T c83ce95fd6fa41478de0e2168be201b1 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.061064 1150 raft_consensus.cc:383] T c83ce95fd6fa41478de0e2168be201b1 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:32.061120 1150 raft_consensus.cc:738] T c83ce95fd6fa41478de0e2168be201b1 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:32.061363 1150 consensus_queue.cc:260] T c83ce95fd6fa41478de0e2168be201b1 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.062077 1150 ts_tablet_manager.cc:1428] T c83ce95fd6fa41478de0e2168be201b1 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250901 01:52:32.062378 1150 tablet_bootstrap.cc:492] T 0cba5329ab31440999faec599d704633 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:32.064308 1150 tablet_bootstrap.cc:654] T 0cba5329ab31440999faec599d704633 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:32.083021 2118 tablet_replica.cc:331] T b85d64e02cc646e380581ac9f59dee9e P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:32.083616 2118 raft_consensus.cc:2241] T b85d64e02cc646e380581ac9f59dee9e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:32.084573 2118 raft_consensus.cc:2270] T b85d64e02cc646e380581ac9f59dee9e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:32.086802 2113 ts_tablet_manager.cc:1918] T 56475ba491a24301a534304811f30140 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:32.089740 2118 ts_tablet_manager.cc:1905] T b85d64e02cc646e380581ac9f59dee9e P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:32.090286 2113 log.cc:1199] T 56475ba491a24301a534304811f30140 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/56475ba491a24301a534304811f30140
I20250901 01:52:32.090694 2113 ts_tablet_manager.cc:1939] T 56475ba491a24301a534304811f30140 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:32.092679 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 173) took 5464 ms. Trace:
I20250901 01:52:32.095255 1151 tablet_bootstrap.cc:492] T becbaae4fd7e44faa679e9479465a7c4 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
W20250901 01:52:32.096452 992 consensus_peers.cc:489] T e6c10426f60f43e8bde69d9b1b9c8eb6 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250901 01:52:32.095847 2113 rpcz_store.cc:276] 0901 01:52:26.628160 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.628199 (+ 39us) service_pool.cc:225] Handling call
0901 01:52:31.956599 (+5328400us) ts_tablet_manager.cc:1145] Deleting tablet 56475ba491a24301a534304811f30140
0901 01:52:31.956613 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:32.006047 (+ 49434us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:32.086673 (+ 80626us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:32.092658 (+ 5985us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5328267}
I20250901 01:52:32.098049 1151 ts_tablet_manager.cc:1397] T becbaae4fd7e44faa679e9479465a7c4 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.313s user 0.115s sys 0.163s
I20250901 01:52:32.099857 2113 tablet_replica.cc:331] T b81a12b38ac64efabd44aeb192d181a0 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:32.100034 2113 raft_consensus.cc:2241] T b81a12b38ac64efabd44aeb192d181a0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:32.100157 2113 raft_consensus.cc:2270] T b81a12b38ac64efabd44aeb192d181a0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:32.100605 2113 ts_tablet_manager.cc:1905] T b81a12b38ac64efabd44aeb192d181a0 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:32.104847 1151 raft_consensus.cc:357] T becbaae4fd7e44faa679e9479465a7c4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.105015 1151 raft_consensus.cc:383] T becbaae4fd7e44faa679e9479465a7c4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:32.105067 1151 raft_consensus.cc:738] T becbaae4fd7e44faa679e9479465a7c4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:32.106139 1151 consensus_queue.cc:260] T becbaae4fd7e44faa679e9479465a7c4 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.113530 1151 ts_tablet_manager.cc:1428] T becbaae4fd7e44faa679e9479465a7c4 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.014s user 0.001s sys 0.000s
I20250901 01:52:32.113822 1151 tablet_bootstrap.cc:492] T fa7fb88ded21432a8757f37ec3db5e71 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:32.115191 1151 tablet_bootstrap.cc:654] T fa7fb88ded21432a8757f37ec3db5e71 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:32.099143 1150 tablet_bootstrap.cc:492] T 0cba5329ab31440999faec599d704633 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:32.119323 2608 raft_consensus.cc:491] T 81ce448b148447ddbc41875d863e0518 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:32.127010 2608 raft_consensus.cc:513] T 81ce448b148447ddbc41875d863e0518 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.127710 2608 leader_election.cc:290] T 81ce448b148447ddbc41875d863e0518 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:32.128655 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "81ce448b148447ddbc41875d863e0518" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:32.129361 859 leader_election.cc:343] T 81ce448b148447ddbc41875d863e0518 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
W20250901 01:52:32.129844 858 leader_election.cc:336] T 81ce448b148447ddbc41875d863e0518 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:32.129973 858 leader_election.cc:304] T 81ce448b148447ddbc41875d863e0518 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:32.130223 2608 raft_consensus.cc:2747] T 81ce448b148447ddbc41875d863e0518 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:32.130443 1150 ts_tablet_manager.cc:1397] T 0cba5329ab31440999faec599d704633 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.068s user 0.003s sys 0.000s
I20250901 01:52:32.131304 1150 raft_consensus.cc:357] T 0cba5329ab31440999faec599d704633 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.131536 1150 raft_consensus.cc:383] T 0cba5329ab31440999faec599d704633 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:32.131609 1150 raft_consensus.cc:738] T 0cba5329ab31440999faec599d704633 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:32.131671 2113 ts_tablet_manager.cc:1918] T b81a12b38ac64efabd44aeb192d181a0 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:32.131759 2113 log.cc:1199] T b81a12b38ac64efabd44aeb192d181a0 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/b81a12b38ac64efabd44aeb192d181a0
I20250901 01:52:32.131937 1150 consensus_queue.cc:260] T 0cba5329ab31440999faec599d704633 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.132683 1150 ts_tablet_manager.cc:1428] T 0cba5329ab31440999faec599d704633 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250901 01:52:32.133005 1150 tablet_bootstrap.cc:492] T 3dae00aa38204ee3ac77325eb334be42 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:32.134943 1150 tablet_bootstrap.cc:654] T 3dae00aa38204ee3ac77325eb334be42 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:32.137070 2113 ts_tablet_manager.cc:1939] T b81a12b38ac64efabd44aeb192d181a0 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:32.138432 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 174) took 5509 ms. Trace:
I20250901 01:52:32.139058 2113 rpcz_store.cc:276] 0901 01:52:26.629134 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.629176 (+ 42us) service_pool.cc:225] Handling call
0901 01:52:32.099811 (+5470635us) ts_tablet_manager.cc:1145] Deleting tablet b81a12b38ac64efabd44aeb192d181a0
0901 01:52:32.099820 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:32.125871 (+ 26051us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:32.131574 (+ 5703us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:32.138411 (+ 6837us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5461803}
I20250901 01:52:32.139726 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet b81a12b38ac64efabd44aeb192d181a0 (table default.Table_e21873578fc14cc3b67e08478ec42192 [id=a404c32b3e8146f288ce7ea3a017ab96]) successfully deleted
I20250901 01:52:32.140069 2113 tablet_replica.cc:331] T a289a66b2dfa44dc8c8547ff83c1716e P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:32.140269 2113 raft_consensus.cc:2241] T a289a66b2dfa44dc8c8547ff83c1716e P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:32.140589 2113 raft_consensus.cc:2270] T a289a66b2dfa44dc8c8547ff83c1716e P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:32.141095 2113 ts_tablet_manager.cc:1905] T a289a66b2dfa44dc8c8547ff83c1716e P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:32.154670 2113 ts_tablet_manager.cc:1918] T a289a66b2dfa44dc8c8547ff83c1716e P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:32.161016 2608 raft_consensus.cc:491] T dabebc77f72e4bb19e264be2f496202a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:32.161166 2608 raft_consensus.cc:513] T dabebc77f72e4bb19e264be2f496202a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.161796 2608 leader_election.cc:290] T dabebc77f72e4bb19e264be2f496202a P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:32.162987 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dabebc77f72e4bb19e264be2f496202a" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:32.163790 858 leader_election.cc:336] T dabebc77f72e4bb19e264be2f496202a P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:32.164065 2113 log.cc:1199] T a289a66b2dfa44dc8c8547ff83c1716e P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/a289a66b2dfa44dc8c8547ff83c1716e
I20250901 01:52:32.164527 2113 ts_tablet_manager.cc:1939] T a289a66b2dfa44dc8c8547ff83c1716e P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
W20250901 01:52:32.165020 859 leader_election.cc:343] T dabebc77f72e4bb19e264be2f496202a P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:32.165114 859 leader_election.cc:304] T dabebc77f72e4bb19e264be2f496202a P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:32.165568 2608 raft_consensus.cc:2747] T dabebc77f72e4bb19e264be2f496202a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:32.165647 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 175) took 5519 ms. Trace:
I20250901 01:52:32.165727 2113 rpcz_store.cc:276] 0901 01:52:26.645887 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.645944 (+ 57us) service_pool.cc:225] Handling call
0901 01:52:32.140021 (+5494077us) ts_tablet_manager.cc:1145] Deleting tablet a289a66b2dfa44dc8c8547ff83c1716e
0901 01:52:32.140032 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:32.151688 (+ 11656us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:32.154570 (+ 2882us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:32.165629 (+ 11059us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5493917}
I20250901 01:52:32.165860 1490 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_b3e694397be14321a668e46c09c51a71" } modify_external_catalogs: true
I20250901 01:52:32.166919 2608 raft_consensus.cc:491] T eee34044c39d4013baf78d3705489626 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:32.167024 2608 raft_consensus.cc:513] T eee34044c39d4013baf78d3705489626 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.167152 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet a289a66b2dfa44dc8c8547ff83c1716e (table default.Table_4f2be95a1b824f06ba16e56abcdebc8a [id=434cfb105b894499b15bf360a8cffda4]) successfully deleted
I20250901 01:52:32.167521 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 176) took 5496 ms. Trace:
I20250901 01:52:32.167572 2608 leader_election.cc:290] T eee34044c39d4013baf78d3705489626 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:32.167595 2113 rpcz_store.cc:276] 0901 01:52:26.670683 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.670743 (+ 60us) service_pool.cc:225] Handling call
0901 01:52:32.167416 (+5496673us) ts_tablet_manager.cc:1145] Deleting tablet 36e10328c34242ebb4dbd8f294162696
0901 01:52:32.167426 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:32.167507 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5496493}
I20250901 01:52:32.168919 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 177) took 5497 ms. Trace:
I20250901 01:52:32.169845 1150 tablet_bootstrap.cc:492] T 3dae00aa38204ee3ac77325eb334be42 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:32.170327 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eee34044c39d4013baf78d3705489626" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:32.170138 2113 rpcz_store.cc:276] 0901 01:52:26.671597 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.671646 (+ 49us) service_pool.cc:225] Handling call
0901 01:52:32.168824 (+5497178us) ts_tablet_manager.cc:1145] Deleting tablet 3363fd67b1ad44849ecb7b4090b61a72
0901 01:52:32.168833 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:32.168905 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5497033}
W20250901 01:52:32.171097 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 3363fd67b1ad44849ecb7b4090b61a72 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3363fd67b1ad44849ecb7b4090b61a72 already in progress: creating tablet
I20250901 01:52:32.171440 2113 tablet_replica.cc:331] T 500e3c33598b45a4afa983da7cb0845a P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:32.171797 1489 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 219fe3882d884ee49b08671b547cac85
I20250901 01:52:32.171813 2113 raft_consensus.cc:2241] T 500e3c33598b45a4afa983da7cb0845a P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
W20250901 01:52:32.172080 1489 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 219fe3882d884ee49b08671b547cac85: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:32.172119 2113 raft_consensus.cc:2270] T 500e3c33598b45a4afa983da7cb0845a P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:32.173424 1483 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_7aa28ad9ee8c49859af40f9ccb61af33" } new_table_name: "default.Table_b80d3ae8523443b389b558591480b627" modify_external_catalogs: true
I20250901 01:52:32.171054 1488 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_7d149b0571834312849553b6e8c8f2ea" } new_table_name: "default.Table_946298d2a13646aeb4c189678909d1be" modify_external_catalogs: true
I20250901 01:52:32.176241 1489 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_938421db3cf74cf8b878089bac931818" } new_table_name: "default.Table_d1df9c0ae1a04b21a133e93cf4ae363a" modify_external_catalogs: true
I20250901 01:52:32.172848 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 219fe3882d884ee49b08671b547cac85 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:31 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:32.177181 2113 ts_tablet_manager.cc:1905] T 500e3c33598b45a4afa983da7cb0845a P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:32.186048 1488 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5ddbc7e6820247a987f4d54757d1191d consensus state has no leader
I20250901 01:52:32.186627 1488 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_dc856980ff4c4aafb92cf1fd4f29e694" } modify_external_catalogs: true
I20250901 01:52:32.186894 1488 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_dc856980ff4c4aafb92cf1fd4f29e694" } modify_external_catalogs: true
I20250901 01:52:32.193321 2113 ts_tablet_manager.cc:1918] T 500e3c33598b45a4afa983da7cb0845a P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:32.196686 2113 log.cc:1199] T 500e3c33598b45a4afa983da7cb0845a P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/500e3c33598b45a4afa983da7cb0845a
I20250901 01:52:32.197257 2113 ts_tablet_manager.cc:1939] T 500e3c33598b45a4afa983da7cb0845a P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:32.198413 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 178) took 5513 ms. Trace:
I20250901 01:52:32.199277 2113 rpcz_store.cc:276] 0901 01:52:26.684536 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.684593 (+ 57us) service_pool.cc:225] Handling call
0901 01:52:32.171397 (+5486804us) ts_tablet_manager.cc:1145] Deleting tablet 500e3c33598b45a4afa983da7cb0845a
0901 01:52:32.171407 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:32.180873 (+ 9466us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:32.193215 (+ 12342us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:32.198393 (+ 5178us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5486612}
I20250901 01:52:32.199965 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 500e3c33598b45a4afa983da7cb0845a (table default.Table_8ac222c93e36436c9a3ecb2c489f6d74 [id=44b75816b8b24af0a9530d4a12c2416e]) successfully deleted
W20250901 01:52:32.169987 858 leader_election.cc:336] T eee34044c39d4013baf78d3705489626 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:32.200490 2113 tablet_replica.cc:331] T afc002ae2486476b9208e1cf8215157c P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
W20250901 01:52:32.200472 859 leader_election.cc:343] T eee34044c39d4013baf78d3705489626 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:32.200618 858 leader_election.cc:304] T eee34044c39d4013baf78d3705489626 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:32.200949 2357 raft_consensus.cc:2747] T eee34044c39d4013baf78d3705489626 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:32.201028 2608 raft_consensus.cc:491] T 45f6e8ba99b74898a2922b36768c6c36 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:32.201165 2357 raft_consensus.cc:491] T 0cba5329ab31440999faec599d704633 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:32.201152 2608 raft_consensus.cc:513] T 45f6e8ba99b74898a2922b36768c6c36 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.201288 2357 raft_consensus.cc:513] T 0cba5329ab31440999faec599d704633 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.201843 2357 leader_election.cc:290] T 0cba5329ab31440999faec599d704633 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:32.201918 2608 leader_election.cc:290] T 45f6e8ba99b74898a2922b36768c6c36 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:32.203100 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 219fe3882d884ee49b08671b547cac85 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:31 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:32.203079 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cba5329ab31440999faec599d704633" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:32.203497 2113 raft_consensus.cc:2241] T afc002ae2486476b9208e1cf8215157c P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:32.203691 2113 raft_consensus.cc:2270] T afc002ae2486476b9208e1cf8215157c P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:32.203507 1055 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45f6e8ba99b74898a2922b36768c6c36" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:32.204564 859 leader_election.cc:343] T 0cba5329ab31440999faec599d704633 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
W20250901 01:52:32.204784 859 leader_election.cc:343] T 45f6e8ba99b74898a2922b36768c6c36 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
W20250901 01:52:32.204720 858 leader_election.cc:336] T 0cba5329ab31440999faec599d704633 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:32.204916 858 leader_election.cc:304] T 0cba5329ab31440999faec599d704633 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:32.204944 2718 raft_consensus.cc:491] T c83ce95fd6fa41478de0e2168be201b1 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:32.205106 2718 raft_consensus.cc:513] T c83ce95fd6fa41478de0e2168be201b1 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:32.205242 858 leader_election.cc:336] T 45f6e8ba99b74898a2922b36768c6c36 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:32.205358 858 leader_election.cc:304] T 45f6e8ba99b74898a2922b36768c6c36 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:32.205542 1150 ts_tablet_manager.cc:1397] T 3dae00aa38204ee3ac77325eb334be42 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.073s user 0.003s sys 0.000s
I20250901 01:52:32.205575 2357 raft_consensus.cc:2747] T 0cba5329ab31440999faec599d704633 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:32.205716 2357 raft_consensus.cc:2747] T 45f6e8ba99b74898a2922b36768c6c36 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:32.206286 1150 raft_consensus.cc:357] T 3dae00aa38204ee3ac77325eb334be42 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.206449 1150 raft_consensus.cc:383] T 3dae00aa38204ee3ac77325eb334be42 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:32.206507 1150 raft_consensus.cc:738] T 3dae00aa38204ee3ac77325eb334be42 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:32.207047 2718 leader_election.cc:290] T c83ce95fd6fa41478de0e2168be201b1 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:32.206990 1150 consensus_queue.cc:260] T 3dae00aa38204ee3ac77325eb334be42 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.207415 1055 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c83ce95fd6fa41478de0e2168be201b1" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:32.207787 859 leader_election.cc:343] T c83ce95fd6fa41478de0e2168be201b1 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
W20250901 01:52:32.207962 858 leader_election.cc:336] T c83ce95fd6fa41478de0e2168be201b1 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:32.208076 858 leader_election.cc:304] T c83ce95fd6fa41478de0e2168be201b1 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:32.208098 1150 ts_tablet_manager.cc:1428] T 3dae00aa38204ee3ac77325eb334be42 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250901 01:52:32.208353 2608 raft_consensus.cc:2747] T c83ce95fd6fa41478de0e2168be201b1 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:32.208971 2113 ts_tablet_manager.cc:1905] T afc002ae2486476b9208e1cf8215157c P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:32.215998 1150 tablet_bootstrap.cc:492] T 5d68ea97c71942c7b55b967ba870e0d7 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:32.221822 2113 ts_tablet_manager.cc:1918] T afc002ae2486476b9208e1cf8215157c P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:32.223116 1151 tablet_bootstrap.cc:492] T fa7fb88ded21432a8757f37ec3db5e71 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:32.228134 1151 ts_tablet_manager.cc:1397] T fa7fb88ded21432a8757f37ec3db5e71 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.114s user 0.004s sys 0.022s
I20250901 01:52:32.228917 1151 raft_consensus.cc:357] T fa7fb88ded21432a8757f37ec3db5e71 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.229086 1151 raft_consensus.cc:383] T fa7fb88ded21432a8757f37ec3db5e71 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:32.229135 1151 raft_consensus.cc:738] T fa7fb88ded21432a8757f37ec3db5e71 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:32.229331 1151 consensus_queue.cc:260] T fa7fb88ded21432a8757f37ec3db5e71 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.223363 2666 raft_consensus.cc:491] T 5e2472ee5d6e4d3bbf1f24027ba531ca P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:32.229612 2666 raft_consensus.cc:513] T 5e2472ee5d6e4d3bbf1f24027ba531ca P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.230106 2666 leader_election.cc:290] T 5e2472ee5d6e4d3bbf1f24027ba531ca P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:32.233150 1151 ts_tablet_manager.cc:1428] T fa7fb88ded21432a8757f37ec3db5e71 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20250901 01:52:32.233464 1151 tablet_bootstrap.cc:492] T e2aa32450a584395ae07b49f76a7b26d P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:32.233125 1150 tablet_bootstrap.cc:654] T 5d68ea97c71942c7b55b967ba870e0d7 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:32.234776 992 leader_election.cc:343] T 5e2472ee5d6e4d3bbf1f24027ba531ca P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 5e2472ee5d6e4d3bbf1f24027ba531ca
W20250901 01:52:32.235415 992 leader_election.cc:336] T 5e2472ee5d6e4d3bbf1f24027ba531ca P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:32.235510 992 leader_election.cc:304] T 5e2472ee5d6e4d3bbf1f24027ba531ca P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:32.235671 1151 tablet_bootstrap.cc:654] T e2aa32450a584395ae07b49f76a7b26d P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:32.235752 2666 raft_consensus.cc:2747] T 5e2472ee5d6e4d3bbf1f24027ba531ca P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:32.236586 2113 log.cc:1199] T afc002ae2486476b9208e1cf8215157c P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/afc002ae2486476b9208e1cf8215157c
I20250901 01:52:32.236977 2113 ts_tablet_manager.cc:1939] T afc002ae2486476b9208e1cf8215157c P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:32.237659 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 179) took 5533 ms. Trace:
I20250901 01:52:32.237910 2113 rpcz_store.cc:276] 0901 01:52:26.704198 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.704242 (+ 44us) service_pool.cc:225] Handling call
0901 01:52:32.200297 (+5496055us) ts_tablet_manager.cc:1145] Deleting tablet afc002ae2486476b9208e1cf8215157c
0901 01:52:32.200308 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:32.218725 (+ 18417us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:32.221702 (+ 2977us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:32.237642 (+ 15940us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2736,"tablet-delete.queue_time_us":5495885}
I20250901 01:52:32.237902 2118 ts_tablet_manager.cc:1918] T b85d64e02cc646e380581ac9f59dee9e P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:32.238636 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet afc002ae2486476b9208e1cf8215157c (table default.Table_ad2bde750a6f4c9aad14a28a9a0848c9 [id=da00d16114194d41826ef0da1f9423a9]) successfully deleted
I20250901 01:52:32.239105 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 180) took 5534 ms. Trace:
I20250901 01:52:32.239434 2113 rpcz_store.cc:276] 0901 01:52:26.704823 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.704866 (+ 43us) service_pool.cc:225] Handling call
0901 01:52:32.239016 (+5534150us) ts_tablet_manager.cc:1145] Deleting tablet 75e7a90815e449c39f386901a504143c
0901 01:52:32.239026 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:32.239091 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5534015}
I20250901 01:52:32.239666 2118 log.cc:1199] T b85d64e02cc646e380581ac9f59dee9e P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/b85d64e02cc646e380581ac9f59dee9e
W20250901 01:52:32.240175 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 75e7a90815e449c39f386901a504143c because the tablet was not found. No further retry: Not found: Tablet not found: 75e7a90815e449c39f386901a504143c
I20250901 01:52:32.240585 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 181) took 5533 ms. Trace:
I20250901 01:52:32.240911 2113 rpcz_store.cc:276] 0901 01:52:26.707179 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.707233 (+ 54us) service_pool.cc:225] Handling call
0901 01:52:32.240510 (+5533277us) ts_tablet_manager.cc:1145] Deleting tablet 4d27fb61943c45d4aca67689cc3a9ff3
0901 01:52:32.240518 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:32.240570 (+ 52us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5533151}
W20250901 01:52:32.241472 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 4d27fb61943c45d4aca67689cc3a9ff3 because the tablet was not found. No further retry: Not found: Tablet not found: 4d27fb61943c45d4aca67689cc3a9ff3
I20250901 01:52:32.241809 2113 tablet_replica.cc:331] T 7a607fac57994001b2304e5d81c9decc P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:32.242172 2113 raft_consensus.cc:2241] T 7a607fac57994001b2304e5d81c9decc P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:32.242926 2118 ts_tablet_manager.cc:1939] T b85d64e02cc646e380581ac9f59dee9e P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:32.243510 2113 raft_consensus.cc:2270] T 7a607fac57994001b2304e5d81c9decc P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:32.244225 2113 ts_tablet_manager.cc:1905] T 7a607fac57994001b2304e5d81c9decc P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:32.247290 2666 raft_consensus.cc:491] T c58edaa76ebf47dfb1d9d5fab6de7dfa P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:32.247430 2666 raft_consensus.cc:513] T c58edaa76ebf47dfb1d9d5fab6de7dfa P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.248039 2666 leader_election.cc:290] T c58edaa76ebf47dfb1d9d5fab6de7dfa P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:32.250157 992 leader_election.cc:343] T c58edaa76ebf47dfb1d9d5fab6de7dfa P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: c58edaa76ebf47dfb1d9d5fab6de7dfa
W20250901 01:52:32.251015 992 leader_election.cc:336] T c58edaa76ebf47dfb1d9d5fab6de7dfa P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:32.251154 992 leader_election.cc:304] T c58edaa76ebf47dfb1d9d5fab6de7dfa P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:32.251415 2666 raft_consensus.cc:2747] T c58edaa76ebf47dfb1d9d5fab6de7dfa P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250901 01:52:32.253463 1151 tablet_bootstrap.cc:492] T e2aa32450a584395ae07b49f76a7b26d P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:32.253633 1151 ts_tablet_manager.cc:1397] T e2aa32450a584395ae07b49f76a7b26d P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.004s
I20250901 01:52:32.254323 1151 raft_consensus.cc:357] T e2aa32450a584395ae07b49f76a7b26d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.254467 1151 raft_consensus.cc:383] T e2aa32450a584395ae07b49f76a7b26d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:32.254508 1151 raft_consensus.cc:738] T e2aa32450a584395ae07b49f76a7b26d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:32.254652 1151 consensus_queue.cc:260] T e2aa32450a584395ae07b49f76a7b26d P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.255313 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 149) took 5070 ms. Trace:
I20250901 01:52:32.256291 1151 ts_tablet_manager.cc:1428] T e2aa32450a584395ae07b49f76a7b26d P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250901 01:52:32.255674 2118 rpcz_store.cc:276] 0901 01:52:27.184957 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:27.185056 (+ 99us) service_pool.cc:225] Handling call
0901 01:52:32.082959 (+4897903us) ts_tablet_manager.cc:1145] Deleting tablet b85d64e02cc646e380581ac9f59dee9e
0901 01:52:32.082970 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:32.159313 (+ 76343us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:32.236912 (+ 77599us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:32.255124 (+ 18212us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4897713}
I20250901 01:52:32.257838 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet b85d64e02cc646e380581ac9f59dee9e (table default.Table_7be95f26c04746828f38a19ddb67afc0 [id=2ec1e340c6924cefa540916968475e1f]) successfully deleted
I20250901 01:52:32.257402 2412 consensus_queue.cc:1035] T cb7d7e1796c0456f8de98fb812b9e374 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:32.258718 1151 tablet_bootstrap.cc:492] T e8e7d2a72f704f52a8f8d46911a8a9e4 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:32.260242 1151 tablet_bootstrap.cc:654] T e8e7d2a72f704f52a8f8d46911a8a9e4 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:32.264204 2118 tablet_replica.cc:331] T d10c193452954c198bba686169b8025b P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:32.265866 2118 raft_consensus.cc:2241] T d10c193452954c198bba686169b8025b P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:32.266989 2118 raft_consensus.cc:2270] T d10c193452954c198bba686169b8025b P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:32.271093 2113 ts_tablet_manager.cc:1918] T 7a607fac57994001b2304e5d81c9decc P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:32.271250 2113 log.cc:1199] T 7a607fac57994001b2304e5d81c9decc P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/7a607fac57994001b2304e5d81c9decc
I20250901 01:52:32.271574 2113 ts_tablet_manager.cc:1939] T 7a607fac57994001b2304e5d81c9decc P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:32.272293 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 182) took 5564 ms. Trace:
I20250901 01:52:32.272377 2113 rpcz_store.cc:276] 0901 01:52:26.708056 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.708096 (+ 40us) service_pool.cc:225] Handling call
0901 01:52:32.241767 (+5533671us) ts_tablet_manager.cc:1145] Deleting tablet 7a607fac57994001b2304e5d81c9decc
0901 01:52:32.241775 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:32.247749 (+ 5974us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:32.270988 (+ 23239us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:32.272274 (+ 1286us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5533540}
I20250901 01:52:32.272904 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 7a607fac57994001b2304e5d81c9decc (table default.Table_048a628c8a35498f94012e430b3c29bc [id=5014fdf96714467f98fd130c3b66f37a]) successfully deleted
I20250901 01:52:32.273869 1150 tablet_bootstrap.cc:492] T 5d68ea97c71942c7b55b967ba870e0d7 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:32.274003 1150 ts_tablet_manager.cc:1397] T 5d68ea97c71942c7b55b967ba870e0d7 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.058s user 0.002s sys 0.003s
I20250901 01:52:32.276002 1150 raft_consensus.cc:357] T 5d68ea97c71942c7b55b967ba870e0d7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.276182 1150 raft_consensus.cc:383] T 5d68ea97c71942c7b55b967ba870e0d7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:32.276247 1150 raft_consensus.cc:738] T 5d68ea97c71942c7b55b967ba870e0d7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:32.276716 1150 consensus_queue.cc:260] T 5d68ea97c71942c7b55b967ba870e0d7 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.277422 1150 ts_tablet_manager.cc:1428] T 5d68ea97c71942c7b55b967ba870e0d7 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250901 01:52:32.277720 1150 tablet_bootstrap.cc:492] T b8ca4e66c6a74f9d8d90a69be5d6a58f P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:32.278335 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 183) took 5569 ms. Trace:
I20250901 01:52:32.278437 2113 rpcz_store.cc:276] 0901 01:52:26.708731 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.708771 (+ 40us) service_pool.cc:225] Handling call
0901 01:52:32.272662 (+5563891us) ts_tablet_manager.cc:1145] Deleting tablet e9b179dba2cc49428ba8090c962d6136
0901 01:52:32.272672 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:32.278311 (+ 5639us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5563766}
I20250901 01:52:32.278749 2113 tablet_replica.cc:331] T 05630a72ddae4de0b890dece2feacc31 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:32.278927 2113 raft_consensus.cc:2241] T 05630a72ddae4de0b890dece2feacc31 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250901 01:52:32.279481 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet e9b179dba2cc49428ba8090c962d6136 because the tablet was not found. No further retry: Not found: Tablet not found: e9b179dba2cc49428ba8090c962d6136
I20250901 01:52:32.279786 1150 tablet_bootstrap.cc:654] T b8ca4e66c6a74f9d8d90a69be5d6a58f P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:32.282450 2113 raft_consensus.cc:2270] T 05630a72ddae4de0b890dece2feacc31 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:32.283036 2113 ts_tablet_manager.cc:1905] T 05630a72ddae4de0b890dece2feacc31 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:32.284716 2118 ts_tablet_manager.cc:1905] T d10c193452954c198bba686169b8025b P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:32.298480 2113 ts_tablet_manager.cc:1918] T 05630a72ddae4de0b890dece2feacc31 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:32.298736 2113 log.cc:1199] T 05630a72ddae4de0b890dece2feacc31 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/05630a72ddae4de0b890dece2feacc31
I20250901 01:52:32.299139 2113 ts_tablet_manager.cc:1939] T 05630a72ddae4de0b890dece2feacc31 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:32.304790 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 184) took 5595 ms. Trace:
I20250901 01:52:32.305032 2113 rpcz_store.cc:276] 0901 01:52:26.709274 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.709312 (+ 38us) service_pool.cc:225] Handling call
0901 01:52:32.278701 (+5569389us) ts_tablet_manager.cc:1145] Deleting tablet 05630a72ddae4de0b890dece2feacc31
0901 01:52:32.278710 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:32.290175 (+ 11465us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:32.298357 (+ 8182us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:32.304754 (+ 6397us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3353,"tablet-delete.queue_time_us":5569267}
I20250901 01:52:32.305851 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 05630a72ddae4de0b890dece2feacc31 (table default.Table_ee242c34410e4c6887405d04a4b26124 [id=c9a2f96383664a4e8524c286cd9e4149]) successfully deleted
I20250901 01:52:32.306241 2113 tablet_replica.cc:331] T 5e2472ee5d6e4d3bbf1f24027ba531ca P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:32.306468 2113 raft_consensus.cc:2241] T 5e2472ee5d6e4d3bbf1f24027ba531ca P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:32.306663 2113 raft_consensus.cc:2270] T 5e2472ee5d6e4d3bbf1f24027ba531ca P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:32.307241 2113 ts_tablet_manager.cc:1905] T 5e2472ee5d6e4d3bbf1f24027ba531ca P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:32.310092 1150 tablet_bootstrap.cc:492] T b8ca4e66c6a74f9d8d90a69be5d6a58f P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:32.310231 1150 ts_tablet_manager.cc:1397] T b8ca4e66c6a74f9d8d90a69be5d6a58f P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.001s
I20250901 01:52:32.310915 1150 raft_consensus.cc:357] T b8ca4e66c6a74f9d8d90a69be5d6a58f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.311075 1150 raft_consensus.cc:383] T b8ca4e66c6a74f9d8d90a69be5d6a58f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:32.311127 1150 raft_consensus.cc:738] T b8ca4e66c6a74f9d8d90a69be5d6a58f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:32.311328 1150 consensus_queue.cc:260] T b8ca4e66c6a74f9d8d90a69be5d6a58f P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.311900 1150 ts_tablet_manager.cc:1428] T b8ca4e66c6a74f9d8d90a69be5d6a58f P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250901 01:52:32.312429 1150 tablet_bootstrap.cc:492] T 2df2240631844defa433270a3829d65b P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:32.313951 1150 tablet_bootstrap.cc:654] T 2df2240631844defa433270a3829d65b P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:32.323493 2613 consensus_queue.cc:1035] T 5f52698fb56d472d9e91fb10cb10c69a P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:32.324079 2113 ts_tablet_manager.cc:1918] T 5e2472ee5d6e4d3bbf1f24027ba531ca P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:32.324260 2113 log.cc:1199] T 5e2472ee5d6e4d3bbf1f24027ba531ca P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/5e2472ee5d6e4d3bbf1f24027ba531ca
I20250901 01:52:32.324621 2113 ts_tablet_manager.cc:1939] T 5e2472ee5d6e4d3bbf1f24027ba531ca P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:32.325337 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 185) took 5615 ms. Trace:
I20250901 01:52:32.332466 2113 rpcz_store.cc:276] 0901 01:52:26.710262 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.710309 (+ 47us) service_pool.cc:225] Handling call
0901 01:52:32.306195 (+5595886us) ts_tablet_manager.cc:1145] Deleting tablet 5e2472ee5d6e4d3bbf1f24027ba531ca
0901 01:52:32.306205 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:32.318374 (+ 12169us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:32.323976 (+ 5602us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:32.325319 (+ 1343us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5595761}
I20250901 01:52:32.333302 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 5e2472ee5d6e4d3bbf1f24027ba531ca (table default.Table_d1302673d68f424ebb54b220911ce385 [id=453153553bea48319e132857f628d48a]) successfully deleted
I20250901 01:52:32.333832 2113 tablet_replica.cc:331] T c58edaa76ebf47dfb1d9d5fab6de7dfa P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:32.334156 2113 raft_consensus.cc:2241] T c58edaa76ebf47dfb1d9d5fab6de7dfa P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:32.334352 2118 ts_tablet_manager.cc:1918] T d10c193452954c198bba686169b8025b P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:32.334478 2118 log.cc:1199] T d10c193452954c198bba686169b8025b P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/d10c193452954c198bba686169b8025b
I20250901 01:52:32.334425 2113 raft_consensus.cc:2270] T c58edaa76ebf47dfb1d9d5fab6de7dfa P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:32.334841 2118 ts_tablet_manager.cc:1939] T d10c193452954c198bba686169b8025b P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:32.335342 2113 ts_tablet_manager.cc:1905] T c58edaa76ebf47dfb1d9d5fab6de7dfa P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:32.335597 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 150) took 5150 ms. Trace:
I20250901 01:52:32.335700 2118 rpcz_store.cc:276] 0901 01:52:27.185432 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:27.185500 (+ 68us) service_pool.cc:225] Handling call
0901 01:52:32.257529 (+5072029us) ts_tablet_manager.cc:1145] Deleting tablet d10c193452954c198bba686169b8025b
0901 01:52:32.263973 (+ 6444us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:32.288830 (+ 24857us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:32.334236 (+ 45406us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:32.335574 (+ 1338us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5071848}
I20250901 01:52:32.336153 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet d10c193452954c198bba686169b8025b (table default.Table_03fb48eaab9b42858ab7cefb722e43a8 [id=6c0576410d7340d6aa493a747e95565b]) successfully deleted
I20250901 01:52:32.349684 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 33f637f6a8134215bb1995ec38440073 (table default.Table_3d5afc22c4e14d6990aea31564dfde20 [id=557afe29976d40f6bfb6ae4ca348f887]). Expected version 2 got 0
I20250901 01:52:32.350174 1492 catalog_manager.cc:5582] T df01d5df9fc2424eaf9de4e68e781803 P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: UNKNOWN } } }
I20250901 01:52:32.351070 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 151) took 5165 ms. Trace:
I20250901 01:52:32.352756 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet d0c3c363dda444519a17dca63ad9caa7 (table default.Table_42422c44ca254d8ca989134c2433391f [id=8c1b263824ec4c15a5ec424364c78895]). Expected version 3 got 0
I20250901 01:52:32.352936 2118 rpcz_store.cc:276] 0901 01:52:27.187456 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:27.187518 (+ 62us) service_pool.cc:225] Handling call
0901 01:52:32.350952 (+5163434us) ts_tablet_manager.cc:1145] Deleting tablet 1a60c10e42104502ac2d3b80b8126ceb
0901 01:52:32.350962 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:32.351046 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5163261}
I20250901 01:52:32.353207 1151 tablet_bootstrap.cc:492] T e8e7d2a72f704f52a8f8d46911a8a9e4 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:32.353438 1151 ts_tablet_manager.cc:1397] T e8e7d2a72f704f52a8f8d46911a8a9e4 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.095s user 0.017s sys 0.025s
I20250901 01:52:32.354256 1151 raft_consensus.cc:357] T e8e7d2a72f704f52a8f8d46911a8a9e4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.354543 1151 raft_consensus.cc:383] T e8e7d2a72f704f52a8f8d46911a8a9e4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:32.354664 1151 raft_consensus.cc:738] T e8e7d2a72f704f52a8f8d46911a8a9e4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:32.354440 1150 tablet_bootstrap.cc:492] T 2df2240631844defa433270a3829d65b P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:32.354936 1151 consensus_queue.cc:260] T e8e7d2a72f704f52a8f8d46911a8a9e4 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.355074 1150 ts_tablet_manager.cc:1397] T 2df2240631844defa433270a3829d65b P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.000s
I20250901 01:52:32.355644 1151 ts_tablet_manager.cc:1428] T e8e7d2a72f704f52a8f8d46911a8a9e4 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250901 01:52:32.355955 1151 tablet_bootstrap.cc:492] T 3eba54e5ddd14c518f4cbd539268ae37 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:32.355871 1150 raft_consensus.cc:357] T 2df2240631844defa433270a3829d65b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.356045 1150 raft_consensus.cc:383] T 2df2240631844defa433270a3829d65b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:32.356106 1150 raft_consensus.cc:738] T 2df2240631844defa433270a3829d65b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:32.356298 1150 consensus_queue.cc:260] T 2df2240631844defa433270a3829d65b P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.356848 1150 ts_tablet_manager.cc:1428] T 2df2240631844defa433270a3829d65b P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250901 01:52:32.357316 1150 tablet_bootstrap.cc:492] T 5dd0c6b806574caf9126dad94833fcf7 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:32.359542 1151 tablet_bootstrap.cc:654] T 3eba54e5ddd14c518f4cbd539268ae37 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:32.359691 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 152) took 5171 ms. Trace:
I20250901 01:52:32.359802 2118 rpcz_store.cc:276] 0901 01:52:27.187970 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:27.188051 (+ 81us) service_pool.cc:225] Handling call
0901 01:52:32.359580 (+5171529us) ts_tablet_manager.cc:1145] Deleting tablet 56475ba491a24301a534304811f30140
0901 01:52:32.359591 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:32.359673 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5171387}
W20250901 01:52:32.354853 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 1a60c10e42104502ac2d3b80b8126ceb because the tablet was not found. No further retry: Not found: Tablet not found: 1a60c10e42104502ac2d3b80b8126ceb
W20250901 01:52:32.360774 992 consensus_peers.cc:489] T 5f52698fb56d472d9e91fb10cb10c69a P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250901 01:52:32.360898 992 consensus_peers.cc:489] T a8483e7212d94daba7e5401256c33aef P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250901 01:52:32.360981 992 consensus_peers.cc:489] T cb7d7e1796c0456f8de98fb812b9e374 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:32.360738 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 45f6e8ba99b74898a2922b36768c6c36 (table default.Table_938421db3cf74cf8b878089bac931818 [id=9ce26dcf307149b595a43b6ac4562fba]). Expected version 1 got 0
I20250901 01:52:32.361956 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet e9f1f892658b4736859ff507536920a1 (table default.Table_8825dd28afbe464193eb1b668cc9326a [id=9aa62cb634314a0a849612c54c1220d9]). Expected version 1 got 0
I20250901 01:52:32.362262 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 153) took 5151 ms. Trace:
I20250901 01:52:32.363376 2118 rpcz_store.cc:276] 0901 01:52:27.210971 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:27.211229 (+ 258us) service_pool.cc:225] Handling call
0901 01:52:32.362176 (+5150947us) ts_tablet_manager.cc:1145] Deleting tablet afc002ae2486476b9208e1cf8215157c
0901 01:52:32.362185 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:32.362245 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5150754}
W20250901 01:52:32.363935 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet afc002ae2486476b9208e1cf8215157c because the tablet was not found. No further retry: Not found: Tablet not found: afc002ae2486476b9208e1cf8215157c
I20250901 01:52:32.364576 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 5f5396045979450193d461ba1334a5c9 (table default.Table_2b946a63bd574ac0971fa3d3b9c9a6a4 [id=9c3930e9fcfe4b49ba4bea67a2b0ad07]). Expected version 2 got 0
I20250901 01:52:32.364833 2118 tablet_replica.cc:331] T ff013ea889d44cba8797834e4611b489 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:32.365087 2118 raft_consensus.cc:2241] T ff013ea889d44cba8797834e4611b489 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:32.368664 2118 raft_consensus.cc:2270] T ff013ea889d44cba8797834e4611b489 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:32.369050 1492 catalog_manager.cc:5582] T 4cde5f4f02044b7bbd7d43b49aac4ae8 P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: UNKNOWN } } }
I20250901 01:52:32.370857 2118 ts_tablet_manager.cc:1905] T ff013ea889d44cba8797834e4611b489 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:32.371125 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 4cde5f4f02044b7bbd7d43b49aac4ae8 (table default.Table_d53a456dbca64daf92aeb5fb4517b417 [id=77ae3376281349e087a3b3cee41f2a09]). Expected version 1 got 0
I20250901 01:52:32.371855 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 315c15cf0afc44d4a79cb29294281d60 (table default.Table_7aa28ad9ee8c49859af40f9ccb61af33 [id=aed8d4b4fa6a4fbf850cf5718610108f]). Expected version 1 got 0
I20250901 01:52:32.373631 2113 ts_tablet_manager.cc:1918] T c58edaa76ebf47dfb1d9d5fab6de7dfa P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:32.373807 2113 log.cc:1199] T c58edaa76ebf47dfb1d9d5fab6de7dfa P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/c58edaa76ebf47dfb1d9d5fab6de7dfa
I20250901 01:52:32.374151 2113 ts_tablet_manager.cc:1939] T c58edaa76ebf47dfb1d9d5fab6de7dfa P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:32.368522 1150 tablet_bootstrap.cc:654] T 5dd0c6b806574caf9126dad94833fcf7 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:32.375104 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet b164b4a67a9143ceaff69b0ce49d63fe (table default.Table_4505d77e4269443995012fef50c7e7b5 [id=c1b1e9e3876244f68d31c63ce7bdebf1]). Expected version 2 got 0
I20250901 01:52:32.375278 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet ae21b9ae2e3b472eb67fb5f96388a50f (table default.Table_046289f393ff432aae0b033bdec1fb17 [id=db57b18a9ce24e068b362f5fbe940531]). Expected version 1 got 0
I20250901 01:52:32.376158 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 186) took 5664 ms. Trace:
I20250901 01:52:32.376243 2113 rpcz_store.cc:276] 0901 01:52:26.711323 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.711361 (+ 38us) service_pool.cc:225] Handling call
0901 01:52:32.333744 (+5622383us) ts_tablet_manager.cc:1145] Deleting tablet c58edaa76ebf47dfb1d9d5fab6de7dfa
0901 01:52:32.333761 (+ 17us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:32.340616 (+ 6855us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:32.373525 (+ 32909us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:32.376140 (+ 2615us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1149,"tablet-delete.queue_time_us":5622226}
I20250901 01:52:32.376585 2113 tablet_replica.cc:331] T e249474dcbf642189560c91ec3a99fdf P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:32.376788 2113 raft_consensus.cc:2241] T e249474dcbf642189560c91ec3a99fdf P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:32.376829 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet c58edaa76ebf47dfb1d9d5fab6de7dfa (table default.Table_3df3e67f179f46a8be496e3bfe3cb89f [id=3fa0578ede344c0a8ff5d00b3703ad74]) successfully deleted
I20250901 01:52:32.377447 2113 raft_consensus.cc:2270] T e249474dcbf642189560c91ec3a99fdf P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:32.377664 1492 catalog_manager.cc:5582] T 6c0abbfc8d904dce989ce51961e4237d P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00c282f779c1477d920f24dc0dc16628 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "00c282f779c1477d920f24dc0dc16628" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } }
I20250901 01:52:32.377990 2113 ts_tablet_manager.cc:1905] T e249474dcbf642189560c91ec3a99fdf P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:32.378000 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 6c0abbfc8d904dce989ce51961e4237d (table default.Table_3f9422b08b0349b9a5466218f0fe1b06 [id=f0dfb3811e744bd4ba6d59195cebde6f]). Expected version 2 got 0
I20250901 01:52:32.378268 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 98c56eb62ad84f79b04f17a43d44c479 (table default.Table_50ff85e8f1314f4499e76299441bc110 [id=00e7f7fe6dc44f2787b9dd8e3cbf4a51]). Expected version 1 got 0
I20250901 01:52:32.378417 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet dabebc77f72e4bb19e264be2f496202a (table default.Table_2755f7caf53340afbd230181643366db [id=a4d0af97df5949cca940a81a41b8e454]). Expected version 1 got 0
I20250901 01:52:32.378558 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 0e4009963cfe42d6ba164fc828897f09 (table default.Table_1adc70cccd2f4cb3a79df96861cb4181 [id=24fffc7ebf5a47bdb37211910301acf7]). Expected version 1 got 0
I20250901 01:52:32.378769 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 93bcac1fd95d44dc927abfca81692921 (table default.Table_b8f8ee74648c406eaf589d3ab80934db [id=b8f69313439543be9c11eea88dfa69c5]). Expected version 1 got 0
I20250901 01:52:32.379033 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 508406601c994e42a5c5b1f665e74957 (table default.Table_345b7f9fe8214aa6878f20a166125364 [id=4aabc328f6374ec5b1dea26f778208b2]). Expected version 1 got 0
I20250901 01:52:32.379338 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 5d68ea97c71942c7b55b967ba870e0d7 (table default.Table_f610cb75906d4c5cb3b57cfdb80ecefc [id=36915a3e843340ba8ef3578ac35a9af4]). Expected version 1 got 0
I20250901 01:52:32.379496 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 2973b59bb1df41d18d9a5b6a29ae336f (table default.Table_7790af96855e45c0a316ba3a4a87c9ed [id=746845bd84c6444b8a1e5683eaba4a68]). Expected version 1 got 0
I20250901 01:52:32.380172 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet c6f7c525f59b49938b697de98d4b5835 (table default.Table_db61e055dbd7450898e82c4c0c4763be [id=aabca67f6f754820977ac9c4c20f3e4f]). Expected version 1 got 0
I20250901 01:52:32.380802 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 61ddcf86c9f34726939cc116434f0822 (table default.Table_63d0f2ead9d946d39ce40162ebd769d6 [id=f9dcf4937b1143248bfa9d88053fce17]). Expected version 3 got 0
I20250901 01:52:32.381018 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 858218c4427c41e980b2b5c969a5af40 (table default.Table_6d47266af0ed433eb1f2abbc81d18e69 [id=264cc469a73841e79538843479a165ab]). Expected version 1 got 0
I20250901 01:52:32.385247 2113 ts_tablet_manager.cc:1918] T e249474dcbf642189560c91ec3a99fdf P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:32.385447 2113 log.cc:1199] T e249474dcbf642189560c91ec3a99fdf P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/e249474dcbf642189560c91ec3a99fdf
I20250901 01:52:32.385838 2113 ts_tablet_manager.cc:1939] T e249474dcbf642189560c91ec3a99fdf P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:32.387161 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 187) took 5674 ms. Trace:
I20250901 01:52:32.390588 2113 rpcz_store.cc:276] 0901 01:52:26.712365 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.712423 (+ 58us) service_pool.cc:225] Handling call
0901 01:52:32.376531 (+5664108us) ts_tablet_manager.cc:1145] Deleting tablet e249474dcbf642189560c91ec3a99fdf
0901 01:52:32.376541 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:32.381875 (+ 5334us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:32.385169 (+ 3294us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:32.387140 (+ 1971us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5663966}
I20250901 01:52:32.391175 1761 raft_consensus.cc:491] T fa7fb88ded21432a8757f37ec3db5e71 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:32.392194 1761 raft_consensus.cc:513] T fa7fb88ded21432a8757f37ec3db5e71 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.392798 1761 leader_election.cc:290] T fa7fb88ded21432a8757f37ec3db5e71 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:32.393150 2113 tablet_replica.cc:331] T eaa521c52a2f4408b59770bd2a05907c P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:32.393414 2113 raft_consensus.cc:2241] T eaa521c52a2f4408b59770bd2a05907c P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:32.393774 2113 raft_consensus.cc:2270] T eaa521c52a2f4408b59770bd2a05907c P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:32.398864 2113 ts_tablet_manager.cc:1905] T eaa521c52a2f4408b59770bd2a05907c P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:32.400538 992 leader_election.cc:343] T fa7fb88ded21432a8757f37ec3db5e71 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: fa7fb88ded21432a8757f37ec3db5e71
W20250901 01:52:32.408937 992 leader_election.cc:336] T fa7fb88ded21432a8757f37ec3db5e71 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:32.409161 992 leader_election.cc:304] T fa7fb88ded21432a8757f37ec3db5e71 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:32.409443 1761 raft_consensus.cc:2747] T fa7fb88ded21432a8757f37ec3db5e71 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:32.425887 1151 tablet_bootstrap.cc:492] T 3eba54e5ddd14c518f4cbd539268ae37 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:32.426148 1151 ts_tablet_manager.cc:1397] T 3eba54e5ddd14c518f4cbd539268ae37 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.070s user 0.010s sys 0.014s
I20250901 01:52:32.426960 1151 raft_consensus.cc:357] T 3eba54e5ddd14c518f4cbd539268ae37 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.427206 1151 raft_consensus.cc:383] T 3eba54e5ddd14c518f4cbd539268ae37 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:32.427295 1151 raft_consensus.cc:738] T 3eba54e5ddd14c518f4cbd539268ae37 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:32.427534 1151 consensus_queue.cc:260] T 3eba54e5ddd14c518f4cbd539268ae37 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.428172 1151 ts_tablet_manager.cc:1428] T 3eba54e5ddd14c518f4cbd539268ae37 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250901 01:52:32.427609 2113 ts_tablet_manager.cc:1918] T eaa521c52a2f4408b59770bd2a05907c P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:32.429387 2113 log.cc:1199] T eaa521c52a2f4408b59770bd2a05907c P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/eaa521c52a2f4408b59770bd2a05907c
I20250901 01:52:32.430307 1761 consensus_queue.cc:1035] T 6ba88af177874544b764cc32420e7ac9 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:32.431007 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 81ce448b148447ddbc41875d863e0518 (table default.Table_6e461d9cb0ae4dd095975bbc21533794 [id=49673fa73f8c436e9b8026efc4f3cbbc]). Expected version 1 got 0
I20250901 01:52:32.432164 1150 tablet_bootstrap.cc:492] T 5dd0c6b806574caf9126dad94833fcf7 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:32.432297 1150 ts_tablet_manager.cc:1397] T 5dd0c6b806574caf9126dad94833fcf7 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.075s user 0.001s sys 0.005s
I20250901 01:52:32.433586 2357 raft_consensus.cc:491] T 6ba8241644984394ada95b561e76467e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:32.433368 1150 raft_consensus.cc:357] T 5dd0c6b806574caf9126dad94833fcf7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.433725 2357 raft_consensus.cc:513] T 6ba8241644984394ada95b561e76467e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.433856 1150 raft_consensus.cc:383] T 5dd0c6b806574caf9126dad94833fcf7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:32.433912 1150 raft_consensus.cc:738] T 5dd0c6b806574caf9126dad94833fcf7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:32.434325 1150 consensus_queue.cc:260] T 5dd0c6b806574caf9126dad94833fcf7 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.434720 2666 raft_consensus.cc:491] T eb19e3a424324a24824f6caaf5848952 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:32.434821 2666 raft_consensus.cc:513] T eb19e3a424324a24824f6caaf5848952 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.435284 1150 ts_tablet_manager.cc:1428] T 5dd0c6b806574caf9126dad94833fcf7 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250901 01:52:32.435285 2666 leader_election.cc:290] T eb19e3a424324a24824f6caaf5848952 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:32.435532 1150 tablet_bootstrap.cc:492] T 3a2bc5ad1cd243daa3c92fceca8ba414 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:32.434353 2357 leader_election.cc:290] T 6ba8241644984394ada95b561e76467e P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:32.436380 1055 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ba8241644984394ada95b561e76467e" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:32.436486 858 leader_election.cc:336] T 6ba8241644984394ada95b561e76467e P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:32.436523 922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eb19e3a424324a24824f6caaf5848952" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:32.436692 922 raft_consensus.cc:2466] T eb19e3a424324a24824f6caaf5848952 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
W20250901 01:52:32.437095 859 leader_election.cc:343] T 6ba8241644984394ada95b561e76467e P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:32.437108 992 leader_election.cc:304] T eb19e3a424324a24824f6caaf5848952 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:32.437186 859 leader_election.cc:304] T 6ba8241644984394ada95b561e76467e P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:32.437376 2666 raft_consensus.cc:2802] T eb19e3a424324a24824f6caaf5848952 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:32.437439 2666 raft_consensus.cc:491] T eb19e3a424324a24824f6caaf5848952 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:32.437497 2666 raft_consensus.cc:3058] T eb19e3a424324a24824f6caaf5848952 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:32.437610 2357 raft_consensus.cc:2747] T 6ba8241644984394ada95b561e76467e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:32.440024 2118 ts_tablet_manager.cc:1918] T ff013ea889d44cba8797834e4611b489 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250901 01:52:32.440189 992 leader_election.cc:336] T eb19e3a424324a24824f6caaf5848952 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:32.440192 2666 raft_consensus.cc:513] T eb19e3a424324a24824f6caaf5848952 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.440970 2666 leader_election.cc:290] T eb19e3a424324a24824f6caaf5848952 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:32.441031 1492 catalog_manager.cc:5582] T 3239e58531d445fa80165cf678bc277c P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: UNKNOWN } } }
I20250901 01:52:32.441445 2113 ts_tablet_manager.cc:1939] T eaa521c52a2f4408b59770bd2a05907c P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:32.441919 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet c83ce95fd6fa41478de0e2168be201b1 (table default.Table_a525265208584713b1b3465d63130cd9 [id=bfd87b9b708449d3b16e3e0639561a49]). Expected version 1 got 0
I20250901 01:52:32.442389 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet ed1b41340ce742a88c3e4fd85cd25ca6 (table default.Table_ce2f3b515c8f4c1bb557c5ed84b48dba [id=0baa4993b72e4a8d94ee3059c9d2db6c]). Expected version 2 got 0
W20250901 01:52:32.442617 992 leader_election.cc:336] T eb19e3a424324a24824f6caaf5848952 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:32.440280 2357 raft_consensus.cc:491] T f98d1d8d660443cb8066a547e5d9b77f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:32.443815 2118 log.cc:1199] T ff013ea889d44cba8797834e4611b489 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/ff013ea889d44cba8797834e4611b489
I20250901 01:52:32.443845 2357 raft_consensus.cc:513] T f98d1d8d660443cb8066a547e5d9b77f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.440713 1150 tablet_bootstrap.cc:654] T 3a2bc5ad1cd243daa3c92fceca8ba414 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:32.442054 922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eb19e3a424324a24824f6caaf5848952" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:32.444211 922 raft_consensus.cc:3058] T eb19e3a424324a24824f6caaf5848952 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:32.444380 2357 leader_election.cc:290] T f98d1d8d660443cb8066a547e5d9b77f P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:32.445190 1055 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f98d1d8d660443cb8066a547e5d9b77f" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:32.446135 859 leader_election.cc:343] T f98d1d8d660443cb8066a547e5d9b77f P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
W20250901 01:52:32.446334 858 leader_election.cc:336] T f98d1d8d660443cb8066a547e5d9b77f P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:32.446416 858 leader_election.cc:304] T f98d1d8d660443cb8066a547e5d9b77f P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:32.446447 2118 ts_tablet_manager.cc:1939] T ff013ea889d44cba8797834e4611b489 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:32.446774 2357 raft_consensus.cc:2747] T f98d1d8d660443cb8066a547e5d9b77f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:32.447077 1151 tablet_bootstrap.cc:492] T 246b33338de647689d7404c70676e76f P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:32.447101 922 raft_consensus.cc:2466] T eb19e3a424324a24824f6caaf5848952 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:32.447118 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 154) took 5235 ms. Trace:
I20250901 01:52:32.448314 2118 rpcz_store.cc:276] 0901 01:52:27.211169 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:27.211219 (+ 50us) service_pool.cc:225] Handling call
0901 01:52:32.364775 (+5153556us) ts_tablet_manager.cc:1145] Deleting tablet ff013ea889d44cba8797834e4611b489
0901 01:52:32.364785 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:32.422259 (+ 57474us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:32.439925 (+ 17666us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:32.447101 (+ 7176us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5153318}
I20250901 01:52:32.448706 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 155) took 5189 ms. Trace:
I20250901 01:52:32.448827 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet ff013ea889d44cba8797834e4611b489 (table default.Table_0f91070385df441b96af4d5406a94724 [id=4747cdd47a954e0ab19609c9526b688d]) successfully deleted
I20250901 01:52:32.448796 2118 rpcz_store.cc:276] 0901 01:52:27.258759 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:27.258806 (+ 47us) service_pool.cc:225] Handling call
0901 01:52:32.448616 (+5189810us) ts_tablet_manager.cc:1145] Deleting tablet 75e7a90815e449c39f386901a504143c
0901 01:52:32.448626 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:32.448691 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5189630}
I20250901 01:52:32.449080 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 157) took 5190 ms. Trace:
I20250901 01:52:32.449151 2118 rpcz_store.cc:276] 0901 01:52:27.259007 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:27.259044 (+ 37us) service_pool.cc:225] Handling call
0901 01:52:32.449011 (+5189967us) ts_tablet_manager.cc:1145] Deleting tablet 74b709cb5ec64006b2f12bead8217d80
0901 01:52:32.449019 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:32.449068 (+ 49us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5189788}
W20250901 01:52:32.449249 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 75e7a90815e449c39f386901a504143c because the tablet was not found. No further retry: Not found: Tablet not found: 75e7a90815e449c39f386901a504143c
I20250901 01:52:32.449420 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 156) took 5190 ms. Trace:
I20250901 01:52:32.449487 2118 rpcz_store.cc:276] 0901 01:52:27.258892 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:27.259200 (+ 308us) service_pool.cc:225] Handling call
0901 01:52:32.449350 (+5190150us) ts_tablet_manager.cc:1145] Deleting tablet 05630a72ddae4de0b890dece2feacc31
0901 01:52:32.449358 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:32.449406 (+ 48us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5189989}
W20250901 01:52:32.449615 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 74b709cb5ec64006b2f12bead8217d80 because the tablet was not found. No further retry: Not found: Tablet not found: 74b709cb5ec64006b2f12bead8217d80
I20250901 01:52:32.449680 992 leader_election.cc:304] T eb19e3a424324a24824f6caaf5848952 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:32.449782 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 158) took 5190 ms. Trace:
I20250901 01:52:32.449852 2118 rpcz_store.cc:276] 0901 01:52:27.259151 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:27.259393 (+ 242us) service_pool.cc:225] Handling call
0901 01:52:32.449704 (+5190311us) ts_tablet_manager.cc:1145] Deleting tablet 8dd587304e6342edbe96c9897b40e5c9
0901 01:52:32.449712 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:32.449755 (+ 43us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5190175}
I20250901 01:52:32.450080 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 159) took 5182 ms. Trace:
I20250901 01:52:32.450138 2118 rpcz_store.cc:276] 0901 01:52:27.267970 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:27.268320 (+ 350us) service_pool.cc:225] Handling call
0901 01:52:32.450018 (+5181698us) ts_tablet_manager.cc:1145] Deleting tablet 40d74d7b5a3b4d1dae09eebd9e07a0c4
0901 01:52:32.450027 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:32.450070 (+ 43us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5181456}
I20250901 01:52:32.450351 2118 tablet_replica.cc:331] T 75345ea2a3dc40fc8c9f8090d8839d2d P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
W20250901 01:52:32.450419 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 05630a72ddae4de0b890dece2feacc31 because the tablet was not found. No further retry: Not found: Tablet not found: 05630a72ddae4de0b890dece2feacc31
I20250901 01:52:32.450510 2118 raft_consensus.cc:2241] T 75345ea2a3dc40fc8c9f8090d8839d2d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:32.450632 2118 raft_consensus.cc:2270] T 75345ea2a3dc40fc8c9f8090d8839d2d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250901 01:52:32.450644 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 8dd587304e6342edbe96c9897b40e5c9 because the tablet was not found. No further retry: Not found: Tablet not found: 8dd587304e6342edbe96c9897b40e5c9
W20250901 01:52:32.451171 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 40d74d7b5a3b4d1dae09eebd9e07a0c4 because the tablet was not found. No further retry: Not found: Tablet not found: 40d74d7b5a3b4d1dae09eebd9e07a0c4
I20250901 01:52:32.452917 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskddMNdm/build/asan/bin/kudu pid 492
I20250901 01:52:32.453979 1151 tablet_bootstrap.cc:654] T 246b33338de647689d7404c70676e76f P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:32.455320 556 generic_service.cc:196] Checking for leaks (request via RPC)
I20250901 01:52:32.458456 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 188) took 5727 ms. Trace:
I20250901 01:52:32.460067 1761 raft_consensus.cc:2802] T eb19e3a424324a24824f6caaf5848952 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:32.460180 1761 raft_consensus.cc:695] T eb19e3a424324a24824f6caaf5848952 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:32.460386 1761 consensus_queue.cc:237] T eb19e3a424324a24824f6caaf5848952 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.460099 2113 rpcz_store.cc:276] 0901 01:52:26.730698 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.730754 (+ 56us) service_pool.cc:225] Handling call
0901 01:52:32.393105 (+5662351us) ts_tablet_manager.cc:1145] Deleting tablet eaa521c52a2f4408b59770bd2a05907c
0901 01:52:32.393116 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:32.406207 (+ 13091us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:32.425005 (+ 18798us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:32.458430 (+ 33425us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5662171}
I20250901 01:52:32.461144 2113 tablet_replica.cc:331] T c754129977904f978f87fd89d7092e36 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:32.461390 2113 raft_consensus.cc:2241] T c754129977904f978f87fd89d7092e36 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:32.461549 2113 raft_consensus.cc:2270] T c754129977904f978f87fd89d7092e36 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:32.461802 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet eaa521c52a2f4408b59770bd2a05907c (table default.Table_14cf61059c5145b0a02159aae2a3e457 [id=a33b9b981bd64d7981829d663b0b48b5]) successfully deleted
I20250901 01:52:32.462153 2113 ts_tablet_manager.cc:1905] T c754129977904f978f87fd89d7092e36 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:32.470831 2666 consensus_queue.cc:1035] T df01d5df9fc2424eaf9de4e68e781803 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250901 01:52:32.470935 992 consensus_peers.cc:489] T 5afae316ff164413be0d25d4bc0f58dc P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250901 01:52:32.471131 992 consensus_peers.cc:489] T ad45c90ba0da43c9860745e534664a98 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250901 01:52:32.472903 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ddf9efadfebe423db947169a4cfffd81"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:32.473994 1486 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_345b7f9fe8214aa6878f20a166125364" } new_table_name: "default.Table_7f5b278599994063862981b6b041ca8c" modify_external_catalogs: true
W20250901 01:52:32.482596 992 consensus_peers.cc:489] T 3618a3eb4f174c43805ce7dc0e5315cb P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250901 01:52:32.489462 992 consensus_peers.cc:489] T b13afacd37474781877b9f2cd20b313a P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:32.494994 2113 ts_tablet_manager.cc:1918] T c754129977904f978f87fd89d7092e36 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:32.502902 2113 log.cc:1199] T c754129977904f978f87fd89d7092e36 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/c754129977904f978f87fd89d7092e36
E20250901 01:52:32.505481 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 0c88c339e59a4c1da21756cba0645bdd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d25639c7008f4e29af6226039e4b976b'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:32.505640 1696 catalog_manager.cc:6205] Error selecting replicas for tablet a656adbf1b29431687e0989f0db3459c, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2d3d2227e6a24bd2a86f457e553e698b'. Need at least 3 replicas, but only 2 tablet servers are available
W20250901 01:52:32.497752 992 consensus_peers.cc:489] T 6ba88af177874544b764cc32420e7ac9 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
E20250901 01:52:32.505772 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 494b6488aa0d4355b4caa46004267465, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_3ef97ecacc3042099ec1cc5c3a7a0763'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:32.505857 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 0ee3f35392954817b350e5f48653be78, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_442e11b9211c4b209c68f653d7bfc313'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:32.505973 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae4844183a3c4a04a775e6fd878391a3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_aee74198b51d4f089538c4f2bf3ce4dc'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:32.506081 1696 catalog_manager.cc:6205] Error selecting replicas for tablet be8dfbc466564842a2fdb4953eee939b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_75ba3ba70ca24a94920e7bce96190ff2'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:32.506192 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9745582cebfe47b58c21d0191ab4cb9b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77728af9eee44f29a1e2131d5b75b7ca'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:32.506296 1696 catalog_manager.cc:6205] Error selecting replicas for tablet e2ed32b3292141278af28026528e96c2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b5077f55951d48be92a67407ffe26a08'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:32.506413 1696 catalog_manager.cc:6205] Error selecting replicas for tablet eea6bfabe6cc4f83905c24f0f7a5dfc9, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_e0902dfbdd984f85b3fdac8d283731f7'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:32.506522 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5a7fa68d29644ae8afd2711e39dde59a, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_f9e2a9182f6340bfa551fa2fc2eb2429'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:32.506645 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ff2bd017417c41b5b8225b8f00283e63, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_97526c9a6c9f4acbb6dedd78475571e0'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:32.506740 1696 catalog_manager.cc:6205] Error selecting replicas for tablet dcc60bb000f440f1aca9133928b5ce20, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2fbca8463d6546c9b8f5b018fc60251d'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:32.506862 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 65e5cdc10e9642b7b0df775cb02861fa, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_1b5729d9768546fe99eaafc4e2604f42'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:32.506961 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f8f923ddcd2740bcb7e521f4f57c4034, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_5e65c5f068b84995a2d7a76fa65dda0f'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:32.507073 1696 catalog_manager.cc:6205] Error selecting replicas for tablet e4d8330a4e454d1b8d643e17a6996741, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ce16a1b94f714c2699ee6cd916bb0215'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:32.507171 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 0813cc68d4b0488ba60a523e4479971b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d1302673d68f424ebb54b220911ce385'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:32.507261 1696 catalog_manager.cc:6205] Error selecting replicas for tablet b1ba0e19dc0b462e80c537bf2d183f5b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_66b111271c15410aab23682f8f6081ac'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:32.507334 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 8e10a01e77d14cf28c2c5e0205e595b9, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_83cf2d86a1284544ba2e7aa9ebb66bb7'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:32.507427 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 05a547a8866149cb95d23e960a737237, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_583e53c2e86544e9be68eb27ab6068a3'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:32.507508 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5f0e054f62de4cf6bc6f69ad6664b192, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ef6832d0845044ed9f165ca3541b4c3f'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:32.507603 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae9a4ad4fdfb4771ba23a91f7007447e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_5e84ae9a9c2a4084a1f9c5ffb4594de2'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:32.507696 1696 catalog_manager.cc:6205] Error selecting replicas for tablet d602c8986c75492ead9936ac04e2622d, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_82d91a1ba2eb4265896824028dec8d3a'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:32.507831 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 93aa22afc0ad4d7fa24807ea3c5d5c21, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0f4b4094d2974b7591875375eda6bed0'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:32.507930 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9074543dea9949e7a085717b40913348, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_53e30205c5fd488eb3b48c685a159d04'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:32.508041 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 3a62034e688348ed9e2ac9de009ed3a2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_409dad98d7eb455a8e16edfb4acedb34'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:32.508144 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 18498c9cb09141d08c1c5c0854b63ebd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77f4810bc83c4d948531da2a45a3792e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:32.508260 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f5e3316adeae47cd848833dd84b972bb, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7570c3012f8a4ae7bcfc7346eadac92a'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:32.508529 2357 raft_consensus.cc:491] T b8ca4e66c6a74f9d8d90a69be5d6a58f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:32.508649 2113 ts_tablet_manager.cc:1939] T c754129977904f978f87fd89d7092e36 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:32.508661 2357 raft_consensus.cc:513] T b8ca4e66c6a74f9d8d90a69be5d6a58f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.509243 2357 leader_election.cc:290] T b8ca4e66c6a74f9d8d90a69be5d6a58f P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:32.509393 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 189) took 5778 ms. Trace:
I20250901 01:52:32.509472 2113 rpcz_store.cc:276] 0901 01:52:26.731377 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.731423 (+ 46us) service_pool.cc:225] Handling call
0901 01:52:32.461085 (+5729662us) ts_tablet_manager.cc:1145] Deleting tablet c754129977904f978f87fd89d7092e36
0901 01:52:32.461099 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:32.487548 (+ 26449us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:32.494888 (+ 7340us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:32.509375 (+ 14487us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5729496}
I20250901 01:52:32.510063 1055 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8ca4e66c6a74f9d8d90a69be5d6a58f" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
E20250901 01:52:32.508360 1696 catalog_manager.cc:6205] Error selecting replicas for tablet a6b539e160194da3a849d9e2f1ec9859, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_03fb48eaab9b42858ab7cefb722e43a8'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:32.509970 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet c754129977904f978f87fd89d7092e36 (table default.Table_e113aaaa38db48fdb16d32569b81b930 [id=f09f20dc4db844a8abc825e9086093ba]) successfully deleted
E20250901 01:52:32.510422 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 6ce9521b925b4f97bb4a810ce77e147e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_9e04f976a93f4d30baccf90fa3622dfe'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:32.511140 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 190) took 5779 ms. Trace:
W20250901 01:52:32.511196 859 leader_election.cc:343] T b8ca4e66c6a74f9d8d90a69be5d6a58f P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
W20250901 01:52:32.511507 858 leader_election.cc:336] T b8ca4e66c6a74f9d8d90a69be5d6a58f P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:32.511425 2113 rpcz_store.cc:276] 0901 01:52:26.731988 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.732033 (+ 45us) service_pool.cc:225] Handling call
0901 01:52:32.511051 (+5779018us) ts_tablet_manager.cc:1145] Deleting tablet 17e8340dbd1a406cb782dad3fe88cc7e
0901 01:52:32.511061 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:32.511123 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5778885}
I20250901 01:52:32.511600 858 leader_election.cc:304] T b8ca4e66c6a74f9d8d90a69be5d6a58f P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:32.511818 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 17e8340dbd1a406cb782dad3fe88cc7e because the tablet was not found. No further retry: Not found: Tablet not found: 17e8340dbd1a406cb782dad3fe88cc7e
I20250901 01:52:32.511844 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 191) took 5779 ms. Trace:
I20250901 01:52:32.512022 2357 raft_consensus.cc:2747] T b8ca4e66c6a74f9d8d90a69be5d6a58f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:32.512252 2357 raft_consensus.cc:491] T 26c8ed083cea4dc0acca80d00b11b29b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:32.512346 2357 raft_consensus.cc:513] T 26c8ed083cea4dc0acca80d00b11b29b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.512912 2357 leader_election.cc:290] T 26c8ed083cea4dc0acca80d00b11b29b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:32.513386 1055 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26c8ed083cea4dc0acca80d00b11b29b" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:32.511909 2113 rpcz_store.cc:276] 0901 01:52:26.732639 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.732680 (+ 41us) service_pool.cc:225] Handling call
0901 01:52:32.511769 (+5779089us) ts_tablet_manager.cc:1145] Deleting tablet e25923b2daf9429683b2d44a75a56189
0901 01:52:32.511778 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:32.511829 (+ 51us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5778950}
W20250901 01:52:32.513789 859 leader_election.cc:343] T 26c8ed083cea4dc0acca80d00b11b29b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
E20250901 01:52:32.510522 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2f61edb50d7a42198bfd70639c7bf4de, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_a5c783cbdb694af69047b31d99ff8def'. Need at least 3 replicas, but only 2 tablet servers are available
W20250901 01:52:32.514180 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet e25923b2daf9429683b2d44a75a56189 because the tablet was not found. No further retry: Not found: Tablet not found: e25923b2daf9429683b2d44a75a56189
I20250901 01:52:32.514396 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 192) took 5781 ms. Trace:
W20250901 01:52:32.514701 858 leader_election.cc:336] T 26c8ed083cea4dc0acca80d00b11b29b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:32.514806 858 leader_election.cc:304] T 26c8ed083cea4dc0acca80d00b11b29b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:32.515202 2357 raft_consensus.cc:2747] T 26c8ed083cea4dc0acca80d00b11b29b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:32.520879 2118 ts_tablet_manager.cc:1905] T 75345ea2a3dc40fc8c9f8090d8839d2d P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:32.528937 1150 tablet_bootstrap.cc:492] T 3a2bc5ad1cd243daa3c92fceca8ba414 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:32.531649 1150 ts_tablet_manager.cc:1397] T 3a2bc5ad1cd243daa3c92fceca8ba414 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.096s user 0.006s sys 0.001s
I20250901 01:52:32.541054 1150 raft_consensus.cc:357] T 3a2bc5ad1cd243daa3c92fceca8ba414 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.541414 1150 raft_consensus.cc:383] T 3a2bc5ad1cd243daa3c92fceca8ba414 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:32.541517 1150 raft_consensus.cc:738] T 3a2bc5ad1cd243daa3c92fceca8ba414 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:32.541829 1150 consensus_queue.cc:260] T 3a2bc5ad1cd243daa3c92fceca8ba414 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.519764 1151 tablet_bootstrap.cc:492] T 246b33338de647689d7404c70676e76f P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:32.542321 1151 ts_tablet_manager.cc:1397] T 246b33338de647689d7404c70676e76f P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.095s user 0.009s sys 0.009s
I20250901 01:52:32.543006 1151 raft_consensus.cc:357] T 246b33338de647689d7404c70676e76f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.543150 1151 raft_consensus.cc:383] T 246b33338de647689d7404c70676e76f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:32.543192 1151 raft_consensus.cc:738] T 246b33338de647689d7404c70676e76f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:32.543370 1151 consensus_queue.cc:260] T 246b33338de647689d7404c70676e76f P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.543841 1151 ts_tablet_manager.cc:1428] T 246b33338de647689d7404c70676e76f P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250901 01:52:32.544108 1151 tablet_bootstrap.cc:492] T 410d44e5f1974ed69e5dee65e98856a4 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:32.544863 1150 ts_tablet_manager.cc:1428] T 3a2bc5ad1cd243daa3c92fceca8ba414 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250901 01:52:32.545154 1150 tablet_bootstrap.cc:492] T ced4813805004bba9c4667a79ac201fc P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:32.545885 1151 tablet_bootstrap.cc:654] T 410d44e5f1974ed69e5dee65e98856a4 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:32.541558 992 consensus_peers.cc:489] T df01d5df9fc2424eaf9de4e68e781803 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250901 01:52:32.546876 992 consensus_peers.cc:489] T 81c36693199945d98465579b509852e2 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250901 01:52:32.548671 2113 rpcz_store.cc:276] 0901 01:52:26.733184 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.733230 (+ 46us) service_pool.cc:225] Handling call
0901 01:52:32.514318 (+5781088us) ts_tablet_manager.cc:1145] Deleting tablet 6741c56499b44f648fb032c8cf50205b
0901 01:52:32.514327 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:32.514381 (+ 54us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5780958}
W20250901 01:52:32.553049 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 6741c56499b44f648fb032c8cf50205b because the tablet was not found. No further retry: Not found: Tablet not found: 6741c56499b44f648fb032c8cf50205b
I20250901 01:52:32.553532 1150 tablet_bootstrap.cc:654] T ced4813805004bba9c4667a79ac201fc P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:32.556183 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 193) took 5822 ms. Trace:
I20250901 01:52:32.556270 2113 rpcz_store.cc:276] 0901 01:52:26.733789 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.733829 (+ 40us) service_pool.cc:225] Handling call
0901 01:52:32.556084 (+5822255us) ts_tablet_manager.cc:1145] Deleting tablet 46bfbdff3f904715a8c24907b20fcab0
0901 01:52:32.556094 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:32.556166 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5822128}
W20250901 01:52:32.556725 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 46bfbdff3f904715a8c24907b20fcab0 because the tablet was not found. No further retry: Not found: Tablet not found: 46bfbdff3f904715a8c24907b20fcab0
I20250901 01:52:32.560070 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 195) took 5825 ms. Trace:
I20250901 01:52:32.562572 1491 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 81ce448b148447ddbc41875d863e0518 (table default.Table_6e461d9cb0ae4dd095975bbc21533794 [id=49673fa73f8c436e9b8026efc4f3cbbc]). Expected version 1 got 0
I20250901 01:52:32.562964 1491 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 315c15cf0afc44d4a79cb29294281d60 (table default.Table_7aa28ad9ee8c49859af40f9ccb61af33 [id=aed8d4b4fa6a4fbf850cf5718610108f]). Expected version 1 got 0
I20250901 01:52:32.563277 1491 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet ed1b41340ce742a88c3e4fd85cd25ca6 (table default.Table_ce2f3b515c8f4c1bb557c5ed84b48dba [id=0baa4993b72e4a8d94ee3059c9d2db6c]). Expected version 2 got 0
I20250901 01:52:32.560191 2113 rpcz_store.cc:276] 0901 01:52:26.734895 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.734937 (+ 42us) service_pool.cc:225] Handling call
0901 01:52:32.557189 (+5822252us) ts_tablet_manager.cc:1145] Deleting tablet 3c12d6a70ac24dce82e851782e6fe5aa
0901 01:52:32.557198 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:32.560051 (+ 2853us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5822136}
I20250901 01:52:32.563443 1491 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet c83ce95fd6fa41478de0e2168be201b1 (table default.Table_a525265208584713b1b3465d63130cd9 [id=bfd87b9b708449d3b16e3e0639561a49]). Expected version 1 got 0
I20250901 01:52:32.563845 2113 tablet_replica.cc:331] T 0dbdcb76eeba4fefa56e7835c23c0316 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
W20250901 01:52:32.563835 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 3c12d6a70ac24dce82e851782e6fe5aa because the tablet was not found. No further retry: Not found: Tablet not found: 3c12d6a70ac24dce82e851782e6fe5aa
I20250901 01:52:32.564062 1491 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet c6f7c525f59b49938b697de98d4b5835 (table default.Table_db61e055dbd7450898e82c4c0c4763be [id=aabca67f6f754820977ac9c4c20f3e4f]). Expected version 1 got 0
I20250901 01:52:32.565814 2118 ts_tablet_manager.cc:1918] T 75345ea2a3dc40fc8c9f8090d8839d2d P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:32.564133 2113 raft_consensus.cc:2241] T 0dbdcb76eeba4fefa56e7835c23c0316 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:32.566134 2113 raft_consensus.cc:2270] T 0dbdcb76eeba4fefa56e7835c23c0316 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:32.566846 2113 ts_tablet_manager.cc:1905] T 0dbdcb76eeba4fefa56e7835c23c0316 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:32.568365 2118 log.cc:1199] T 75345ea2a3dc40fc8c9f8090d8839d2d P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/75345ea2a3dc40fc8c9f8090d8839d2d
I20250901 01:52:32.569049 2118 ts_tablet_manager.cc:1939] T 75345ea2a3dc40fc8c9f8090d8839d2d P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:32.570252 2357 raft_consensus.cc:491] T 2df2240631844defa433270a3829d65b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:32.571053 2357 raft_consensus.cc:513] T 2df2240631844defa433270a3829d65b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.571710 2357 leader_election.cc:290] T 2df2240631844defa433270a3829d65b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:32.573364 1055 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2df2240631844defa433270a3829d65b" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:32.573628 858 leader_election.cc:336] T 2df2240631844defa433270a3829d65b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:32.573873 2357 raft_consensus.cc:491] T 3a2bc5ad1cd243daa3c92fceca8ba414 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:32.573978 2357 raft_consensus.cc:513] T 3a2bc5ad1cd243daa3c92fceca8ba414 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.574503 2357 leader_election.cc:290] T 3a2bc5ad1cd243daa3c92fceca8ba414 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:32.574739 1151 tablet_bootstrap.cc:492] T 410d44e5f1974ed69e5dee65e98856a4 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
W20250901 01:52:32.574853 859 leader_election.cc:343] T 2df2240631844defa433270a3829d65b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:32.574909 1151 ts_tablet_manager.cc:1397] T 410d44e5f1974ed69e5dee65e98856a4 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.004s
I20250901 01:52:32.574945 859 leader_election.cc:304] T 2df2240631844defa433270a3829d65b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:32.576010 2357 raft_consensus.cc:2747] T 2df2240631844defa433270a3829d65b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:32.576661 858 leader_election.cc:336] T 3a2bc5ad1cd243daa3c92fceca8ba414 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:32.576617 1151 raft_consensus.cc:357] T 410d44e5f1974ed69e5dee65e98856a4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.576753 1151 raft_consensus.cc:383] T 410d44e5f1974ed69e5dee65e98856a4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:32.576799 1151 raft_consensus.cc:738] T 410d44e5f1974ed69e5dee65e98856a4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:32.576979 1151 consensus_queue.cc:260] T 410d44e5f1974ed69e5dee65e98856a4 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.577428 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 161) took 5298 ms. Trace:
I20250901 01:52:32.577507 2118 rpcz_store.cc:276] 0901 01:52:27.279054 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:27.279109 (+ 55us) service_pool.cc:225] Handling call
0901 01:52:32.450317 (+5171208us) ts_tablet_manager.cc:1145] Deleting tablet 75345ea2a3dc40fc8c9f8090d8839d2d
0901 01:52:32.450324 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:32.535555 (+ 85231us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:32.565676 (+ 30121us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:32.577408 (+ 11732us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5171034}
I20250901 01:52:32.578001 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 75345ea2a3dc40fc8c9f8090d8839d2d (table default.Table_9734ae6c72e240c698b66ea83c536249 [id=f6d2701cc218463a9d1847e617bfec46]) successfully deleted
I20250901 01:52:32.578809 1150 tablet_bootstrap.cc:492] T ced4813805004bba9c4667a79ac201fc P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:32.578917 1150 ts_tablet_manager.cc:1397] T ced4813805004bba9c4667a79ac201fc P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.034s user 0.002s sys 0.002s
I20250901 01:52:32.579525 1150 raft_consensus.cc:357] T ced4813805004bba9c4667a79ac201fc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.579800 1150 raft_consensus.cc:383] T ced4813805004bba9c4667a79ac201fc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:32.579881 1150 raft_consensus.cc:738] T ced4813805004bba9c4667a79ac201fc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:32.580066 1150 consensus_queue.cc:260] T ced4813805004bba9c4667a79ac201fc P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.580624 1150 ts_tablet_manager.cc:1428] T ced4813805004bba9c4667a79ac201fc P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250901 01:52:32.580605 1151 ts_tablet_manager.cc:1428] T 410d44e5f1974ed69e5dee65e98856a4 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250901 01:52:32.580917 1150 tablet_bootstrap.cc:492] T 315c15cf0afc44d4a79cb29294281d60 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:32.581312 1151 tablet_bootstrap.cc:492] T 75345ea2a3dc40fc8c9f8090d8839d2d P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:32.582559 1150 tablet_bootstrap.cc:654] T 315c15cf0afc44d4a79cb29294281d60 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:32.583868 1055 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3a2bc5ad1cd243daa3c92fceca8ba414" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:32.584204 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 160) took 5305 ms. Trace:
I20250901 01:52:32.584285 2118 rpcz_store.cc:276] 0901 01:52:27.278906 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:27.279319 (+ 413us) service_pool.cc:225] Handling call
0901 01:52:32.584116 (+5304797us) ts_tablet_manager.cc:1145] Deleting tablet f3962b471bdb49aea6e9fa95f1f9dfe2
0901 01:52:32.584126 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:32.584188 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5304630}
W20250901 01:52:32.584749 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet f3962b471bdb49aea6e9fa95f1f9dfe2 because the tablet was not found. No further retry: Not found: Tablet not found: f3962b471bdb49aea6e9fa95f1f9dfe2
W20250901 01:52:32.585058 859 leader_election.cc:343] T 3a2bc5ad1cd243daa3c92fceca8ba414 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:32.585155 859 leader_election.cc:304] T 3a2bc5ad1cd243daa3c92fceca8ba414 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:32.585740 2357 raft_consensus.cc:2747] T 3a2bc5ad1cd243daa3c92fceca8ba414 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:32.587865 2357 raft_consensus.cc:491] T 7f46e9ddadaf4ff986d8c991178daa0c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:32.587972 2357 raft_consensus.cc:513] T 7f46e9ddadaf4ff986d8c991178daa0c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:32.588544 2357 leader_election.cc:290] T 7f46e9ddadaf4ff986d8c991178daa0c P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:32.590137 858 leader_election.cc:336] T 7f46e9ddadaf4ff986d8c991178daa0c P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:32.590922 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 163) took 5305 ms. Trace:
I20250901 01:52:32.591051 2118 rpcz_store.cc:276] 0901 01:52:27.285541 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:27.285606 (+ 65us) service_pool.cc:225] Handling call
0901 01:52:32.590835 (+5305229us) ts_tablet_manager.cc:1145] Deleting tablet 831e42de95854541a49ae6b3e039d3e9
0901 01:52:32.590843 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:32.590906 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5304976}
W20250901 01:52:32.591545 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 831e42de95854541a49ae6b3e039d3e9 because the tablet was not found. No further retry: Not found: Tablet not found: 831e42de95854541a49ae6b3e039d3e9
I20250901 01:52:32.591908 2118 tablet_replica.cc:331] T 088ef4861dc14f81ae470c3aaee5b4e1 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:32.593158 1055 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f46e9ddadaf4ff986d8c991178daa0c" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:32.593500 2118 raft_consensus.cc:2241] T 088ef4861dc14f81ae470c3aaee5b4e1 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:32.593696 2118 raft_consensus.cc:2270] T 088ef4861dc14f81ae470c3aaee5b4e1 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250901 01:52:32.594388 859 leader_election.cc:343] T 7f46e9ddadaf4ff986d8c991178daa0c P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:32.594480 859 leader_election.cc:304] T 7f46e9ddadaf4ff986d8c991178daa0c P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:32.594746 1151 tablet_bootstrap.cc:654] T 75345ea2a3dc40fc8c9f8090d8839d2d P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:32.594990 2357 raft_consensus.cc:2747] T 7f46e9ddadaf4ff986d8c991178daa0c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:32.595618 2118 ts_tablet_manager.cc:1905] T 088ef4861dc14f81ae470c3aaee5b4e1 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:32.595973 2412 raft_consensus.cc:491] T e8e7d2a72f704f52a8f8d46911a8a9e4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:32.596076 2412 raft_consensus.cc:513] T e8e7d2a72f704f52a8f8d46911a8a9e4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.596571 2412 leader_election.cc:290] T e8e7d2a72f704f52a8f8d46911a8a9e4 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:32.599859 2113 ts_tablet_manager.cc:1918] T 0dbdcb76eeba4fefa56e7835c23c0316 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:32.602522 1491 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 5f5396045979450193d461ba1334a5c9 (table default.Table_2b946a63bd574ac0971fa3d3b9c9a6a4 [id=9c3930e9fcfe4b49ba4bea67a2b0ad07]). Expected version 2 got 0
I20250901 01:52:32.602903 1491 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 2973b59bb1df41d18d9a5b6a29ae336f (table default.Table_7790af96855e45c0a316ba3a4a87c9ed [id=746845bd84c6444b8a1e5683eaba4a68]). Expected version 1 got 0
I20250901 01:52:32.603080 1491 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 5d68ea97c71942c7b55b967ba870e0d7 (table default.Table_f610cb75906d4c5cb3b57cfdb80ecefc [id=36915a3e843340ba8ef3578ac35a9af4]). Expected version 1 got 0
I20250901 01:52:32.603338 1491 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet ae21b9ae2e3b472eb67fb5f96388a50f (table default.Table_046289f393ff432aae0b033bdec1fb17 [id=db57b18a9ce24e068b362f5fbe940531]). Expected version 1 got 0
I20250901 01:52:32.603521 1491 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet e9f1f892658b4736859ff507536920a1 (table default.Table_8825dd28afbe464193eb1b668cc9326a [id=9aa62cb634314a0a849612c54c1220d9]). Expected version 1 got 0
I20250901 01:52:32.603684 1491 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 93bcac1fd95d44dc927abfca81692921 (table default.Table_b8f8ee74648c406eaf589d3ab80934db [id=b8f69313439543be9c11eea88dfa69c5]). Expected version 1 got 0
I20250901 01:52:32.603919 1491 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 61ddcf86c9f34726939cc116434f0822 (table default.Table_63d0f2ead9d946d39ce40162ebd769d6 [id=f9dcf4937b1143248bfa9d88053fce17]). Expected version 3 got 0
I20250901 01:52:32.604461 1491 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 0e4009963cfe42d6ba164fc828897f09 (table default.Table_1adc70cccd2f4cb3a79df96861cb4181 [id=24fffc7ebf5a47bdb37211910301acf7]). Expected version 1 got 0
I20250901 01:52:32.604635 1491 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 45f6e8ba99b74898a2922b36768c6c36 (table default.Table_938421db3cf74cf8b878089bac931818 [id=9ce26dcf307149b595a43b6ac4562fba]). Expected version 1 got 0
I20250901 01:52:32.604802 1491 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet dabebc77f72e4bb19e264be2f496202a (table default.Table_2755f7caf53340afbd230181643366db [id=a4d0af97df5949cca940a81a41b8e454]). Expected version 1 got 0
I20250901 01:52:32.604955 1491 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 98c56eb62ad84f79b04f17a43d44c479 (table default.Table_50ff85e8f1314f4499e76299441bc110 [id=00e7f7fe6dc44f2787b9dd8e3cbf4a51]). Expected version 1 got 0
I20250901 01:52:32.605180 1491 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 6c0abbfc8d904dce989ce51961e4237d (table default.Table_3f9422b08b0349b9a5466218f0fe1b06 [id=f0dfb3811e744bd4ba6d59195cebde6f]). Expected version 2 got 0
I20250901 01:52:32.605443 1491 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 508406601c994e42a5c5b1f665e74957 (table default.Table_345b7f9fe8214aa6878f20a166125364 [id=4aabc328f6374ec5b1dea26f778208b2]). Expected version 1 got 0
I20250901 01:52:32.606048 1491 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet b164b4a67a9143ceaff69b0ce49d63fe (table default.Table_4505d77e4269443995012fef50c7e7b5 [id=c1b1e9e3876244f68d31c63ce7bdebf1]). Expected version 2 got 0
I20250901 01:52:32.606529 1491 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 858218c4427c41e980b2b5c969a5af40 (table default.Table_6d47266af0ed433eb1f2abbc81d18e69 [id=264cc469a73841e79538843479a165ab]). Expected version 1 got 0
I20250901 01:52:32.606894 1491 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 4cde5f4f02044b7bbd7d43b49aac4ae8 (table default.Table_d53a456dbca64daf92aeb5fb4517b417 [id=77ae3376281349e087a3b3cee41f2a09]). Expected version 1 got 0
I20250901 01:52:32.607066 1491 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 7954393c8c1c4457ab8bb723c03a9c8b (table default.Table_0cb2185875b646dcac4d1f6f2f4cffd5 [id=543f2074598d4c29972e564be1123fca]). Expected version 6 got 5
I20250901 01:52:32.607605 1491 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 615b2306fca84003b264da9d5f95ddab (table default.Table_76438e5616384140b57dafdc896f11c5 [id=cecdd8d6ca6d445c9fd82decd2a95366]). Expected version 3 got 0
I20250901 01:52:32.607903 1491 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet d0c3c363dda444519a17dca63ad9caa7 (table default.Table_42422c44ca254d8ca989134c2433391f [id=8c1b263824ec4c15a5ec424364c78895]). Expected version 3 got 0
I20250901 01:52:32.608158 1491 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 33f637f6a8134215bb1995ec38440073 (table default.Table_3d5afc22c4e14d6990aea31564dfde20 [id=557afe29976d40f6bfb6ae4ca348f887]). Expected version 2 got 0
W20250901 01:52:32.601678 992 leader_election.cc:343] T e8e7d2a72f704f52a8f8d46911a8a9e4 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: e8e7d2a72f704f52a8f8d46911a8a9e4
I20250901 01:52:32.609501 2113 log.cc:1199] T 0dbdcb76eeba4fefa56e7835c23c0316 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/0dbdcb76eeba4fefa56e7835c23c0316
W20250901 01:52:32.612509 992 leader_election.cc:336] T e8e7d2a72f704f52a8f8d46911a8a9e4 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:32.612665 992 leader_election.cc:304] T e8e7d2a72f704f52a8f8d46911a8a9e4 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:32.612775 2113 ts_tablet_manager.cc:1939] T 0dbdcb76eeba4fefa56e7835c23c0316 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:32.613991 1150 tablet_bootstrap.cc:492] T 315c15cf0afc44d4a79cb29294281d60 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:32.614202 1150 ts_tablet_manager.cc:1397] T 315c15cf0afc44d4a79cb29294281d60 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.000s
I20250901 01:52:32.614943 1150 raft_consensus.cc:357] T 315c15cf0afc44d4a79cb29294281d60 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.615155 1150 raft_consensus.cc:383] T 315c15cf0afc44d4a79cb29294281d60 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:32.615250 1150 raft_consensus.cc:738] T 315c15cf0afc44d4a79cb29294281d60 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:32.615507 1150 consensus_queue.cc:260] T 315c15cf0afc44d4a79cb29294281d60 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.616164 1150 ts_tablet_manager.cc:1428] T 315c15cf0afc44d4a79cb29294281d60 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250901 01:52:32.616489 1150 tablet_bootstrap.cc:492] T 7304937048ca48a7bd153f3cdcfaeff0 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:32.618672 1150 tablet_bootstrap.cc:654] T 7304937048ca48a7bd153f3cdcfaeff0 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:32.612891 2412 raft_consensus.cc:2747] T e8e7d2a72f704f52a8f8d46911a8a9e4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:32.618445 1151 tablet_bootstrap.cc:492] T 75345ea2a3dc40fc8c9f8090d8839d2d P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:32.619109 1761 raft_consensus.cc:491] T becbaae4fd7e44faa679e9479465a7c4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:32.620246 1761 raft_consensus.cc:513] T becbaae4fd7e44faa679e9479465a7c4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.620255 1151 ts_tablet_manager.cc:1397] T 75345ea2a3dc40fc8c9f8090d8839d2d P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.000s
I20250901 01:52:32.620888 1761 leader_election.cc:290] T becbaae4fd7e44faa679e9479465a7c4 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:32.621171 1151 raft_consensus.cc:357] T 75345ea2a3dc40fc8c9f8090d8839d2d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.621328 1151 raft_consensus.cc:383] T 75345ea2a3dc40fc8c9f8090d8839d2d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:32.621378 1151 raft_consensus.cc:738] T 75345ea2a3dc40fc8c9f8090d8839d2d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:32.621555 1151 consensus_queue.cc:260] T 75345ea2a3dc40fc8c9f8090d8839d2d P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:32.621748 992 leader_election.cc:343] T becbaae4fd7e44faa679e9479465a7c4 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: becbaae4fd7e44faa679e9479465a7c4
I20250901 01:52:32.622016 1151 ts_tablet_manager.cc:1428] T 75345ea2a3dc40fc8c9f8090d8839d2d P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250901 01:52:32.622853 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 194) took 5888 ms. Trace:
I20250901 01:52:32.622938 2113 rpcz_store.cc:276] 0901 01:52:26.734752 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.734789 (+ 37us) service_pool.cc:225] Handling call
0901 01:52:32.563794 (+5829005us) ts_tablet_manager.cc:1145] Deleting tablet 0dbdcb76eeba4fefa56e7835c23c0316
0901 01:52:32.563805 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:32.587576 (+ 23771us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:32.599751 (+ 12175us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:32.622837 (+ 23086us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5828695}
I20250901 01:52:32.623324 1151 tablet_bootstrap.cc:492] T 268439ff8d3a4f7a89f3d6395f2079c5 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
W20250901 01:52:32.624214 1486 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 508406601c994e42a5c5b1f665e74957 consensus state has no leader
I20250901 01:52:32.625118 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8a05a6761aae4401a5bfdbeac2287fe3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:32.625308 1151 tablet_bootstrap.cc:654] T 268439ff8d3a4f7a89f3d6395f2079c5 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:32.625311 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 0dbdcb76eeba4fefa56e7835c23c0316 (table default.Table_97f8d5c204174c918a795498e47d404d [id=d74c256799d64024b869f8076b857f63]) successfully deleted
I20250901 01:52:32.626081 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_bb44e6b5af4b47f0bf20b97ae335bd9f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:32.626263 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 198) took 5878 ms. Trace:
I20250901 01:52:32.626399 2113 rpcz_store.cc:276] 0901 01:52:26.748118 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.748152 (+ 34us) service_pool.cc:225] Handling call
0901 01:52:32.626169 (+5878017us) ts_tablet_manager.cc:1145] Deleting tablet 7f64ebd080944190be2e64d02d70a8a7
0901 01:52:32.626177 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:32.626247 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5877871}
I20250901 01:52:32.626852 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 199) took 5877 ms. Trace:
I20250901 01:52:32.626823 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e9927b646c8e41c39a193bf89b98fc91"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:32.626916 2113 rpcz_store.cc:276] 0901 01:52:26.748921 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.748959 (+ 38us) service_pool.cc:225] Handling call
0901 01:52:32.626768 (+5877809us) ts_tablet_manager.cc:1145] Deleting tablet 25ce0fe8c9544f828694184d19d13df9
0901 01:52:32.626778 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:32.626838 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5877682}
I20250901 01:52:32.627234 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 202) took 5862 ms. Trace:
I20250901 01:52:32.627288 2113 rpcz_store.cc:276] 0901 01:52:26.764344 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.764390 (+ 46us) service_pool.cc:225] Handling call
0901 01:52:32.627175 (+5862785us) ts_tablet_manager.cc:1145] Deleting tablet 6cf09e8f299548dba2cce16d7b565bad
0901 01:52:32.627182 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:32.627224 (+ 42us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5862574}
I20250901 01:52:32.627466 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9e07a35936d642bd84afed7f3c2721ba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:32.627583 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 201) took 5863 ms. Trace:
I20250901 01:52:32.627650 2113 rpcz_store.cc:276] 0901 01:52:26.764258 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.764924 (+ 666us) service_pool.cc:225] Handling call
0901 01:52:32.627519 (+5862595us) ts_tablet_manager.cc:1145] Deleting tablet af2c4bcbb00741f789edee109252d8cb
0901 01:52:32.627526 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:32.627574 (+ 48us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5862450}
I20250901 01:52:32.627959 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 203) took 5863 ms. Trace:
I20250901 01:52:32.628015 2113 rpcz_store.cc:276] 0901 01:52:26.764866 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.765142 (+ 276us) service_pool.cc:225] Handling call
0901 01:52:32.627890 (+5862748us) ts_tablet_manager.cc:1145] Deleting tablet b7b802f976f3471faaaecc964ec87e48
0901 01:52:32.627898 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:32.627948 (+ 50us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5862625}
I20250901 01:52:32.628356 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 204) took 5862 ms. Trace:
I20250901 01:52:32.628428 2113 rpcz_store.cc:276] 0901 01:52:26.765553 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.765598 (+ 45us) service_pool.cc:225] Handling call
0901 01:52:32.628271 (+5862673us) ts_tablet_manager.cc:1145] Deleting tablet 764bfeb2a7d14c40815b5ec1b07bdd62
0901 01:52:32.628278 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:32.628346 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5862513}
I20250901 01:52:32.628636 2118 ts_tablet_manager.cc:1918] T 088ef4861dc14f81ae470c3aaee5b4e1 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:32.628819 2118 log.cc:1199] T 088ef4861dc14f81ae470c3aaee5b4e1 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/088ef4861dc14f81ae470c3aaee5b4e1
I20250901 01:52:32.628837 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 205) took 5862 ms. Trace:
I20250901 01:52:32.628917 2113 rpcz_store.cc:276] 0901 01:52:26.765903 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.765943 (+ 40us) service_pool.cc:225] Handling call
0901 01:52:32.628741 (+5862798us) ts_tablet_manager.cc:1145] Deleting tablet 4855448edb4b40c184838aa7f81c112b
0901 01:52:32.628749 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:32.628822 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5862664}
I20250901 01:52:32.629123 2118 ts_tablet_manager.cc:1939] T 088ef4861dc14f81ae470c3aaee5b4e1 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:32.629257 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 207) took 5862 ms. Trace:
I20250901 01:52:32.629315 2113 rpcz_store.cc:276] 0901 01:52:26.766630 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.766678 (+ 48us) service_pool.cc:225] Handling call
0901 01:52:32.629180 (+5862502us) ts_tablet_manager.cc:1145] Deleting tablet c49d1143d54b4c319739488906b0e271
0901 01:52:32.629188 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:32.629247 (+ 59us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5862349}
I20250901 01:52:32.630241 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 208) took 5862 ms. Trace:
I20250901 01:52:32.630313 2113 rpcz_store.cc:276] 0901 01:52:26.766958 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.767009 (+ 51us) service_pool.cc:225] Handling call
0901 01:52:32.629558 (+5862549us) ts_tablet_manager.cc:1145] Deleting tablet 438ae684af234c868a419fbab43b0b98
0901 01:52:32.629566 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:32.629648 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5862405}
I20250901 01:52:32.630481 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 162) took 5345 ms. Trace:
I20250901 01:52:32.630650 2113 tablet_replica.cc:331] T 246b33338de647689d7404c70676e76f P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:32.630561 2118 rpcz_store.cc:276] 0901 01:52:27.285436 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:27.285780 (+ 344us) service_pool.cc:225] Handling call
0901 01:52:32.591863 (+5306083us) ts_tablet_manager.cc:1145] Deleting tablet 088ef4861dc14f81ae470c3aaee5b4e1
0901 01:52:32.591872 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:32.610947 (+ 19075us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:32.628496 (+ 17549us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:32.630458 (+ 1962us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5305935}
I20250901 01:52:32.630815 2113 raft_consensus.cc:2241] T 246b33338de647689d7404c70676e76f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:32.630936 2113 raft_consensus.cc:2270] T 246b33338de647689d7404c70676e76f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:32.630975 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 088ef4861dc14f81ae470c3aaee5b4e1 (table default.Table_9e04f976a93f4d30baccf90fa3622dfe [id=d68960dbe1154bd8a8016f9a5718e5ad]) successfully deleted
I20250901 01:52:32.631515 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 164) took 5345 ms. Trace:
I20250901 01:52:32.631596 2118 rpcz_store.cc:276] 0901 01:52:27.285660 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:27.288459 (+ 2799us) service_pool.cc:225] Handling call
0901 01:52:32.631423 (+5342964us) ts_tablet_manager.cc:1145] Deleting tablet 64fb46b942ff4e4c8749986ecc3dfe65
0901 01:52:32.631432 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:32.631496 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5342772}
W20250901 01:52:32.632047 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 64fb46b942ff4e4c8749986ecc3dfe65 because the tablet was not found. No further retry: Not found: Tablet not found: 64fb46b942ff4e4c8749986ecc3dfe65
W20250901 01:52:32.626824 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 7f64ebd080944190be2e64d02d70a8a7 because the tablet was not found. No further retry: Not found: Tablet not found: 7f64ebd080944190be2e64d02d70a8a7
W20250901 01:52:32.632861 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 25ce0fe8c9544f828694184d19d13df9 because the tablet was not found. No further retry: Not found: Tablet not found: 25ce0fe8c9544f828694184d19d13df9
W20250901 01:52:32.633057 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 6cf09e8f299548dba2cce16d7b565bad because the tablet was not found. No further retry: Not found: Tablet not found: 6cf09e8f299548dba2cce16d7b565bad
W20250901 01:52:32.633251 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet af2c4bcbb00741f789edee109252d8cb because the tablet was not found. No further retry: Not found: Tablet not found: af2c4bcbb00741f789edee109252d8cb
I20250901 01:52:32.633275 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 165) took 5329 ms. Trace:
W20250901 01:52:32.633417 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet b7b802f976f3471faaaecc964ec87e48 because the tablet was not found. No further retry: Not found: Tablet not found: b7b802f976f3471faaaecc964ec87e48
I20250901 01:52:32.633344 2118 rpcz_store.cc:276] 0901 01:52:27.304021 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:27.304067 (+ 46us) service_pool.cc:225] Handling call
0901 01:52:32.633199 (+5329132us) ts_tablet_manager.cc:1145] Deleting tablet 7a607fac57994001b2304e5d81c9decc
0901 01:52:32.633207 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:32.633262 (+ 55us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5328925}
W20250901 01:52:32.633594 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 764bfeb2a7d14c40815b5ec1b07bdd62 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 764bfeb2a7d14c40815b5ec1b07bdd62 already in progress: creating tablet
W20250901 01:52:32.633702 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 7a607fac57994001b2304e5d81c9decc because the tablet was not found. No further retry: Not found: Tablet not found: 7a607fac57994001b2304e5d81c9decc
W20250901 01:52:32.633791 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 4855448edb4b40c184838aa7f81c112b because the tablet was not found. No further retry: Not found: Tablet not found: 4855448edb4b40c184838aa7f81c112b
W20250901 01:52:32.634022 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet c49d1143d54b4c319739488906b0e271 because the tablet was not found. No further retry: Not found: Tablet not found: c49d1143d54b4c319739488906b0e271
I20250901 01:52:32.634193 2113 ts_tablet_manager.cc:1905] T 246b33338de647689d7404c70676e76f P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:32.634238 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 438ae684af234c868a419fbab43b0b98 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 438ae684af234c868a419fbab43b0b98 already in progress: creating tablet
I20250901 01:52:32.633160 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9cf7bc928ae2415a9dbdc1a132f6afcc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:32.635519 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 167) took 5313 ms. Trace:
I20250901 01:52:32.635600 2118 rpcz_store.cc:276] 0901 01:52:27.322456 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:27.322499 (+ 43us) service_pool.cc:225] Handling call
0901 01:52:32.635435 (+5312936us) ts_tablet_manager.cc:1145] Deleting tablet e9b179dba2cc49428ba8090c962d6136
0901 01:52:32.635445 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:32.635501 (+ 56us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5312765}
W20250901 01:52:32.636001 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet e9b179dba2cc49428ba8090c962d6136 because the tablet was not found. No further retry: Not found: Tablet not found: e9b179dba2cc49428ba8090c962d6136
I20250901 01:52:32.635954 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_94c2c079e7074a938567bfe70199922f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:32.637589 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8cb9efc986854b78a316887c7a64c239"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:32.639072 2118 tablet_replica.cc:331] T 85322a864d154f299f2d670fdd2a7bfb P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:32.640200 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1ea1d5e2a3d242018c6a3eb2702ca059"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:32.642050 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_93fe106ff9674e578ba0ff7631c2adbd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:32.639287 2118 raft_consensus.cc:2241] T 85322a864d154f299f2d670fdd2a7bfb P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
W20250901 01:52:32.643762 992 leader_election.cc:336] T becbaae4fd7e44faa679e9479465a7c4 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:32.643940 992 leader_election.cc:304] T becbaae4fd7e44faa679e9479465a7c4 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:32.644196 1761 raft_consensus.cc:2747] T becbaae4fd7e44faa679e9479465a7c4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:32.645063 1486 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_21b93b5fe2ce49c6bc637d8cab41fc06" } new_table_name: "default.Table_2e67ad583a0b413e9190cb55137ccf3a" modify_external_catalogs: true
I20250901 01:52:32.643842 2118 raft_consensus.cc:2270] T 85322a864d154f299f2d670fdd2a7bfb P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:32.646090 2118 ts_tablet_manager.cc:1905] T 85322a864d154f299f2d670fdd2a7bfb P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:32.648233 992 consensus_peers.cc:489] T 15e4d972e8254cf899c3e0b71b6e2bd9 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250901 01:52:32.650858 1485 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 6ba8241644984394ada95b561e76467e
W20250901 01:52:32.651090 1485 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 6ba8241644984394ada95b561e76467e: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:32.651736 1485 master_service.cc:948] ReplaceTablet: received request to replace tablet 56f2c4d2e6ab4bb4bdd6d025843589b6 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:32.652458 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 6ba8241644984394ada95b561e76467e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:31 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:32.652981 1103 heartbeater.cc:499] Master 127.0.104.60:44231 was elected leader, sending a full tablet report...
I20250901 01:52:32.653851 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 6ba8241644984394ada95b561e76467e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:31 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:32.654752 1488 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 182d99cb66024b66b16f1a4e76084260
I20250901 01:52:32.654974 1488 master_service.cc:948] ReplaceTablet: received request to replace tablet 7f46e9ddadaf4ff986d8c991178daa0c from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:32.655313 1490 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 66719d8e9c614657820440ebe8ce7c1e
I20250901 01:52:32.655618 1490 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_9163155cb611406a8ce0d5ae600a15c8" } modify_external_catalogs: true
I20250901 01:52:32.655786 1490 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_9163155cb611406a8ce0d5ae600a15c8" } modify_external_catalogs: true
W20250901 01:52:32.657017 1489 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 45f6e8ba99b74898a2922b36768c6c36 consensus state has no leader
I20250901 01:52:32.657393 1489 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_156d1feafd5246ff8ff1bb7b9ae444a5" } new_table_name: "default.Table_d8e839ac6396439197b8b727af6a31b7" modify_external_catalogs: true
I20250901 01:52:32.658875 1484 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 44af4764d4f64e44aec84ccb84be6f5f
W20250901 01:52:32.659042 1484 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 44af4764d4f64e44aec84ccb84be6f5f: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:32.659453 1484 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:33478 (request call id 1219) took 1090 ms. Trace:
I20250901 01:52:32.659560 1484 rpcz_store.cc:276] 0901 01:52:31.569248 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:31.569307 (+ 59us) service_pool.cc:225] Handling call
0901 01:52:31.589123 (+ 19816us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0901 01:52:31.589302 (+ 179us) catalog_manager.cc:2790] Modifying in-memory table state
0901 01:52:31.589373 (+ 71us) catalog_manager.cc:2799] Locking tablets
0901 01:52:31.757491 (+168118us) catalog_manager.cc:2813] Removing table and tablets from system table
0901 01:52:31.849510 (+ 92019us) catalog_manager.cc:2831] Removing table from by-name map
0901 01:52:32.170322 (+320812us) catalog_manager.cc:2847] Committing in-memory state
0901 01:52:32.658802 (+488480us) catalog_manager.cc:2851] Aborting table tasks
0901 01:52:32.659390 (+ 588us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0901 01:52:31.831603 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:31.831736 (+ 133us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:31.831742 (+ 6us) write_op.cc:435] Acquired schema lock
0901 01:52:31.831745 (+ 3us) tablet.cc:660] Decoding operations
0901 01:52:31.831870 (+ 125us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:31.831881 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:31.831885 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0901 01:52:31.831982 (+ 97us) tablet.cc:699] Row locks acquired
0901 01:52:31.831985 (+ 3us) write_op.cc:260] PREPARE: finished
0901 01:52:31.832016 (+ 31us) write_op.cc:270] Start()
0901 01:52:31.832042 (+ 26us) write_op.cc:276] Timestamp: P: 1756691551831981 usec, L: 0
0901 01:52:31.832046 (+ 4us) op_driver.cc:348] REPLICATION: starting
0901 01:52:31.832199 (+ 153us) log.cc:844] Serialized 756 byte log entry
0901 01:52:31.847534 (+ 15335us) op_driver.cc:464] REPLICATION: finished
0901 01:52:31.847891 (+ 357us) write_op.cc:301] APPLY: starting
0901 01:52:31.847913 (+ 22us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:31.847971 (+ 58us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:31.848008 (+ 37us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:31.848082 (+ 74us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:31.848116 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0901 01:52:31.848123 (+ 7us) write_op.cc:312] APPLY: finished
0901 01:52:31.848200 (+ 77us) log.cc:844] Serialized 29 byte log entry
0901 01:52:31.848265 (+ 65us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:31.848286 (+ 21us) write_op.cc:454] Released schema lock
0901 01:52:31.848292 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0901 01:52:31.848371 (+ 79us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:31.848375 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":294,"apply.run_cpu_time_us":469,"apply.run_wall_time_us":469,"mutex_wait_us":64734,"num_ops":2,"prepare.queue_time_us":73513,"prepare.run_cpu_time_us":689,"prepare.run_wall_time_us":689,"raft.queue_time_us":1291,"raft.run_cpu_time_us":253,"raft.run_wall_time_us":253,"replication_time_us":15441}]]}
I20250901 01:52:32.660385 1484 master_service.cc:948] ReplaceTablet: received request to replace tablet bd9b59f0864a497faf281edd6077668d from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:32.661381 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 44af4764d4f64e44aec84ccb84be6f5f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:31 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:32.663405 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 44af4764d4f64e44aec84ccb84be6f5f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:31 UTC) from {username='slave'} at 127.0.0.1:35946
W20250901 01:52:32.664763 1483 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 315c15cf0afc44d4a79cb29294281d60 consensus state has no leader
I20250901 01:52:32.671276 969 heartbeater.cc:499] Master 127.0.104.60:44231 was elected leader, sending a full tablet report...
I20250901 01:52:32.674865 1492 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: default.Table_0cb2185875b646dcac4d1f6f2f4cffd5 [id=543f2074598d4c29972e564be1123fca] alter complete (version 6)
I20250901 01:52:32.683616 1761 raft_consensus.cc:491] T 410d44e5f1974ed69e5dee65e98856a4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:32.683784 1761 raft_consensus.cc:513] T 410d44e5f1974ed69e5dee65e98856a4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.684434 1761 leader_election.cc:290] T 410d44e5f1974ed69e5dee65e98856a4 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:32.687059 1474 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 5ddbc7e6820247a987f4d54757d1191d (table default.Table_946298d2a13646aeb4c189678909d1be [id=4ae8785dc52c4e89a1a97a1490356898], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 5ddbc7e6820247a987f4d54757d1191d consensus state has no leader
I20250901 01:52:32.695724 1485 master_service.cc:948] ReplaceTablet: received request to replace tablet 61ddcf86c9f34726939cc116434f0822 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:32.699407 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3d334603f7f34f8c88043b812ed6ba17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:32.700273 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f52c87929f23431b98c5e33dbc7b772f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:32.701066 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_74681205efca4d4582a31e509e762f3a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:32.701717 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1bc1234c34dc4cfe8e7c867894369160"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:32.702389 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5c653703d7ed4b99a6582f5d67397079"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:32.703194 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e5a0818a80844b268b5d85e65b917527"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:32.705363 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9c70d9cdba044b408fa4f50f1a291059"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:32.706548 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9a03893c0da94aea98aadc64f2807e5e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:32.715626 858 consensus_peers.cc:489] T 629db80a8ab444e582628578155638fd P 00c282f779c1477d920f24dc0dc16628 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250901 01:52:32.716487 2357 raft_consensus.cc:491] T 315c15cf0afc44d4a79cb29294281d60 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:32.716581 1151 tablet_bootstrap.cc:492] T 268439ff8d3a4f7a89f3d6395f2079c5 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:32.716717 1151 ts_tablet_manager.cc:1397] T 268439ff8d3a4f7a89f3d6395f2079c5 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.093s user 0.014s sys 0.021s
I20250901 01:52:32.716658 2357 raft_consensus.cc:513] T 315c15cf0afc44d4a79cb29294281d60 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.717342 2357 leader_election.cc:290] T 315c15cf0afc44d4a79cb29294281d60 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:32.717499 1151 raft_consensus.cc:357] T 268439ff8d3a4f7a89f3d6395f2079c5 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.717660 1151 raft_consensus.cc:383] T 268439ff8d3a4f7a89f3d6395f2079c5 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:32.717702 1151 raft_consensus.cc:738] T 268439ff8d3a4f7a89f3d6395f2079c5 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:32.717856 1151 consensus_queue.cc:260] T 268439ff8d3a4f7a89f3d6395f2079c5 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.718459 1151 ts_tablet_manager.cc:1428] T 268439ff8d3a4f7a89f3d6395f2079c5 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250901 01:52:32.718762 1151 tablet_bootstrap.cc:492] T da5f69c5b0ca4257ad2ff7918ee2af2e P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
W20250901 01:52:32.718976 858 leader_election.cc:336] T 315c15cf0afc44d4a79cb29294281d60 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:32.720120 1151 tablet_bootstrap.cc:654] T da5f69c5b0ca4257ad2ff7918ee2af2e P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:32.721154 2113 ts_tablet_manager.cc:1918] T 246b33338de647689d7404c70676e76f P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:32.722021 2113 log.cc:1199] T 246b33338de647689d7404c70676e76f P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/246b33338de647689d7404c70676e76f
I20250901 01:52:32.723300 2357 raft_consensus.cc:491] T c422881b40f54971a68530abfbd1945b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:32.723438 2357 raft_consensus.cc:513] T c422881b40f54971a68530abfbd1945b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:32.725303 858 leader_election.cc:336] T c422881b40f54971a68530abfbd1945b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:32.724992 2113 ts_tablet_manager.cc:1939] T 246b33338de647689d7404c70676e76f P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:32.728577 2357 leader_election.cc:290] T c422881b40f54971a68530abfbd1945b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:32.731479 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 209) took 5960 ms. Trace:
I20250901 01:52:32.733490 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_95832127b52847808ff4fb2335094145"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:32.732295 2113 rpcz_store.cc:276] 0901 01:52:26.770473 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.770517 (+ 44us) service_pool.cc:225] Handling call
0901 01:52:32.630600 (+5860083us) ts_tablet_manager.cc:1145] Deleting tablet 246b33338de647689d7404c70676e76f
0901 01:52:32.630619 (+ 19us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:32.663832 (+ 33213us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:32.720531 (+ 56699us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:32.731345 (+ 10814us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5859933}
I20250901 01:52:32.734174 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 246b33338de647689d7404c70676e76f (table default.Table_aee74198b51d4f089538c4f2bf3ce4dc [id=a7b1d2a8ace14ef8bcce0a3bed41e85d]) successfully deleted
I20250901 01:52:32.734526 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ca65d8e23d6a4e79af1ca3370b76cbd0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:32.735204 1484 master_service.cc:948] ReplaceTablet: received request to replace tablet 4cde5f4f02044b7bbd7d43b49aac4ae8 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:32.735781 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 210) took 5964 ms. Trace:
I20250901 01:52:32.736356 2113 rpcz_store.cc:276] 0901 01:52:26.771202 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.771246 (+ 44us) service_pool.cc:225] Handling call
0901 01:52:32.734974 (+5963728us) ts_tablet_manager.cc:1145] Deleting tablet 617cc33d0d23439cb5f1e0c31bf11281
0901 01:52:32.734986 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:32.735656 (+ 670us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5963464}
W20250901 01:52:32.737959 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 617cc33d0d23439cb5f1e0c31bf11281 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 617cc33d0d23439cb5f1e0c31bf11281 already in progress: creating tablet
I20250901 01:52:32.739615 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 211) took 5966 ms. Trace:
I20250901 01:52:32.741611 1150 tablet_bootstrap.cc:492] T 7304937048ca48a7bd153f3cdcfaeff0 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:32.741848 1150 ts_tablet_manager.cc:1397] T 7304937048ca48a7bd153f3cdcfaeff0 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.125s user 0.004s sys 0.011s
I20250901 01:52:32.742784 1150 raft_consensus.cc:357] T 7304937048ca48a7bd153f3cdcfaeff0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:32.743034 1150 raft_consensus.cc:383] T 7304937048ca48a7bd153f3cdcfaeff0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
E20250901 01:52:32.743104 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 0c88c339e59a4c1da21756cba0645bdd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d25639c7008f4e29af6226039e4b976b'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:32.743232 1696 catalog_manager.cc:6205] Error selecting replicas for tablet a656adbf1b29431687e0989f0db3459c, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2d3d2227e6a24bd2a86f457e553e698b'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:32.743252 1150 raft_consensus.cc:738] T 7304937048ca48a7bd153f3cdcfaeff0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
E20250901 01:52:32.743453 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 494b6488aa0d4355b4caa46004267465, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_3ef97ecacc3042099ec1cc5c3a7a0763'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:32.743620 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 0ee3f35392954817b350e5f48653be78, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_442e11b9211c4b209c68f653d7bfc313'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:32.743782 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae4844183a3c4a04a775e6fd878391a3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_aee74198b51d4f089538c4f2bf3ce4dc'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:32.743920 1696 catalog_manager.cc:6205] Error selecting replicas for tablet be8dfbc466564842a2fdb4953eee939b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_75ba3ba70ca24a94920e7bce96190ff2'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:32.743739 1150 consensus_queue.cc:260] T 7304937048ca48a7bd153f3cdcfaeff0 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
E20250901 01:52:32.744077 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9745582cebfe47b58c21d0191ab4cb9b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77728af9eee44f29a1e2131d5b75b7ca'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:32.744186 1696 catalog_manager.cc:6205] Error selecting replicas for tablet e2ed32b3292141278af28026528e96c2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b5077f55951d48be92a67407ffe26a08'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:32.744341 1696 catalog_manager.cc:6205] Error selecting replicas for tablet eea6bfabe6cc4f83905c24f0f7a5dfc9, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_e0902dfbdd984f85b3fdac8d283731f7'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:32.744506 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5a7fa68d29644ae8afd2711e39dde59a, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_f9e2a9182f6340bfa551fa2fc2eb2429'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:32.744669 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ff2bd017417c41b5b8225b8f00283e63, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_97526c9a6c9f4acbb6dedd78475571e0'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:32.744799 1696 catalog_manager.cc:6205] Error selecting replicas for tablet dcc60bb000f440f1aca9133928b5ce20, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2fbca8463d6546c9b8f5b018fc60251d'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:32.744936 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 65e5cdc10e9642b7b0df775cb02861fa, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_1b5729d9768546fe99eaafc4e2604f42'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:32.745059 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f8f923ddcd2740bcb7e521f4f57c4034, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_5e65c5f068b84995a2d7a76fa65dda0f'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:32.745193 1696 catalog_manager.cc:6205] Error selecting replicas for tablet e4d8330a4e454d1b8d643e17a6996741, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ce16a1b94f714c2699ee6cd916bb0215'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:32.745316 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 0813cc68d4b0488ba60a523e4479971b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d1302673d68f424ebb54b220911ce385'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:32.745496 1696 catalog_manager.cc:6205] Error selecting replicas for tablet b1ba0e19dc0b462e80c537bf2d183f5b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_66b111271c15410aab23682f8f6081ac'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:32.745643 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 8e10a01e77d14cf28c2c5e0205e595b9, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_83cf2d86a1284544ba2e7aa9ebb66bb7'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:32.745797 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 05a547a8866149cb95d23e960a737237, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_583e53c2e86544e9be68eb27ab6068a3'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:32.745975 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5f0e054f62de4cf6bc6f69ad6664b192, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ef6832d0845044ed9f165ca3541b4c3f'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:32.746208 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae9a4ad4fdfb4771ba23a91f7007447e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_5e84ae9a9c2a4084a1f9c5ffb4594de2'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:32.746425 1696 catalog_manager.cc:6205] Error selecting replicas for tablet d602c8986c75492ead9936ac04e2622d, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_82d91a1ba2eb4265896824028dec8d3a'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:32.746706 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 93aa22afc0ad4d7fa24807ea3c5d5c21, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0f4b4094d2974b7591875375eda6bed0'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:32.746938 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9074543dea9949e7a085717b40913348, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_53e30205c5fd488eb3b48c685a159d04'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:32.747171 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 3a62034e688348ed9e2ac9de009ed3a2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_409dad98d7eb455a8e16edfb4acedb34'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:32.747381 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 18498c9cb09141d08c1c5c0854b63ebd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77f4810bc83c4d948531da2a45a3792e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:32.747646 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f5e3316adeae47cd848833dd84b972bb, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7570c3012f8a4ae7bcfc7346eadac92a'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:32.747983 1696 catalog_manager.cc:6205] Error selecting replicas for tablet a6b539e160194da3a849d9e2f1ec9859, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_03fb48eaab9b42858ab7cefb722e43a8'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:32.748302 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 6ce9521b925b4f97bb4a810ce77e147e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_9e04f976a93f4d30baccf90fa3622dfe'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:32.748575 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2f61edb50d7a42198bfd70639c7bf4de, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_a5c783cbdb694af69047b31d99ff8def'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:32.749168 1151 tablet_bootstrap.cc:492] T da5f69c5b0ca4257ad2ff7918ee2af2e P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:32.749301 1151 ts_tablet_manager.cc:1397] T da5f69c5b0ca4257ad2ff7918ee2af2e P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.031s user 0.008s sys 0.003s
I20250901 01:52:32.740125 2113 rpcz_store.cc:276] 0901 01:52:26.772571 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.772615 (+ 44us) service_pool.cc:225] Handling call
0901 01:52:32.738685 (+5966070us) ts_tablet_manager.cc:1145] Deleting tablet 88ceba7f48b04d7885cc0164668700a5
0901 01:52:32.738699 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:32.739366 (+ 667us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5965680}
I20250901 01:52:32.749910 1151 raft_consensus.cc:357] T da5f69c5b0ca4257ad2ff7918ee2af2e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.750063 1151 raft_consensus.cc:383] T da5f69c5b0ca4257ad2ff7918ee2af2e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250901 01:52:32.750056 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 88ceba7f48b04d7885cc0164668700a5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 88ceba7f48b04d7885cc0164668700a5 already in progress: creating tablet
I20250901 01:52:32.750108 1151 raft_consensus.cc:738] T da5f69c5b0ca4257ad2ff7918ee2af2e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:32.749663 1150 ts_tablet_manager.cc:1428] T 7304937048ca48a7bd153f3cdcfaeff0 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250901 01:52:32.750263 1151 consensus_queue.cc:260] T da5f69c5b0ca4257ad2ff7918ee2af2e P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.750679 1151 ts_tablet_manager.cc:1428] T da5f69c5b0ca4257ad2ff7918ee2af2e P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250901 01:52:32.750913 1151 tablet_bootstrap.cc:492] T dabebc77f72e4bb19e264be2f496202a P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:32.751479 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 217) took 5961 ms. Trace:
I20250901 01:52:32.751731 1150 tablet_bootstrap.cc:492] T 98c56eb62ad84f79b04f17a43d44c479 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:32.752243 1151 tablet_bootstrap.cc:654] T dabebc77f72e4bb19e264be2f496202a P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:32.752084 2113 rpcz_store.cc:276] 0901 01:52:26.790157 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.790213 (+ 56us) service_pool.cc:225] Handling call
0901 01:52:32.750784 (+5960571us) ts_tablet_manager.cc:1145] Deleting tablet 34042ea5c1b645969c1de41fe587c343
0901 01:52:32.750799 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:32.751280 (+ 481us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5960211}
W20250901 01:52:32.753273 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 34042ea5c1b645969c1de41fe587c343 because the tablet was not found. No further retry: Not found: Tablet not found: 34042ea5c1b645969c1de41fe587c343
I20250901 01:52:32.754513 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 219) took 5951 ms. Trace:
I20250901 01:52:32.754978 2113 rpcz_store.cc:276] 0901 01:52:26.802688 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.802747 (+ 59us) service_pool.cc:225] Handling call
0901 01:52:32.753918 (+5951171us) ts_tablet_manager.cc:1145] Deleting tablet 756cfe84a517459ca015aecf5cd81571
0901 01:52:32.754104 (+ 186us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:32.754329 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5950988}
W20250901 01:52:32.755897 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 756cfe84a517459ca015aecf5cd81571 because the tablet was not found. No further retry: Not found: Tablet not found: 756cfe84a517459ca015aecf5cd81571
I20250901 01:52:32.756525 2113 tablet_replica.cc:331] T 42c1156e8dd84c6fa1396b46de13eb6f P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:32.757365 2113 raft_consensus.cc:2241] T 42c1156e8dd84c6fa1396b46de13eb6f P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:32.757965 2357 raft_consensus.cc:491] T fecf7d538b4c4c619182d5eccb44dd8b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:32.758108 2357 raft_consensus.cc:513] T fecf7d538b4c4c619182d5eccb44dd8b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.758718 2357 leader_election.cc:290] T fecf7d538b4c4c619182d5eccb44dd8b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:32.774531 1486 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b2b172757e50404aae78e818921a7d9d consensus state has no leader
I20250901 01:52:32.775081 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_631bdc9d535a4e0595e6acc32bdff111"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:32.776577 2412 raft_consensus.cc:491] T b85d64e02cc646e380581ac9f59dee9e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:32.776719 2412 raft_consensus.cc:513] T b85d64e02cc646e380581ac9f59dee9e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.777370 2412 leader_election.cc:290] T b85d64e02cc646e380581ac9f59dee9e P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:32.775732 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1008166b914340cbae5e4106d5bdaf18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:32.782181 1150 tablet_bootstrap.cc:654] T 98c56eb62ad84f79b04f17a43d44c479 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:32.782857 992 leader_election.cc:343] T 410d44e5f1974ed69e5dee65e98856a4 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 410d44e5f1974ed69e5dee65e98856a4
I20250901 01:52:32.782917 2113 raft_consensus.cc:2270] T 42c1156e8dd84c6fa1396b46de13eb6f P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:32.786671 1761 consensus_queue.cc:1035] T abf6e3be60f74872b2308dc95bf9cd37 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:32.786798 1482 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: default.Table_63ef1dcd92db46d298446117a62cfba2 [id=c1483e9d0db64d468b12344500f9cac8] alter complete (version 1)
I20250901 01:52:32.788261 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet 4902062977ea42149e3466e999117322 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:18 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:32.788491 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 48a211e174054866b030cdfb0cf947f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:18 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:32.788775 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 31614db4c53a4f3c96fd190d326ce601 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:32.789788 1035 tablet_service.cc:1515] Processing DeleteTablet for tablet 410d44e5f1974ed69e5dee65e98856a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:25 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:32.789826 2113 ts_tablet_manager.cc:1905] T 42c1156e8dd84c6fa1396b46de13eb6f P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:32.789798 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 56475ba491a24301a534304811f30140 with delete_type TABLET_DATA_DELETED (replaced by tablet 7317eb7848ee44698038a4b404d4bf21) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:32.789876 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet b81a12b38ac64efabd44aeb192d181a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:18 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:32.789922 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 1a60c10e42104502ac2d3b80b8126ceb with delete_type TABLET_DATA_DELETED (replaced by tablet 88ceba7f48b04d7885cc0164668700a5) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:32.816762 1150 tablet_bootstrap.cc:492] T 98c56eb62ad84f79b04f17a43d44c479 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:32.821417 2357 raft_consensus.cc:491] T 93bcac1fd95d44dc927abfca81692921 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:32.821556 2357 raft_consensus.cc:513] T 93bcac1fd95d44dc927abfca81692921 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:32.822254 2357 leader_election.cc:290] T 93bcac1fd95d44dc927abfca81692921 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:32.822932 2357 raft_consensus.cc:491] T 4b4aa063ffe948e69e47f03102581899 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:32.823062 2357 raft_consensus.cc:513] T 4b4aa063ffe948e69e47f03102581899 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.823642 2357 leader_election.cc:290] T 4b4aa063ffe948e69e47f03102581899 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:32.820441 1150 ts_tablet_manager.cc:1397] T 98c56eb62ad84f79b04f17a43d44c479 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.069s user 0.002s sys 0.003s
I20250901 01:52:32.838605 1485 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 61ddcf86c9f34726939cc116434f0822
W20250901 01:52:32.838994 1485 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 61ddcf86c9f34726939cc116434f0822: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:32.839838 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 61ddcf86c9f34726939cc116434f0822 with delete_type TABLET_DATA_DELETED (replaced by tablet fd2ce82a4ee24d0faa44980a3967e187) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:32.840354 1484 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 4cde5f4f02044b7bbd7d43b49aac4ae8
W20250901 01:52:32.840636 1484 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 4cde5f4f02044b7bbd7d43b49aac4ae8: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:32.841275 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 4cde5f4f02044b7bbd7d43b49aac4ae8 with delete_type TABLET_DATA_DELETED (replaced by tablet 9f62a9d97f514747b940b488907369f3) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:32.838850 1150 raft_consensus.cc:357] T 98c56eb62ad84f79b04f17a43d44c479 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.843334 2412 raft_consensus.cc:491] T 75345ea2a3dc40fc8c9f8090d8839d2d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:32.843461 2412 raft_consensus.cc:513] T 75345ea2a3dc40fc8c9f8090d8839d2d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.842820 1150 raft_consensus.cc:383] T 98c56eb62ad84f79b04f17a43d44c479 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:32.844054 2412 leader_election.cc:290] T 75345ea2a3dc40fc8c9f8090d8839d2d P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:32.844993 1055 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "315c15cf0afc44d4a79cb29294281d60" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:32.844329 1150 raft_consensus.cc:738] T 98c56eb62ad84f79b04f17a43d44c479 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:32.845678 1055 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c422881b40f54971a68530abfbd1945b" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:32.846282 1055 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fecf7d538b4c4c619182d5eccb44dd8b" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:32.846909 1055 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "93bcac1fd95d44dc927abfca81692921" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:32.848961 1055 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4b4aa063ffe948e69e47f03102581899" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:32.849916 2118 ts_tablet_manager.cc:1918] T 85322a864d154f299f2d670fdd2a7bfb P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20250901 01:52:32.850042 2118 log.cc:1199] T 85322a864d154f299f2d670fdd2a7bfb P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/85322a864d154f299f2d670fdd2a7bfb
I20250901 01:52:32.848369 1150 consensus_queue.cc:260] T 98c56eb62ad84f79b04f17a43d44c479 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.855976 1150 ts_tablet_manager.cc:1428] T 98c56eb62ad84f79b04f17a43d44c479 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.031s user 0.002s sys 0.002s
I20250901 01:52:32.860181 1150 tablet_bootstrap.cc:492] T 008d23f236c348b6be9aee718f8698af P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
W20250901 01:52:32.869009 858 leader_election.cc:336] T fecf7d538b4c4c619182d5eccb44dd8b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:32.872961 858 leader_election.cc:336] T 93bcac1fd95d44dc927abfca81692921 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:32.873102 858 leader_election.cc:336] T 4b4aa063ffe948e69e47f03102581899 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:32.873968 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_96560a52d40d4025bd092afabc06a3ee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:32.875164 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet afc002ae2486476b9208e1cf8215157c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:17 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:32.875900 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet f9b293bf6a17453b9ea9892a6970d0b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:25 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:32.875950 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet 6c05eff658924ca4a9a1e7a37f917d67 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:25 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:32.877518 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet 606fac5c6efe48d1877104b01c23ba5b with delete_type TABLET_DATA_DELETED (replaced by tablet 787be682c0e44248a80d7037fb61dc6c) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:32.877701 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 05630a72ddae4de0b890dece2feacc31 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:18 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:32.878072 1035 tablet_service.cc:1515] Processing DeleteTablet for tablet 06d5eeb70e664f73bf643922d445bc7a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:25 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:32.878245 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet a289a66b2dfa44dc8c8547ff83c1716e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:26 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:32.878412 1035 tablet_service.cc:1515] Processing DeleteTablet for tablet fa7fb88ded21432a8757f37ec3db5e71 with delete_type TABLET_DATA_DELETED (replaced by tablet 81319e373c9a4475abceeeedb507aabc) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:32.878545 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 764bfeb2a7d14c40815b5ec1b07bdd62 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:23 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:32.878666 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet 246b33338de647689d7404c70676e76f with delete_type TABLET_DATA_DELETED (replaced by tablet ae4844183a3c4a04a775e6fd878391a3) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:32.878782 1035 tablet_service.cc:1515] Processing DeleteTablet for tablet 438ae684af234c868a419fbab43b0b98 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:24 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:32.878080 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 75a703c8e20a4a2fa0ab6ec58d41ef92 with delete_type TABLET_DATA_DELETED (replaced by tablet 9745582cebfe47b58c21d0191ab4cb9b) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:32.879029 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_94b01a4d4d604c03ac01f5361800de4a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:32.879552 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_44a93d1a4542493ea8d4b1369d6f0391"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:32.880717 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 61ddcf86c9f34726939cc116434f0822 with delete_type TABLET_DATA_DELETED (replaced by tablet fd2ce82a4ee24d0faa44980a3967e187) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:32.882241 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 4cde5f4f02044b7bbd7d43b49aac4ae8 with delete_type TABLET_DATA_DELETED (replaced by tablet 9f62a9d97f514747b940b488907369f3) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:32.882536 1484 catalog_manager.cc:6652] ReplaceTablet: tablet 4cde5f4f02044b7bbd7d43b49aac4ae8 deleted and replaced by tablet 9f62a9d97f514747b940b488907369f3
I20250901 01:52:32.882922 1485 catalog_manager.cc:6652] ReplaceTablet: tablet 61ddcf86c9f34726939cc116434f0822 deleted and replaced by tablet fd2ce82a4ee24d0faa44980a3967e187
I20250901 01:52:32.883325 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8d54a303913e42bab9c56b7055e24e71"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:32.882848 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a26c7c78360a4244bb1a851a00a0f8b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:32.885146 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 617cc33d0d23439cb5f1e0c31bf11281 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:24 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:32.886010 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet ad1a6be9fcc045d3bb4e71a2f6bd5ccd with delete_type TABLET_DATA_DELETED (replaced by tablet ae9a4ad4fdfb4771ba23a91f7007447e) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:32.886211 1035 tablet_service.cc:1515] Processing DeleteTablet for tablet 36e10328c34242ebb4dbd8f294162696 with delete_type TABLET_DATA_DELETED (replaced by tablet 4abf9725be5d42f3893f07f0a9c7614c) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:32.886365 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet d4c9aa723aee4dc4b6237cb5c3722b52 with delete_type TABLET_DATA_DELETED (replaced by tablet f8f923ddcd2740bcb7e521f4f57c4034) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:32.886520 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 88ceba7f48b04d7885cc0164668700a5 with delete_type TABLET_DATA_DELETED (replaced by tablet 9074543dea9949e7a085717b40913348) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:32.886662 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 7a607fac57994001b2304e5d81c9decc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:17 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:32.886988 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet eee34044c39d4013baf78d3705489626 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:25 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:32.887164 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet da5f69c5b0ca4257ad2ff7918ee2af2e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:26 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:32.887324 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet ddbc15685fca4c708329420109ad0c57 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:26 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:32.887475 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 0dbdcb76eeba4fefa56e7835c23c0316 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:17 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:32.887615 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 3363fd67b1ad44849ecb7b4090b61a72 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:26 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:32.888504 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet f62d96c37cca4143a4b098b7c69979e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:25 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:32.888703 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 7cc1efcb6bff4aba8015d2d3b7f4c923 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:25 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:32.888839 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 500e3c33598b45a4afa983da7cb0845a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:26 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:32.888958 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 831e42de95854541a49ae6b3e039d3e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:16 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:32.889066 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 72ba689f9230411995dc96e2b45601df with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:26 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:32.889215 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet c754129977904f978f87fd89d7092e36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:24 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:32.889384 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet eaa521c52a2f4408b59770bd2a05907c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:32.889570 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 2599f98abfdc4871b573edfc2ad97d22 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:25 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:32.889744 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 8ed6616ce54548ba83ba299c4ecf3c38 with delete_type TABLET_DATA_DELETED (replaced by tablet 8a13346dcb3d4b6ab35c261829d50ae5) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:32.889961 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet c58edaa76ebf47dfb1d9d5fab6de7dfa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:17 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:32.890375 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet e249474dcbf642189560c91ec3a99fdf with delete_type TABLET_DATA_DELETED (replaced by tablet 7f46e9ddadaf4ff986d8c991178daa0c) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:32.890810 1486 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_b80d3ae8523443b389b558591480b627" } new_table_name: "default.Table_d8e85e65a1914147a2c1d1246646fcd2" modify_external_catalogs: true
I20250901 01:52:32.892294 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_849fffdec0a0447bb0f64b282e20d11e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:32.882865 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f581350ecb7542ebbfd80a332d853759"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:32.894143 1150 tablet_bootstrap.cc:654] T 008d23f236c348b6be9aee718f8698af P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:32.895349 1484 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_d6b752d82c2b4be88b762e67bac0d5a1" } modify_external_catalogs: true
I20250901 01:52:32.850411 2118 ts_tablet_manager.cc:1939] T 85322a864d154f299f2d670fdd2a7bfb P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:32.896389 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 168) took 4975 ms. Trace:
I20250901 01:52:32.896996 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 5e2472ee5d6e4d3bbf1f24027ba531ca with delete_type TABLET_DATA_DELETED (replaced by tablet 0813cc68d4b0488ba60a523e4479971b) from {username='slave'} at 127.0.0.1:35946
W20250901 01:52:32.898074 992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:40731, user_credentials={real_user=slave}} blocked reactor thread for 47276.8us
W20250901 01:52:32.898419 992 leader_election.cc:336] T 410d44e5f1974ed69e5dee65e98856a4 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:32.898532 992 leader_election.cc:304] T 410d44e5f1974ed69e5dee65e98856a4 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:32.907565 1484 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_d6b752d82c2b4be88b762e67bac0d5a1" } modify_external_catalogs: true
I20250901 01:52:32.879980 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4efa071134804a939a490bad8f139f3c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:32.913712 1489 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_e53182d58e5244a9b7ba33cb48c939ba" } modify_external_catalogs: true
I20250901 01:52:32.914664 1489 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_e53182d58e5244a9b7ba33cb48c939ba" } modify_external_catalogs: true
I20250901 01:52:32.893177 1485 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_1b5729d9768546fe99eaafc4e2604f42" } modify_external_catalogs: true
I20250901 01:52:32.915819 1485 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_1b5729d9768546fe99eaafc4e2604f42" } modify_external_catalogs: true
W20250901 01:52:32.914515 1482 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.3:51295 (request call id 185) took 2318 ms (client timeout 1000 ms). Trace:
I20250901 01:52:32.896505 2118 rpcz_store.cc:276] 0901 01:52:27.920804 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:27.920849 (+ 45us) service_pool.cc:225] Handling call
0901 01:52:32.636616 (+4715767us) ts_tablet_manager.cc:1145] Deleting tablet 85322a864d154f299f2d670fdd2a7bfb
0901 01:52:32.636625 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:32.704774 (+ 68149us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:32.849788 (+145014us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:32.896352 (+ 46564us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4715582}
I20250901 01:52:32.928117 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 85322a864d154f299f2d670fdd2a7bfb (table default.Table_5a72e7206f604795915f3a577d37f874 [id=1d8f34cc14cb4e90a562ff4424b8577a]) successfully deleted
I20250901 01:52:32.928489 2118 tablet_replica.cc:331] T cb7d7e1796c0456f8de98fb812b9e374 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:32.932585 2357 raft_consensus.cc:491] T 7304937048ca48a7bd153f3cdcfaeff0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:32.932781 2357 raft_consensus.cc:513] T 7304937048ca48a7bd153f3cdcfaeff0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
W20250901 01:52:32.898821 992 consensus_peers.cc:489] T f80f13db501c42c7950499df8fcb6fc8 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250901 01:52:32.933487 2357 leader_election.cc:290] T 7304937048ca48a7bd153f3cdcfaeff0 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:32.903437 2613 raft_consensus.cc:2747] T 410d44e5f1974ed69e5dee65e98856a4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:32.934962 859 leader_election.cc:343] T 315c15cf0afc44d4a79cb29294281d60 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:32.935077 859 leader_election.cc:304] T 315c15cf0afc44d4a79cb29294281d60 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:32.935411 859 leader_election.cc:343] T c422881b40f54971a68530abfbd1945b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:32.935497 859 leader_election.cc:304] T c422881b40f54971a68530abfbd1945b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:32.935812 859 leader_election.cc:343] T fecf7d538b4c4c619182d5eccb44dd8b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:32.935894 859 leader_election.cc:304] T fecf7d538b4c4c619182d5eccb44dd8b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:32.936189 2608 raft_consensus.cc:2747] T 315c15cf0afc44d4a79cb29294281d60 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:32.936349 2608 raft_consensus.cc:2747] T c422881b40f54971a68530abfbd1945b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:32.936484 2608 raft_consensus.cc:2747] T fecf7d538b4c4c619182d5eccb44dd8b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:32.936661 858 leader_election.cc:336] T 7304937048ca48a7bd153f3cdcfaeff0 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:32.945312 992 leader_election.cc:343] T b85d64e02cc646e380581ac9f59dee9e P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: b85d64e02cc646e380581ac9f59dee9e
W20250901 01:52:32.946775 859 leader_election.cc:343] T 93bcac1fd95d44dc927abfca81692921 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:32.946923 859 leader_election.cc:304] T 93bcac1fd95d44dc927abfca81692921 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:32.947290 859 leader_election.cc:343] T 4b4aa063ffe948e69e47f03102581899 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:32.947476 2753 raft_consensus.cc:2747] T 93bcac1fd95d44dc927abfca81692921 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:32.923727 2113 ts_tablet_manager.cc:1918] T 42c1156e8dd84c6fa1396b46de13eb6f P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 01:52:32.947479 859 leader_election.cc:304] T 4b4aa063ffe948e69e47f03102581899 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:32.947783 2113 log.cc:1199] T 42c1156e8dd84c6fa1396b46de13eb6f P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/42c1156e8dd84c6fa1396b46de13eb6f
I20250901 01:52:32.947980 2357 raft_consensus.cc:2747] T 4b4aa063ffe948e69e47f03102581899 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:32.923909 2746 log.cc:927] Time spent T aec951f2cc84499b9162b9fa24d5440e P fd27ab6665304aaea53adbd102dd4f58: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20250901 01:52:32.924539 1151 tablet_bootstrap.cc:492] T dabebc77f72e4bb19e264be2f496202a P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:32.948405 1151 ts_tablet_manager.cc:1397] T dabebc77f72e4bb19e264be2f496202a P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.198s user 0.033s sys 0.037s
I20250901 01:52:32.948760 2113 ts_tablet_manager.cc:1939] T 42c1156e8dd84c6fa1396b46de13eb6f P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:32.948987 2357 raft_consensus.cc:491] T ced4813805004bba9c4667a79ac201fc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:32.949093 2357 raft_consensus.cc:513] T ced4813805004bba9c4667a79ac201fc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.949710 2357 leader_election.cc:290] T ced4813805004bba9c4667a79ac201fc P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:32.951172 1150 tablet_bootstrap.cc:492] T 008d23f236c348b6be9aee718f8698af P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:32.951347 1150 ts_tablet_manager.cc:1397] T 008d23f236c348b6be9aee718f8698af P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.093s user 0.002s sys 0.002s
I20250901 01:52:32.952315 1150 raft_consensus.cc:357] T 008d23f236c348b6be9aee718f8698af P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.952884 1150 raft_consensus.cc:383] T 008d23f236c348b6be9aee718f8698af P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:32.953002 1150 raft_consensus.cc:738] T 008d23f236c348b6be9aee718f8698af P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:32.953287 1150 consensus_queue.cc:260] T 008d23f236c348b6be9aee718f8698af P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.954116 1150 ts_tablet_manager.cc:1428] T 008d23f236c348b6be9aee718f8698af P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
W20250901 01:52:32.926127 1482 rpcz_store.cc:269] 0901 01:52:30.595554 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.599501 (+ 3947us) service_pool.cc:225] Handling call
0901 01:52:31.127535 (+528034us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:31.127543 (+ 8us) write_op.cc:454] Released schema lock
0901 01:52:31.907920 (+780377us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:31.907996 (+ 76us) write_op.cc:454] Released schema lock
0901 01:52:32.914456 (+1006460us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0901 01:52:31.113850 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:31.113952 (+ 102us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:31.113959 (+ 7us) write_op.cc:435] Acquired schema lock
0901 01:52:31.113961 (+ 2us) tablet.cc:660] Decoding operations
0901 01:52:31.114132 (+ 171us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:31.114146 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:31.114151 (+ 5us) tablet.cc:683] Acquiring locks for 4 operations
0901 01:52:31.114304 (+ 153us) tablet.cc:699] Row locks acquired
0901 01:52:31.114307 (+ 3us) write_op.cc:260] PREPARE: finished
0901 01:52:31.114348 (+ 41us) write_op.cc:270] Start()
0901 01:52:31.114384 (+ 36us) write_op.cc:276] Timestamp: P: 1756691551114274 usec, L: 0
0901 01:52:31.114388 (+ 4us) op_driver.cc:348] REPLICATION: starting
0901 01:52:31.114582 (+ 194us) log.cc:844] Serialized 1650 byte log entry
0901 01:52:31.119319 (+ 4737us) op_driver.cc:464] REPLICATION: finished
0901 01:52:31.119419 (+ 100us) write_op.cc:301] APPLY: starting
0901 01:52:31.119434 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:31.119499 (+ 65us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:31.119533 (+ 34us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:31.119634 (+ 101us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:31.119670 (+ 36us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0901 01:52:31.119677 (+ 7us) write_op.cc:312] APPLY: finished
0901 01:52:31.119775 (+ 98us) log.cc:844] Serialized 41 byte log entry
0901 01:52:31.119844 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:31.119876 (+ 32us) write_op.cc:454] Released schema lock
0901 01:52:31.119886 (+ 10us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0901 01:52:31.821922 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:31.822100 (+ 178us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:31.822111 (+ 11us) write_op.cc:435] Acquired schema lock
0901 01:52:31.822115 (+ 4us) tablet.cc:660] Decoding operations
0901 01:52:31.822252 (+ 137us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:31.822267 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:31.822272 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0901 01:52:31.822346 (+ 74us) tablet.cc:699] Row locks acquired
0901 01:52:31.822349 (+ 3us) write_op.cc:260] PREPARE: finished
0901 01:52:31.822392 (+ 43us) write_op.cc:270] Start()
0901 01:52:31.822431 (+ 39us) write_op.cc:276] Timestamp: P: 1756691551822355 usec, L: 0
0901 01:52:31.822436 (+ 5us) op_driver.cc:348] REPLICATION: starting
0901 01:52:31.822647 (+ 211us) log.cc:844] Serialized 356 byte log entry
0901 01:52:31.828829 (+ 6182us) op_driver.cc:464] REPLICATION: finished
0901 01:52:31.828960 (+ 131us) write_op.cc:301] APPLY: starting
0901 01:52:31.828981 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:31.829057 (+ 76us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:31.829107 (+ 50us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:31.829173 (+ 66us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:31.829218 (+ 45us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0901 01:52:31.829230 (+ 12us) write_op.cc:312] APPLY: finished
0901 01:52:31.829313 (+ 83us) log.cc:844] Serialized 23 byte log entry
0901 01:52:31.829380 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:31.829400 (+ 20us) write_op.cc:454] Released schema lock
0901 01:52:31.829406 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0901 01:52:32.660632 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:32.660722 (+ 90us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:32.660726 (+ 4us) write_op.cc:435] Acquired schema lock
0901 01:52:32.660728 (+ 2us) tablet.cc:660] Decoding operations
0901 01:52:32.660822 (+ 94us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:32.660830 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:32.660835 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0901 01:52:32.660897 (+ 62us) tablet.cc:699] Row locks acquired
0901 01:52:32.660900 (+ 3us) write_op.cc:260] PREPARE: finished
0901 01:52:32.660927 (+ 27us) write_op.cc:270] Start()
0901 01:52:32.660950 (+ 23us) write_op.cc:276] Timestamp: P: 1756691552660915 usec, L: 0
0901 01:52:32.660953 (+ 3us) op_driver.cc:348] REPLICATION: starting
0901 01:52:32.661080 (+ 127us) log.cc:844] Serialized 356 byte log entry
0901 01:52:32.784919 (+123839us) op_driver.cc:464] REPLICATION: finished
0901 01:52:32.786285 (+ 1366us) write_op.cc:301] APPLY: starting
0901 01:52:32.786303 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:32.786344 (+ 41us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:32.786377 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:32.786430 (+ 53us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:32.786463 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0901 01:52:32.786470 (+ 7us) write_op.cc:312] APPLY: finished
0901 01:52:32.786543 (+ 73us) log.cc:844] Serialized 23 byte log entry
0901 01:52:32.786601 (+ 58us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:32.786619 (+ 18us) write_op.cc:454] Released schema lock
0901 01:52:32.786626 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0901 01:52:32.786700 (+ 74us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:32.786704 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":113294,"reported_tablets":184,"child_traces":[["op",{"apply.queue_time_us":55,"apply.run_cpu_time_us":532,"apply.run_wall_time_us":531,"num_ops":4,"prepare.queue_time_us":2249,"prepare.run_cpu_time_us":848,"prepare.run_wall_time_us":849,"raft.queue_time_us":414,"raft.run_cpu_time_us":434,"raft.run_wall_time_us":434,"replication_time_us":4901}],["op",{"apply.queue_time_us":70,"apply.run_cpu_time_us":505,"apply.run_wall_time_us":504,"num_ops":1,"prepare.queue_time_us":64897,"prepare.run_cpu_time_us":853,"prepare.run_wall_time_us":853,"raft.queue_time_us":1412,"raft.run_cpu_time_us":519,"raft.run_wall_time_us":519,"replication_time_us":6350}],["op",{"apply.queue_time_us":1297,"apply.run_cpu_time_us":415,"apply.run_wall_time_us":416,"num_ops":1,"prepare.queue_time_us":40,"prepare.run_cpu_time_us":513,"prepare.run_wall_time_us":512,"replication_time_us":123935}]]}
I20250901 01:52:32.955430 1482 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_f35bb824217f41ad8b0007945eaac866" } new_table_name: "default.Table_41032b3c7d7e484a9e10e0a9edf47dd6" modify_external_catalogs: true
I20250901 01:52:32.955864 1150 tablet_bootstrap.cc:492] T d27c8d99528a42ed8dd1c5a7f373bbae P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:32.957526 1150 tablet_bootstrap.cc:654] T d27c8d99528a42ed8dd1c5a7f373bbae P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:32.958894 858 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111) [suppressed 145 similar messages]
I20250901 01:52:32.965655 2357 raft_consensus.cc:491] T 3dae00aa38204ee3ac77325eb334be42 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:32.965814 2357 raft_consensus.cc:513] T 3dae00aa38204ee3ac77325eb334be42 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.966496 2357 leader_election.cc:290] T 3dae00aa38204ee3ac77325eb334be42 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:32.972733 1482 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8151bad707c04ac687c5bddfea573e3f consensus state has no leader
I20250901 01:52:32.973732 1482 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_95e1338a49a04f22b679e363e253262e" } modify_external_catalogs: true
I20250901 01:52:32.974045 1482 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_95e1338a49a04f22b679e363e253262e" } modify_external_catalogs: true
W20250901 01:52:32.979955 858 consensus_peers.cc:489] T b009cc95c819431ea9827a3e453b8da3 P 00c282f779c1477d920f24dc0dc16628 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250901 01:52:32.980072 858 leader_election.cc:336] T ced4813805004bba9c4667a79ac201fc P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:32.980926 1150 tablet_bootstrap.cc:492] T d27c8d99528a42ed8dd1c5a7f373bbae P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:32.981146 1150 ts_tablet_manager.cc:1397] T d27c8d99528a42ed8dd1c5a7f373bbae P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.000s
I20250901 01:52:32.982074 1150 raft_consensus.cc:357] T d27c8d99528a42ed8dd1c5a7f373bbae P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.982299 1150 raft_consensus.cc:383] T d27c8d99528a42ed8dd1c5a7f373bbae P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:32.982424 1150 raft_consensus.cc:738] T d27c8d99528a42ed8dd1c5a7f373bbae P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:32.982653 1150 consensus_queue.cc:260] T d27c8d99528a42ed8dd1c5a7f373bbae P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.983140 1150 ts_tablet_manager.cc:1428] T d27c8d99528a42ed8dd1c5a7f373bbae P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250901 01:52:32.983503 1150 tablet_bootstrap.cc:492] T 438ae684af234c868a419fbab43b0b98 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:32.985576 1150 tablet_bootstrap.cc:654] T 438ae684af234c868a419fbab43b0b98 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:32.986771 2118 raft_consensus.cc:2241] T cb7d7e1796c0456f8de98fb812b9e374 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:32.987005 2118 raft_consensus.cc:2270] T cb7d7e1796c0456f8de98fb812b9e374 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:32.987473 1483 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet c83ce95fd6fa41478de0e2168be201b1 (table default.Table_a525265208584713b1b3465d63130cd9 [id=bfd87b9b708449d3b16e3e0639561a49]). Expected version 1 got 0
I20250901 01:52:32.987831 2118 ts_tablet_manager.cc:1905] T cb7d7e1796c0456f8de98fb812b9e374 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:32.991258 1151 raft_consensus.cc:357] T dabebc77f72e4bb19e264be2f496202a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.991436 1151 raft_consensus.cc:383] T dabebc77f72e4bb19e264be2f496202a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:32.991490 1151 raft_consensus.cc:738] T dabebc77f72e4bb19e264be2f496202a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:32.991660 1151 consensus_queue.cc:260] T dabebc77f72e4bb19e264be2f496202a P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:32.992132 1151 ts_tablet_manager.cc:1428] T dabebc77f72e4bb19e264be2f496202a P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.044s user 0.000s sys 0.001s
I20250901 01:52:32.992381 1151 tablet_bootstrap.cc:492] T 858218c4427c41e980b2b5c969a5af40 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:32.993986 1151 tablet_bootstrap.cc:654] T 858218c4427c41e980b2b5c969a5af40 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:32.995064 858 leader_election.cc:336] T 3dae00aa38204ee3ac77325eb334be42 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:32.995159 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 221) took 6145 ms. Trace:
I20250901 01:52:32.995237 2113 rpcz_store.cc:276] 0901 01:52:26.850093 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.850137 (+ 44us) service_pool.cc:225] Handling call
0901 01:52:32.756371 (+5906234us) ts_tablet_manager.cc:1145] Deleting tablet 42c1156e8dd84c6fa1396b46de13eb6f
0901 01:52:32.756380 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:32.862025 (+105645us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:32.923606 (+ 61581us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:32.995138 (+ 71532us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":50690,"tablet-delete.queue_time_us":5906049}
I20250901 01:52:32.995405 1483 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 81ce448b148447ddbc41875d863e0518 (table default.Table_6e461d9cb0ae4dd095975bbc21533794 [id=49673fa73f8c436e9b8026efc4f3cbbc]). Expected version 1 got 0
I20250901 01:52:32.995576 2113 tablet_replica.cc:331] T 7442c6ff699141e1aff208ebdc9d192a P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:32.995721 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 42c1156e8dd84c6fa1396b46de13eb6f (table default.Table_b5d681f51fc84395bbabe1e5b4cfb358 [id=40d05d90b0354334840ad17764f5b6bb]) successfully deleted
I20250901 01:52:32.995787 2113 raft_consensus.cc:2241] T 7442c6ff699141e1aff208ebdc9d192a P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:32.996104 2113 raft_consensus.cc:2270] T 7442c6ff699141e1aff208ebdc9d192a P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:32.996582 1483 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 858218c4427c41e980b2b5c969a5af40 (table default.Table_6d47266af0ed433eb1f2abbc81d18e69 [id=264cc469a73841e79538843479a165ab]). Expected version 1 got 0
I20250901 01:52:32.996599 2113 ts_tablet_manager.cc:1905] T 7442c6ff699141e1aff208ebdc9d192a P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:32.997123 1483 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet c6f7c525f59b49938b697de98d4b5835 (table default.Table_db61e055dbd7450898e82c4c0c4763be [id=aabca67f6f754820977ac9c4c20f3e4f]). Expected version 1 got 0
I20250901 01:52:32.997459 1483 catalog_manager.cc:5582] T 5afae316ff164413be0d25d4bc0f58dc P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: UNKNOWN } } }
I20250901 01:52:32.998129 1483 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet b2b172757e50404aae78e818921a7d9d (table default.Table_2e67ad583a0b413e9190cb55137ccf3a [id=5419e36671ec470a98dff53acdbd4760]). Expected version 1 got 0
I20250901 01:52:32.998286 1483 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 2973b59bb1df41d18d9a5b6a29ae336f (table default.Table_7790af96855e45c0a316ba3a4a87c9ed [id=746845bd84c6444b8a1e5683eaba4a68]). Expected version 1 got 0
I20250901 01:52:32.999373 1483 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 5d68ea97c71942c7b55b967ba870e0d7 (table default.Table_f610cb75906d4c5cb3b57cfdb80ecefc [id=36915a3e843340ba8ef3578ac35a9af4]). Expected version 1 got 0
I20250901 01:52:32.999706 1483 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 508406601c994e42a5c5b1f665e74957 (table default.Table_7f5b278599994063862981b6b041ca8c [id=4aabc328f6374ec5b1dea26f778208b2]). Expected version 2 got 0
I20250901 01:52:33.000010 1483 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 93bcac1fd95d44dc927abfca81692921 (table default.Table_b8f8ee74648c406eaf589d3ab80934db [id=b8f69313439543be9c11eea88dfa69c5]). Expected version 1 got 0
I20250901 01:52:33.000228 1483 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 0e4009963cfe42d6ba164fc828897f09 (table default.Table_1adc70cccd2f4cb3a79df96861cb4181 [id=24fffc7ebf5a47bdb37211910301acf7]). Expected version 1 got 0
I20250901 01:52:33.000375 1483 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet dabebc77f72e4bb19e264be2f496202a (table default.Table_2755f7caf53340afbd230181643366db [id=a4d0af97df5949cca940a81a41b8e454]). Expected version 1 got 0
I20250901 01:52:33.000591 1483 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 98c56eb62ad84f79b04f17a43d44c479 (table default.Table_50ff85e8f1314f4499e76299441bc110 [id=00e7f7fe6dc44f2787b9dd8e3cbf4a51]). Expected version 1 got 0
I20250901 01:52:33.002599 1483 catalog_manager.cc:5582] T eb19e3a424324a24824f6caaf5848952 P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: UNKNOWN } } }
W20250901 01:52:33.003167 992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:40561, user_credentials={real_user=slave}} blocked reactor thread for 57526.3us
I20250901 01:52:33.003281 2113 ts_tablet_manager.cc:1918] T 7442c6ff699141e1aff208ebdc9d192a P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 01:52:33.003521 2113 log.cc:1199] T 7442c6ff699141e1aff208ebdc9d192a P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/7442c6ff699141e1aff208ebdc9d192a
I20250901 01:52:33.003841 1483 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 6c0abbfc8d904dce989ce51961e4237d (table default.Table_3f9422b08b0349b9a5466218f0fe1b06 [id=f0dfb3811e744bd4ba6d59195cebde6f]). Expected version 2 got 0
I20250901 01:52:33.003979 2113 ts_tablet_manager.cc:1939] T 7442c6ff699141e1aff208ebdc9d192a P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:33.003926 2666 consensus_queue.cc:1035] T b164b4a67a9143ceaff69b0ce49d63fe P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:33.004688 1151 tablet_bootstrap.cc:492] T 858218c4427c41e980b2b5c969a5af40 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:33.004827 1151 ts_tablet_manager.cc:1397] T 858218c4427c41e980b2b5c969a5af40 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20250901 01:52:33.005934 1483 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet ae21b9ae2e3b472eb67fb5f96388a50f (table default.Table_046289f393ff432aae0b033bdec1fb17 [id=db57b18a9ce24e068b362f5fbe940531]). Expected version 1 got 0
I20250901 01:52:33.005970 1151 raft_consensus.cc:357] T 858218c4427c41e980b2b5c969a5af40 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.006247 1151 raft_consensus.cc:383] T 858218c4427c41e980b2b5c969a5af40 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:33.006346 1151 raft_consensus.cc:738] T 858218c4427c41e980b2b5c969a5af40 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:33.006197 1483 catalog_manager.cc:5582] T b164b4a67a9143ceaff69b0ce49d63fe P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } }
I20250901 01:52:33.006588 2666 consensus_queue.cc:1035] T d1f2d98712ff4ea6aec665512de068bd P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:33.006669 1483 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet b164b4a67a9143ceaff69b0ce49d63fe (table default.Table_4505d77e4269443995012fef50c7e7b5 [id=c1b1e9e3876244f68d31c63ce7bdebf1]). Expected version 2 got 0
I20250901 01:52:33.006721 1151 consensus_queue.cc:260] T 858218c4427c41e980b2b5c969a5af40 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.007186 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 222) took 6137 ms. Trace:
I20250901 01:52:33.007084 1483 catalog_manager.cc:5582] T 6ba88af177874544b764cc32420e7ac9 P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: HEALTHY } } }
I20250901 01:52:33.007462 1151 ts_tablet_manager.cc:1428] T 858218c4427c41e980b2b5c969a5af40 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250901 01:52:33.007314 2113 rpcz_store.cc:276] 0901 01:52:26.869833 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.869888 (+ 55us) service_pool.cc:225] Handling call
0901 01:52:32.995526 (+6125638us) ts_tablet_manager.cc:1145] Deleting tablet 7442c6ff699141e1aff208ebdc9d192a
0901 01:52:32.995538 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:32.999601 (+ 4063us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:33.003186 (+ 3585us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:33.007166 (+ 3980us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":930,"tablet-delete.queue_time_us":6125426}
I20250901 01:52:33.007628 1483 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet aec951f2cc84499b9162b9fa24d5440e (table default.Table_d8e839ac6396439197b8b727af6a31b7 [id=55c31cd42fac43e6b8eddc048f565d0a]). Expected version 2 got 1
I20250901 01:52:33.007896 2113 tablet_replica.cc:331] T 061b760d8e6647c6ab0b5d538840a230 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:33.008116 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 7442c6ff699141e1aff208ebdc9d192a (table default.Table_4e2ea69d21394b9db0447f7ec8766795 [id=0bc0d862df03433dacaf6af87a8390cd]) successfully deleted
I20250901 01:52:33.007759 1151 tablet_bootstrap.cc:492] T 764bfeb2a7d14c40815b5ec1b07bdd62 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:33.008373 1483 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 5f5396045979450193d461ba1334a5c9 (table default.Table_2b946a63bd574ac0971fa3d3b9c9a6a4 [id=9c3930e9fcfe4b49ba4bea67a2b0ad07]). Expected version 2 got 0
I20250901 01:52:33.008479 2113 raft_consensus.cc:2241] T 061b760d8e6647c6ab0b5d538840a230 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:33.009811 2113 raft_consensus.cc:2270] T 061b760d8e6647c6ab0b5d538840a230 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:33.009940 1151 tablet_bootstrap.cc:654] T 764bfeb2a7d14c40815b5ec1b07bdd62 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:33.010468 2113 ts_tablet_manager.cc:1905] T 061b760d8e6647c6ab0b5d538840a230 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:33.014887 1483 catalog_manager.cc:5582] T b13afacd37474781877b9f2cd20b313a P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } }
I20250901 01:52:33.016132 1483 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 315c15cf0afc44d4a79cb29294281d60 (table default.Table_b80d3ae8523443b389b558591480b627 [id=aed8d4b4fa6a4fbf850cf5718610108f]). Expected version 2 got 0
I20250901 01:52:33.016812 1483 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet e9f1f892658b4736859ff507536920a1 (table default.Table_8825dd28afbe464193eb1b668cc9326a [id=9aa62cb634314a0a849612c54c1220d9]). Expected version 1 got 0
W20250901 01:52:33.006707 992 leader_election.cc:343] T 75345ea2a3dc40fc8c9f8090d8839d2d P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 75345ea2a3dc40fc8c9f8090d8839d2d
W20250901 01:52:33.021932 992 leader_election.cc:336] T b85d64e02cc646e380581ac9f59dee9e P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:33.022048 992 leader_election.cc:304] T b85d64e02cc646e380581ac9f59dee9e P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:33.022300 2412 raft_consensus.cc:2747] T b85d64e02cc646e380581ac9f59dee9e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:33.023389 992 consensus_peers.cc:489] T 4fd0e4f9d86c4362af96ce9964b72328 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20250901 01:52:33.025315 1055 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7304937048ca48a7bd153f3cdcfaeff0" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:33.027005 2118 ts_tablet_manager.cc:1918] T cb7d7e1796c0456f8de98fb812b9e374 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:33.031553 2357 raft_consensus.cc:491] T 858218c4427c41e980b2b5c969a5af40 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:33.031740 2357 raft_consensus.cc:513] T 858218c4427c41e980b2b5c969a5af40 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.032473 2357 leader_election.cc:290] T 858218c4427c41e980b2b5c969a5af40 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:33.033361 2118 log.cc:1199] T cb7d7e1796c0456f8de98fb812b9e374 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/cb7d7e1796c0456f8de98fb812b9e374
I20250901 01:52:33.034087 2118 ts_tablet_manager.cc:1939] T cb7d7e1796c0456f8de98fb812b9e374 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:33.034198 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ced4813805004bba9c4667a79ac201fc" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:33.034375 858 leader_election.cc:336] T 858218c4427c41e980b2b5c969a5af40 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:33.034798 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3dae00aa38204ee3ac77325eb334be42" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:33.035210 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 169) took 5110 ms. Trace:
I20250901 01:52:33.035348 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "858218c4427c41e980b2b5c969a5af40" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:33.035555 1052 raft_consensus.cc:2466] T 858218c4427c41e980b2b5c969a5af40 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00c282f779c1477d920f24dc0dc16628 in term 0.
W20250901 01:52:33.036480 859 leader_election.cc:343] T 7304937048ca48a7bd153f3cdcfaeff0 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:33.037273 859 leader_election.cc:304] T 7304937048ca48a7bd153f3cdcfaeff0 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:33.037740 2357 raft_consensus.cc:2747] T 7304937048ca48a7bd153f3cdcfaeff0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:33.038097 1151 tablet_bootstrap.cc:492] T 764bfeb2a7d14c40815b5ec1b07bdd62 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:33.038245 1151 ts_tablet_manager.cc:1397] T 764bfeb2a7d14c40815b5ec1b07bdd62 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.000s
I20250901 01:52:33.035352 2118 rpcz_store.cc:276] 0901 01:52:27.924889 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:27.924946 (+ 57us) service_pool.cc:225] Handling call
0901 01:52:32.928432 (+5003486us) ts_tablet_manager.cc:1145] Deleting tablet cb7d7e1796c0456f8de98fb812b9e374
0901 01:52:32.928444 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:33.010673 (+ 82229us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:33.026881 (+ 16208us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:33.035187 (+ 8306us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":58033,"tablet-delete.queue_time_us":5003279}
I20250901 01:52:33.038796 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet cb7d7e1796c0456f8de98fb812b9e374 (table default.Table_c4148158ed7b48a3946559ba978e72fd [id=327c5a63904540508142ab600c308757]) successfully deleted
I20250901 01:52:33.038969 1151 raft_consensus.cc:357] T 764bfeb2a7d14c40815b5ec1b07bdd62 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.039150 1151 raft_consensus.cc:383] T 764bfeb2a7d14c40815b5ec1b07bdd62 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:33.039202 1151 raft_consensus.cc:738] T 764bfeb2a7d14c40815b5ec1b07bdd62 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:33.039410 1151 consensus_queue.cc:260] T 764bfeb2a7d14c40815b5ec1b07bdd62 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.039649 2118 tablet_replica.cc:331] T 3697821cd29446bb908b020599df0db9 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:33.039929 1151 ts_tablet_manager.cc:1428] T 764bfeb2a7d14c40815b5ec1b07bdd62 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250901 01:52:33.039948 2118 raft_consensus.cc:2241] T 3697821cd29446bb908b020599df0db9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:33.040284 1151 tablet_bootstrap.cc:492] T 617cc33d0d23439cb5f1e0c31bf11281 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:33.040513 2118 raft_consensus.cc:2270] T 3697821cd29446bb908b020599df0db9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:33.041966 1151 tablet_bootstrap.cc:654] T 617cc33d0d23439cb5f1e0c31bf11281 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:33.042660 2118 ts_tablet_manager.cc:1905] T 3697821cd29446bb908b020599df0db9 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:33.042354 992 consensus_peers.cc:489] T b164b4a67a9143ceaff69b0ce49d63fe P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250901 01:52:33.051177 992 consensus_peers.cc:489] T abf6e3be60f74872b2308dc95bf9cd37 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250901 01:52:33.051342 992 leader_election.cc:336] T 75345ea2a3dc40fc8c9f8090d8839d2d P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:33.051493 992 leader_election.cc:304] T 75345ea2a3dc40fc8c9f8090d8839d2d P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:33.051749 1150 tablet_bootstrap.cc:492] T 438ae684af234c868a419fbab43b0b98 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:33.051885 1150 ts_tablet_manager.cc:1397] T 438ae684af234c868a419fbab43b0b98 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.068s user 0.007s sys 0.002s
I20250901 01:52:33.052019 1761 raft_consensus.cc:2747] T 75345ea2a3dc40fc8c9f8090d8839d2d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:33.053817 2666 raft_consensus.cc:491] T 3eba54e5ddd14c518f4cbd539268ae37 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:33.053934 2666 raft_consensus.cc:513] T 3eba54e5ddd14c518f4cbd539268ae37 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.054352 2666 leader_election.cc:290] T 3eba54e5ddd14c518f4cbd539268ae37 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:33.056108 859 leader_election.cc:343] T ced4813805004bba9c4667a79ac201fc P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:33.056232 1150 raft_consensus.cc:357] T 438ae684af234c868a419fbab43b0b98 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.056233 859 leader_election.cc:304] T ced4813805004bba9c4667a79ac201fc P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:33.057459 859 leader_election.cc:343] T 3dae00aa38204ee3ac77325eb334be42 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:33.057575 859 leader_election.cc:304] T 3dae00aa38204ee3ac77325eb334be42 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:33.057794 2753 raft_consensus.cc:2747] T ced4813805004bba9c4667a79ac201fc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:33.056380 1150 raft_consensus.cc:383] T 438ae684af234c868a419fbab43b0b98 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:33.057955 2753 raft_consensus.cc:2747] T 3dae00aa38204ee3ac77325eb334be42 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:33.057919 859 leader_election.cc:304] T 858218c4427c41e980b2b5c969a5af40 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:33.057919 1150 raft_consensus.cc:738] T 438ae684af234c868a419fbab43b0b98 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:33.058269 1150 consensus_queue.cc:260] T 438ae684af234c868a419fbab43b0b98 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.058516 2608 raft_consensus.cc:2802] T 858218c4427c41e980b2b5c969a5af40 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:33.058579 2608 raft_consensus.cc:491] T 858218c4427c41e980b2b5c969a5af40 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:33.058640 2608 raft_consensus.cc:3058] T 858218c4427c41e980b2b5c969a5af40 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:33.061120 2608 raft_consensus.cc:513] T 858218c4427c41e980b2b5c969a5af40 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.061733 2608 leader_election.cc:290] T 858218c4427c41e980b2b5c969a5af40 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:33.062458 1150 ts_tablet_manager.cc:1428] T 438ae684af234c868a419fbab43b0b98 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20250901 01:52:33.062819 1150 tablet_bootstrap.cc:492] T d0463a8d655d4db4800b4e1b221c8315 P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
W20250901 01:52:33.063416 858 leader_election.cc:336] T 858218c4427c41e980b2b5c969a5af40 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:33.063699 2666 raft_consensus.cc:491] T 764bfeb2a7d14c40815b5ec1b07bdd62 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:33.063829 2666 raft_consensus.cc:513] T 764bfeb2a7d14c40815b5ec1b07bdd62 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.064185 2118 ts_tablet_manager.cc:1918] T 3697821cd29446bb908b020599df0db9 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:33.064270 2118 log.cc:1199] T 3697821cd29446bb908b020599df0db9 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/3697821cd29446bb908b020599df0db9
I20250901 01:52:33.064321 2666 leader_election.cc:290] T 764bfeb2a7d14c40815b5ec1b07bdd62 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:33.064661 2666 consensus_queue.cc:1035] T b13afacd37474781877b9f2cd20b313a P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250901 01:52:33.064898 992 consensus_peers.cc:489] T d10c193452954c198bba686169b8025b P fd27ab6665304aaea53adbd102dd4f58 -> Peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Couldn't send request to peer 00c282f779c1477d920f24dc0dc16628. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d10c193452954c198bba686169b8025b. This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:33.064946 2412 consensus_queue.cc:1035] T 5afae316ff164413be0d25d4bc0f58dc P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:33.049970 2113 ts_tablet_manager.cc:1918] T 061b760d8e6647c6ab0b5d538840a230 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 01:52:33.065148 2113 log.cc:1199] T 061b760d8e6647c6ab0b5d538840a230 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/061b760d8e6647c6ab0b5d538840a230
I20250901 01:52:33.065441 2118 ts_tablet_manager.cc:1939] T 3697821cd29446bb908b020599df0db9 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:33.065577 2113 ts_tablet_manager.cc:1939] T 061b760d8e6647c6ab0b5d538840a230 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:33.066314 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 170) took 5123 ms. Trace:
I20250901 01:52:33.066491 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 223) took 6097 ms. Trace:
I20250901 01:52:33.066414 2118 rpcz_store.cc:276] 0901 01:52:27.942704 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:27.942769 (+ 65us) service_pool.cc:225] Handling call
0901 01:52:33.039510 (+5096741us) ts_tablet_manager.cc:1145] Deleting tablet 3697821cd29446bb908b020599df0db9
0901 01:52:33.039597 (+ 87us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:33.052671 (+ 13074us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:33.064089 (+ 11418us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:33.066296 (+ 2207us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5096564}
I20250901 01:52:33.066617 2113 rpcz_store.cc:276] 0901 01:52:26.968789 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:26.968861 (+ 72us) service_pool.cc:225] Handling call
0901 01:52:33.007847 (+6038986us) ts_tablet_manager.cc:1145] Deleting tablet 061b760d8e6647c6ab0b5d538840a230
0901 01:52:33.007857 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:33.026446 (+ 18589us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:33.049832 (+ 23386us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:33.066467 (+ 16635us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6038546}
I20250901 01:52:33.066900 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 3697821cd29446bb908b020599df0db9 (table default.Table_efc44c841f0b4ae2a168be904751b65c [id=8106fbf3fbe244dabcc6f134e512a9d4]) successfully deleted
I20250901 01:52:33.066987 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 225) took 6026 ms. Trace:
I20250901 01:52:33.067062 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 061b760d8e6647c6ab0b5d538840a230 (table default.Table_fbba493de02249499d26467bf15c3d5f [id=748daa63137c48a5b151bd34dc5b13fe]) successfully deleted
I20250901 01:52:33.067059 2113 rpcz_store.cc:276] 0901 01:52:27.040960 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:27.041021 (+ 61us) service_pool.cc:225] Handling call
0901 01:52:33.066869 (+6025848us) ts_tablet_manager.cc:1145] Deleting tablet 008d23f236c348b6be9aee718f8698af
0901 01:52:33.066879 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:33.066971 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6025667}
I20250901 01:52:33.067252 2118 tablet_replica.cc:331] T 9fbd44641d9e43c3b592e388e4adf350 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:33.067363 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 226) took 6023 ms. Trace:
I20250901 01:52:33.067425 2113 rpcz_store.cc:276] 0901 01:52:27.044179 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:27.044238 (+ 59us) service_pool.cc:225] Handling call
0901 01:52:33.067250 (+6023012us) ts_tablet_manager.cc:1145] Deleting tablet 3dae00aa38204ee3ac77325eb334be42
0901 01:52:33.067276 (+ 26us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:33.067347 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6022818}
I20250901 01:52:33.067625 2118 raft_consensus.cc:2241] T 9fbd44641d9e43c3b592e388e4adf350 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
W20250901 01:52:33.067631 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 008d23f236c348b6be9aee718f8698af because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 008d23f236c348b6be9aee718f8698af already in progress: creating tablet
I20250901 01:52:33.067729 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 227) took 5990 ms. Trace:
I20250901 01:52:33.067759 2118 raft_consensus.cc:2270] T 9fbd44641d9e43c3b592e388e4adf350 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:33.067792 2113 rpcz_store.cc:276] 0901 01:52:27.077040 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:27.078890 (+ 1850us) service_pool.cc:225] Handling call
0901 01:52:33.067641 (+5988751us) ts_tablet_manager.cc:1145] Deleting tablet 435b1cca5a7d4d49959a310cb250b2a0
0901 01:52:33.067651 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:33.067716 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5988536}
I20250901 01:52:33.068056 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 229) took 5961 ms. Trace:
I20250901 01:52:33.068126 2113 rpcz_store.cc:276] 0901 01:52:27.106528 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:27.106591 (+ 63us) service_pool.cc:225] Handling call
0901 01:52:33.067967 (+5961376us) ts_tablet_manager.cc:1145] Deleting tablet f23df6f253a943fa8e146bb8c811ef51
0901 01:52:33.067974 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:33.068044 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5961170}
I20250901 01:52:33.068346 2113 tablet_replica.cc:331] T 641a83a3e50d451189c33c7b7b4ca070 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
W20250901 01:52:33.065039 992 consensus_peers.cc:489] T ff013ea889d44cba8797834e4611b489 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Couldn't send request to peer 00c282f779c1477d920f24dc0dc16628. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ff013ea889d44cba8797834e4611b489. This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:33.068684 1483 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet d0c3c363dda444519a17dca63ad9caa7 (table default.Table_42422c44ca254d8ca989134c2433391f [id=8c1b263824ec4c15a5ec424364c78895]). Expected version 3 got 0
I20250901 01:52:33.070039 2412 raft_consensus.cc:491] T e2aa32450a584395ae07b49f76a7b26d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:33.070156 2412 raft_consensus.cc:513] T e2aa32450a584395ae07b49f76a7b26d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.070595 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "858218c4427c41e980b2b5c969a5af40" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58"
I20250901 01:52:33.070619 2412 leader_election.cc:290] T e2aa32450a584395ae07b49f76a7b26d P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:33.070760 1052 raft_consensus.cc:3058] T 858218c4427c41e980b2b5c969a5af40 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:33.070793 2412 raft_consensus.cc:491] T 268439ff8d3a4f7a89f3d6395f2079c5 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:33.070904 2412 raft_consensus.cc:513] T 268439ff8d3a4f7a89f3d6395f2079c5 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.071385 2412 leader_election.cc:290] T 268439ff8d3a4f7a89f3d6395f2079c5 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:33.072244 1151 tablet_bootstrap.cc:492] T 617cc33d0d23439cb5f1e0c31bf11281 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:33.072458 1151 ts_tablet_manager.cc:1397] T 617cc33d0d23439cb5f1e0c31bf11281 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.000s
I20250901 01:52:33.073185 1151 raft_consensus.cc:357] T 617cc33d0d23439cb5f1e0c31bf11281 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.073374 1151 raft_consensus.cc:383] T 617cc33d0d23439cb5f1e0c31bf11281 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:33.073458 1151 raft_consensus.cc:738] T 617cc33d0d23439cb5f1e0c31bf11281 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:33.073666 1052 raft_consensus.cc:2466] T 858218c4427c41e980b2b5c969a5af40 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c282f779c1477d920f24dc0dc16628 in term 1.
I20250901 01:52:33.073689 1151 consensus_queue.cc:260] T 617cc33d0d23439cb5f1e0c31bf11281 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:33.074108 858 consensus_peers.cc:489] T f6a8b425968e49deaecfa3adacfd3f1f P 00c282f779c1477d920f24dc0dc16628 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250901 01:52:33.074891 1151 ts_tablet_manager.cc:1428] T 617cc33d0d23439cb5f1e0c31bf11281 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250901 01:52:33.075155 1151 tablet_bootstrap.cc:492] T c6f7c525f59b49938b697de98d4b5835 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:33.076370 922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "268439ff8d3a4f7a89f3d6395f2079c5" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:33.076606 922 raft_consensus.cc:2466] T 268439ff8d3a4f7a89f3d6395f2079c5 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
W20250901 01:52:33.076785 992 leader_election.cc:336] T 764bfeb2a7d14c40815b5ec1b07bdd62 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:33.076898 992 leader_election.cc:336] T 3eba54e5ddd14c518f4cbd539268ae37 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:33.076992 992 consensus_peers.cc:489] T d1f2d98712ff4ea6aec665512de068bd P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:33.077036 859 leader_election.cc:304] T 858218c4427c41e980b2b5c969a5af40 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:33.077315 1151 tablet_bootstrap.cc:654] T c6f7c525f59b49938b697de98d4b5835 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:33.077736 2608 raft_consensus.cc:2802] T 858218c4427c41e980b2b5c969a5af40 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:33.077931 2608 raft_consensus.cc:695] T 858218c4427c41e980b2b5c969a5af40 P 00c282f779c1477d920f24dc0dc16628 [term 1 LEADER]: Becoming Leader. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Running, Role: LEADER
I20250901 01:52:33.078227 2608 consensus_queue.cc:237] T 858218c4427c41e980b2b5c969a5af40 P 00c282f779c1477d920f24dc0dc16628 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.081400 2113 raft_consensus.cc:2241] T 641a83a3e50d451189c33c7b7b4ca070 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:33.081560 2113 raft_consensus.cc:2270] T 641a83a3e50d451189c33c7b7b4ca070 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:33.082064 2113 ts_tablet_manager.cc:1905] T 641a83a3e50d451189c33c7b7b4ca070 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:33.084152 2118 ts_tablet_manager.cc:1905] T 9fbd44641d9e43c3b592e388e4adf350 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:33.086165 992 leader_election.cc:343] T e2aa32450a584395ae07b49f76a7b26d P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: e2aa32450a584395ae07b49f76a7b26d
I20250901 01:52:33.086369 992 leader_election.cc:304] T 268439ff8d3a4f7a89f3d6395f2079c5 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:33.086575 1761 raft_consensus.cc:2802] T 268439ff8d3a4f7a89f3d6395f2079c5 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:33.086652 1761 raft_consensus.cc:491] T 268439ff8d3a4f7a89f3d6395f2079c5 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:33.086715 1761 raft_consensus.cc:3058] T 268439ff8d3a4f7a89f3d6395f2079c5 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:33.089187 1761 raft_consensus.cc:513] T 268439ff8d3a4f7a89f3d6395f2079c5 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.089744 1761 leader_election.cc:290] T 268439ff8d3a4f7a89f3d6395f2079c5 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:33.090231 2357 raft_consensus.cc:491] T 4fca1435d82447f289fda4035aa2ed89 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:33.090338 2357 raft_consensus.cc:513] T 4fca1435d82447f289fda4035aa2ed89 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.090909 2357 leader_election.cc:290] T 4fca1435d82447f289fda4035aa2ed89 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:33.092496 992 leader_election.cc:336] T e2aa32450a584395ae07b49f76a7b26d P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:33.092633 992 leader_election.cc:304] T e2aa32450a584395ae07b49f76a7b26d P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
W20250901 01:52:33.092967 992 leader_election.cc:336] T 268439ff8d3a4f7a89f3d6395f2079c5 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:33.093039 858 leader_election.cc:336] T 4fca1435d82447f289fda4035aa2ed89 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:33.093863 922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "268439ff8d3a4f7a89f3d6395f2079c5" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:33.094048 922 raft_consensus.cc:3058] T 268439ff8d3a4f7a89f3d6395f2079c5 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:33.094089 919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3eba54e5ddd14c518f4cbd539268ae37" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:33.094237 919 raft_consensus.cc:2466] T 3eba54e5ddd14c518f4cbd539268ae37 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:33.094805 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "764bfeb2a7d14c40815b5ec1b07bdd62" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:33.094944 916 raft_consensus.cc:2466] T 764bfeb2a7d14c40815b5ec1b07bdd62 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:33.096477 1761 raft_consensus.cc:2747] T e2aa32450a584395ae07b49f76a7b26d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:33.096824 922 raft_consensus.cc:2466] T 268439ff8d3a4f7a89f3d6395f2079c5 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
W20250901 01:52:33.068735 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 435b1cca5a7d4d49959a310cb250b2a0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 435b1cca5a7d4d49959a310cb250b2a0 already in progress: creating tablet
I20250901 01:52:33.099936 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4fca1435d82447f289fda4035aa2ed89" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:33.100207 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet f23df6f253a943fa8e146bb8c811ef51 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f23df6f253a943fa8e146bb8c811ef51 already in progress: creating tablet
I20250901 01:52:33.101487 992 leader_election.cc:304] T 3eba54e5ddd14c518f4cbd539268ae37 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:33.101908 992 leader_election.cc:304] T 764bfeb2a7d14c40815b5ec1b07bdd62 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:33.102150 2412 raft_consensus.cc:2802] T 3eba54e5ddd14c518f4cbd539268ae37 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:33.102217 2412 raft_consensus.cc:491] T 3eba54e5ddd14c518f4cbd539268ae37 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:33.102278 2412 raft_consensus.cc:3058] T 3eba54e5ddd14c518f4cbd539268ae37 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:33.102255 992 leader_election.cc:304] T 268439ff8d3a4f7a89f3d6395f2079c5 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
W20250901 01:52:33.103235 859 leader_election.cc:343] T 4fca1435d82447f289fda4035aa2ed89 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:33.103335 859 leader_election.cc:304] T 4fca1435d82447f289fda4035aa2ed89 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:33.103659 2357 raft_consensus.cc:2747] T 4fca1435d82447f289fda4035aa2ed89 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:33.104007 1761 raft_consensus.cc:2802] T 764bfeb2a7d14c40815b5ec1b07bdd62 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:33.104075 1761 raft_consensus.cc:491] T 764bfeb2a7d14c40815b5ec1b07bdd62 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:33.104135 1761 raft_consensus.cc:3058] T 764bfeb2a7d14c40815b5ec1b07bdd62 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:33.104737 2613 raft_consensus.cc:2802] T 268439ff8d3a4f7a89f3d6395f2079c5 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:33.105335 2113 ts_tablet_manager.cc:1918] T 641a83a3e50d451189c33c7b7b4ca070 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:33.105454 2113 log.cc:1199] T 641a83a3e50d451189c33c7b7b4ca070 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/641a83a3e50d451189c33c7b7b4ca070
I20250901 01:52:33.105850 2113 ts_tablet_manager.cc:1939] T 641a83a3e50d451189c33c7b7b4ca070 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:33.100317 1483 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 33f637f6a8134215bb1995ec38440073 (table default.Table_3d5afc22c4e14d6990aea31564dfde20 [id=557afe29976d40f6bfb6ae4ca348f887]). Expected version 2 got 0
I20250901 01:52:33.106724 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 230) took 5999 ms. Trace:
I20250901 01:52:33.106772 1761 raft_consensus.cc:513] T 764bfeb2a7d14c40815b5ec1b07bdd62 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.106792 1483 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 45f6e8ba99b74898a2922b36768c6c36 (table default.Table_d1df9c0ae1a04b21a133e93cf4ae363a [id=9ce26dcf307149b595a43b6ac4562fba]). Expected version 2 got 0
I20250901 01:52:33.106825 2113 rpcz_store.cc:276] 0901 01:52:27.107093 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:27.107133 (+ 40us) service_pool.cc:225] Handling call
0901 01:52:33.068310 (+5961177us) ts_tablet_manager.cc:1145] Deleting tablet 641a83a3e50d451189c33c7b7b4ca070
0901 01:52:33.068317 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:33.085007 (+ 16690us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:33.105222 (+ 20215us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:33.106700 (+ 1478us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5961027}
I20250901 01:52:33.107210 2113 tablet_replica.cc:331] T d10c193452954c198bba686169b8025b P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:33.107336 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 641a83a3e50d451189c33c7b7b4ca070 (table default.Table_ef6832d0845044ed9f165ca3541b4c3f [id=0e69044796ea4292b1dc100db2f26cee]) successfully deleted
I20250901 01:52:33.107445 2113 raft_consensus.cc:2241] T d10c193452954c198bba686169b8025b P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:33.107834 2113 raft_consensus.cc:2270] T d10c193452954c198bba686169b8025b P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:33.107980 1761 leader_election.cc:290] T 764bfeb2a7d14c40815b5ec1b07bdd62 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:33.108777 992 leader_election.cc:336] T 268439ff8d3a4f7a89f3d6395f2079c5 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:33.109920 1150 tablet_bootstrap.cc:654] T d0463a8d655d4db4800b4e1b221c8315 P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:33.110951 2412 raft_consensus.cc:513] T 3eba54e5ddd14c518f4cbd539268ae37 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.111443 2412 leader_election.cc:290] T 3eba54e5ddd14c518f4cbd539268ae37 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:33.116998 2357 raft_consensus.cc:491] T 5dd0c6b806574caf9126dad94833fcf7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:33.117157 2357 raft_consensus.cc:513] T 5dd0c6b806574caf9126dad94833fcf7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.117605 922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3eba54e5ddd14c518f4cbd539268ae37" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:33.117799 2357 leader_election.cc:290] T 5dd0c6b806574caf9126dad94833fcf7 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:33.117820 922 raft_consensus.cc:3058] T 3eba54e5ddd14c518f4cbd539268ae37 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:33.118950 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "764bfeb2a7d14c40815b5ec1b07bdd62" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:33.119079 914 raft_consensus.cc:3058] T 764bfeb2a7d14c40815b5ec1b07bdd62 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
W20250901 01:52:33.120227 858 leader_election.cc:336] T 5dd0c6b806574caf9126dad94833fcf7 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:33.121177 922 raft_consensus.cc:2466] T 3eba54e5ddd14c518f4cbd539268ae37 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
W20250901 01:52:33.123342 992 leader_election.cc:336] T 764bfeb2a7d14c40815b5ec1b07bdd62 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:33.124166 992 leader_election.cc:304] T 3eba54e5ddd14c518f4cbd539268ae37 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:33.124845 2412 raft_consensus.cc:2802] T 3eba54e5ddd14c518f4cbd539268ae37 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:33.125034 2412 raft_consensus.cc:695] T 3eba54e5ddd14c518f4cbd539268ae37 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:33.125308 2412 consensus_queue.cc:237] T 3eba54e5ddd14c518f4cbd539268ae37 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.129058 914 raft_consensus.cc:2466] T 764bfeb2a7d14c40815b5ec1b07bdd62 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:33.129772 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5dd0c6b806574caf9126dad94833fcf7" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:33.130055 1150 tablet_bootstrap.cc:492] T d0463a8d655d4db4800b4e1b221c8315 P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:33.130204 1150 ts_tablet_manager.cc:1397] T d0463a8d655d4db4800b4e1b221c8315 P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.067s user 0.005s sys 0.000s
W20250901 01:52:33.131186 859 leader_election.cc:343] T 5dd0c6b806574caf9126dad94833fcf7 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:33.131331 859 leader_election.cc:304] T 5dd0c6b806574caf9126dad94833fcf7 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:33.131225 992 leader_election.cc:304] T 764bfeb2a7d14c40815b5ec1b07bdd62 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:33.131923 2113 ts_tablet_manager.cc:1905] T d10c193452954c198bba686169b8025b P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
E20250901 01:52:33.132697 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 0c88c339e59a4c1da21756cba0645bdd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d25639c7008f4e29af6226039e4b976b'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.132872 1696 catalog_manager.cc:6205] Error selecting replicas for tablet a656adbf1b29431687e0989f0db3459c, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2d3d2227e6a24bd2a86f457e553e698b'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.133015 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 494b6488aa0d4355b4caa46004267465, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_3ef97ecacc3042099ec1cc5c3a7a0763'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.133126 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 0ee3f35392954817b350e5f48653be78, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_442e11b9211c4b209c68f653d7bfc313'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.133252 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae4844183a3c4a04a775e6fd878391a3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_aee74198b51d4f089538c4f2bf3ce4dc'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.133360 1696 catalog_manager.cc:6205] Error selecting replicas for tablet be8dfbc466564842a2fdb4953eee939b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_75ba3ba70ca24a94920e7bce96190ff2'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.133493 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9745582cebfe47b58c21d0191ab4cb9b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77728af9eee44f29a1e2131d5b75b7ca'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.133601 1696 catalog_manager.cc:6205] Error selecting replicas for tablet e2ed32b3292141278af28026528e96c2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b5077f55951d48be92a67407ffe26a08'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.133729 1696 catalog_manager.cc:6205] Error selecting replicas for tablet eea6bfabe6cc4f83905c24f0f7a5dfc9, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_e0902dfbdd984f85b3fdac8d283731f7'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.133854 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5a7fa68d29644ae8afd2711e39dde59a, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_f9e2a9182f6340bfa551fa2fc2eb2429'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.133981 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ff2bd017417c41b5b8225b8f00283e63, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_97526c9a6c9f4acbb6dedd78475571e0'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.134089 1696 catalog_manager.cc:6205] Error selecting replicas for tablet dcc60bb000f440f1aca9133928b5ce20, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2fbca8463d6546c9b8f5b018fc60251d'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.134227 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f8f923ddcd2740bcb7e521f4f57c4034, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_5e65c5f068b84995a2d7a76fa65dda0f'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.134302 1696 catalog_manager.cc:6205] Error selecting replicas for tablet e4d8330a4e454d1b8d643e17a6996741, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ce16a1b94f714c2699ee6cd916bb0215'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.134423 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 0813cc68d4b0488ba60a523e4479971b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d1302673d68f424ebb54b220911ce385'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.134536 1696 catalog_manager.cc:6205] Error selecting replicas for tablet b1ba0e19dc0b462e80c537bf2d183f5b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_66b111271c15410aab23682f8f6081ac'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.134672 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 8e10a01e77d14cf28c2c5e0205e595b9, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_83cf2d86a1284544ba2e7aa9ebb66bb7'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.134780 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 05a547a8866149cb95d23e960a737237, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_583e53c2e86544e9be68eb27ab6068a3'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.134943 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5f0e054f62de4cf6bc6f69ad6664b192, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ef6832d0845044ed9f165ca3541b4c3f'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.135058 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae9a4ad4fdfb4771ba23a91f7007447e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_5e84ae9a9c2a4084a1f9c5ffb4594de2'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.135192 1696 catalog_manager.cc:6205] Error selecting replicas for tablet d602c8986c75492ead9936ac04e2622d, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_82d91a1ba2eb4265896824028dec8d3a'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.135314 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 93aa22afc0ad4d7fa24807ea3c5d5c21, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0f4b4094d2974b7591875375eda6bed0'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.135425 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9074543dea9949e7a085717b40913348, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_53e30205c5fd488eb3b48c685a159d04'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.135499 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 3a62034e688348ed9e2ac9de009ed3a2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_409dad98d7eb455a8e16edfb4acedb34'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.135636 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 18498c9cb09141d08c1c5c0854b63ebd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77f4810bc83c4d948531da2a45a3792e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.135749 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f5e3316adeae47cd848833dd84b972bb, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7570c3012f8a4ae7bcfc7346eadac92a'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.135886 1696 catalog_manager.cc:6205] Error selecting replicas for tablet a6b539e160194da3a849d9e2f1ec9859, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_03fb48eaab9b42858ab7cefb722e43a8'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.136008 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 6ce9521b925b4f97bb4a810ce77e147e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_9e04f976a93f4d30baccf90fa3622dfe'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.136137 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2f61edb50d7a42198bfd70639c7bf4de, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_a5c783cbdb694af69047b31d99ff8def'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:33.136646 2608 raft_consensus.cc:2747] T 5dd0c6b806574caf9126dad94833fcf7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:33.145284 992 leader_election.cc:336] T 3eba54e5ddd14c518f4cbd539268ae37 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:33.146919 1151 tablet_bootstrap.cc:492] T c6f7c525f59b49938b697de98d4b5835 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:33.147157 1151 ts_tablet_manager.cc:1397] T c6f7c525f59b49938b697de98d4b5835 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.072s user 0.006s sys 0.003s
I20250901 01:52:33.148126 1151 raft_consensus.cc:357] T c6f7c525f59b49938b697de98d4b5835 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.148470 1151 raft_consensus.cc:383] T c6f7c525f59b49938b697de98d4b5835 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:33.148583 1151 raft_consensus.cc:738] T c6f7c525f59b49938b697de98d4b5835 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:33.148896 1151 consensus_queue.cc:260] T c6f7c525f59b49938b697de98d4b5835 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.150120 2780 raft_consensus.cc:2802] T 764bfeb2a7d14c40815b5ec1b07bdd62 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:33.150317 2780 raft_consensus.cc:695] T 764bfeb2a7d14c40815b5ec1b07bdd62 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:33.150662 2780 consensus_queue.cc:237] T 764bfeb2a7d14c40815b5ec1b07bdd62 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.153769 1150 raft_consensus.cc:357] T d0463a8d655d4db4800b4e1b221c8315 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.153988 1150 raft_consensus.cc:383] T d0463a8d655d4db4800b4e1b221c8315 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:33.154073 1150 raft_consensus.cc:738] T d0463a8d655d4db4800b4e1b221c8315 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:33.154307 1150 consensus_queue.cc:260] T d0463a8d655d4db4800b4e1b221c8315 P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.156366 2118 ts_tablet_manager.cc:1918] T 9fbd44641d9e43c3b592e388e4adf350 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:33.156492 2118 log.cc:1199] T 9fbd44641d9e43c3b592e388e4adf350 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/9fbd44641d9e43c3b592e388e4adf350
I20250901 01:52:33.156940 2118 ts_tablet_manager.cc:1939] T 9fbd44641d9e43c3b592e388e4adf350 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:33.155058 2778 tablet.cc:1722] T aec951f2cc84499b9162b9fa24d5440e P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250901 01:52:33.158501 2779 tablet.cc:1722] T aec951f2cc84499b9162b9fa24d5440e P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250901 01:52:33.160992 2613 raft_consensus.cc:695] T 268439ff8d3a4f7a89f3d6395f2079c5 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:33.161392 2613 consensus_queue.cc:237] T 268439ff8d3a4f7a89f3d6395f2079c5 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.158210 1488 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 81ce448b148447ddbc41875d863e0518 (table default.Table_6e461d9cb0ae4dd095975bbc21533794 [id=49673fa73f8c436e9b8026efc4f3cbbc]). Expected version 1 got 0
I20250901 01:52:33.162958 1488 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 315c15cf0afc44d4a79cb29294281d60 (table default.Table_b80d3ae8523443b389b558591480b627 [id=aed8d4b4fa6a4fbf850cf5718610108f]). Expected version 2 got 0
I20250901 01:52:33.163271 1488 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet c83ce95fd6fa41478de0e2168be201b1 (table default.Table_a525265208584713b1b3465d63130cd9 [id=bfd87b9b708449d3b16e3e0639561a49]). Expected version 1 got 0
I20250901 01:52:33.163915 1488 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 5f5396045979450193d461ba1334a5c9 (table default.Table_2b946a63bd574ac0971fa3d3b9c9a6a4 [id=9c3930e9fcfe4b49ba4bea67a2b0ad07]). Expected version 2 got 0
I20250901 01:52:33.164301 1488 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 2973b59bb1df41d18d9a5b6a29ae336f (table default.Table_7790af96855e45c0a316ba3a4a87c9ed [id=746845bd84c6444b8a1e5683eaba4a68]). Expected version 1 got 0
I20250901 01:52:33.164512 1488 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 5d68ea97c71942c7b55b967ba870e0d7 (table default.Table_f610cb75906d4c5cb3b57cfdb80ecefc [id=36915a3e843340ba8ef3578ac35a9af4]). Expected version 1 got 0
I20250901 01:52:33.164700 1488 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet b2b172757e50404aae78e818921a7d9d (table default.Table_2e67ad583a0b413e9190cb55137ccf3a [id=5419e36671ec470a98dff53acdbd4760]). Expected version 1 got 0
I20250901 01:52:33.164923 1488 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet ae21b9ae2e3b472eb67fb5f96388a50f (table default.Table_046289f393ff432aae0b033bdec1fb17 [id=db57b18a9ce24e068b362f5fbe940531]). Expected version 1 got 0
I20250901 01:52:33.165138 1488 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet e9f1f892658b4736859ff507536920a1 (table default.Table_8825dd28afbe464193eb1b668cc9326a [id=9aa62cb634314a0a849612c54c1220d9]). Expected version 1 got 0
I20250901 01:52:33.165844 1151 ts_tablet_manager.cc:1428] T c6f7c525f59b49938b697de98d4b5835 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.019s user 0.002s sys 0.002s
I20250901 01:52:33.166122 1151 tablet_bootstrap.cc:492] T 26c8ed083cea4dc0acca80d00b11b29b P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:33.167774 1151 tablet_bootstrap.cc:654] T 26c8ed083cea4dc0acca80d00b11b29b P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:33.169540 1150 ts_tablet_manager.cc:1428] T d0463a8d655d4db4800b4e1b221c8315 P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.039s user 0.002s sys 0.000s
I20250901 01:52:33.169837 1150 tablet_bootstrap.cc:492] T bd9b59f0864a497faf281edd6077668d P 00c282f779c1477d920f24dc0dc16628: Bootstrap starting.
I20250901 01:52:33.173451 1150 tablet_bootstrap.cc:654] T bd9b59f0864a497faf281edd6077668d P 00c282f779c1477d920f24dc0dc16628: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:33.179078 1488 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 93bcac1fd95d44dc927abfca81692921 (table default.Table_b8f8ee74648c406eaf589d3ab80934db [id=b8f69313439543be9c11eea88dfa69c5]). Expected version 1 got 0
I20250901 01:52:33.179359 1488 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet c6f7c525f59b49938b697de98d4b5835 (table default.Table_db61e055dbd7450898e82c4c0c4763be [id=aabca67f6f754820977ac9c4c20f3e4f]). Expected version 1 got 0
I20250901 01:52:33.180629 1488 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 0e4009963cfe42d6ba164fc828897f09 (table default.Table_1adc70cccd2f4cb3a79df96861cb4181 [id=24fffc7ebf5a47bdb37211910301acf7]). Expected version 1 got 0
I20250901 01:52:33.181574 1488 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 45f6e8ba99b74898a2922b36768c6c36 (table default.Table_d1df9c0ae1a04b21a133e93cf4ae363a [id=9ce26dcf307149b595a43b6ac4562fba]). Expected version 2 got 0
I20250901 01:52:33.181746 1488 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet dabebc77f72e4bb19e264be2f496202a (table default.Table_2755f7caf53340afbd230181643366db [id=a4d0af97df5949cca940a81a41b8e454]). Expected version 1 got 0
I20250901 01:52:33.181929 1488 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 98c56eb62ad84f79b04f17a43d44c479 (table default.Table_50ff85e8f1314f4499e76299441bc110 [id=00e7f7fe6dc44f2787b9dd8e3cbf4a51]). Expected version 1 got 0
I20250901 01:52:33.182183 1488 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 6c0abbfc8d904dce989ce51961e4237d (table default.Table_3f9422b08b0349b9a5466218f0fe1b06 [id=f0dfb3811e744bd4ba6d59195cebde6f]). Expected version 2 got 0
I20250901 01:52:33.185465 1151 tablet_bootstrap.cc:492] T 26c8ed083cea4dc0acca80d00b11b29b P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:33.185653 1151 ts_tablet_manager.cc:1397] T 26c8ed083cea4dc0acca80d00b11b29b P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.002s
I20250901 01:52:33.182431 1488 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 508406601c994e42a5c5b1f665e74957 (table default.Table_7f5b278599994063862981b6b041ca8c [id=4aabc328f6374ec5b1dea26f778208b2]). Expected version 2 got 0
I20250901 01:52:33.186455 1151 raft_consensus.cc:357] T 26c8ed083cea4dc0acca80d00b11b29b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.186746 1151 raft_consensus.cc:383] T 26c8ed083cea4dc0acca80d00b11b29b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:33.186903 1151 raft_consensus.cc:738] T 26c8ed083cea4dc0acca80d00b11b29b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:33.186904 1488 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet aec951f2cc84499b9162b9fa24d5440e (table default.Table_d8e839ac6396439197b8b727af6a31b7 [id=55c31cd42fac43e6b8eddc048f565d0a]). Expected version 2 got 1
I20250901 01:52:33.187180 1151 consensus_queue.cc:260] T 26c8ed083cea4dc0acca80d00b11b29b P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.187306 1488 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet b164b4a67a9143ceaff69b0ce49d63fe (table default.Table_4505d77e4269443995012fef50c7e7b5 [id=c1b1e9e3876244f68d31c63ce7bdebf1]). Expected version 2 got 0
I20250901 01:52:33.186470 2113 ts_tablet_manager.cc:1918] T d10c193452954c198bba686169b8025b P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:33.187644 2113 log.cc:1199] T d10c193452954c198bba686169b8025b P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/d10c193452954c198bba686169b8025b
I20250901 01:52:33.187678 1488 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 858218c4427c41e980b2b5c969a5af40 (table default.Table_6d47266af0ed433eb1f2abbc81d18e69 [id=264cc469a73841e79538843479a165ab]). Expected version 1 got 0
I20250901 01:52:33.187949 1151 ts_tablet_manager.cc:1428] T 26c8ed083cea4dc0acca80d00b11b29b P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250901 01:52:33.188024 2113 ts_tablet_manager.cc:1939] T d10c193452954c198bba686169b8025b P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:33.188300 1151 tablet_bootstrap.cc:492] T 4fca1435d82447f289fda4035aa2ed89 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:33.188606 1488 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet d0c3c363dda444519a17dca63ad9caa7 (table default.Table_42422c44ca254d8ca989134c2433391f [id=8c1b263824ec4c15a5ec424364c78895]). Expected version 3 got 0
I20250901 01:52:33.188742 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 231) took 6019 ms. Trace:
I20250901 01:52:33.188866 1488 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 33f637f6a8134215bb1995ec38440073 (table default.Table_3d5afc22c4e14d6990aea31564dfde20 [id=557afe29976d40f6bfb6ae4ca348f887]). Expected version 2 got 0
I20250901 01:52:33.188824 2113 rpcz_store.cc:276] 0901 01:52:27.169697 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:27.169763 (+ 66us) service_pool.cc:225] Handling call
0901 01:52:33.107159 (+5937396us) ts_tablet_manager.cc:1145] Deleting tablet d10c193452954c198bba686169b8025b
0901 01:52:33.107172 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:33.143922 (+ 36750us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:33.186369 (+ 42447us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:33.188724 (+ 2355us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5937001}
I20250901 01:52:33.189116 2113 tablet_replica.cc:331] T ff013ea889d44cba8797834e4611b489 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:33.189266 1485 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 65e5cdc10e9642b7b0df775cb02861fa
I20250901 01:52:33.189286 2113 raft_consensus.cc:2241] T ff013ea889d44cba8797834e4611b489 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:33.189553 2113 raft_consensus.cc:2270] T ff013ea889d44cba8797834e4611b489 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:33.189612 1485 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_bbbde49530e34a76b7cb17988ac6baca" } new_table_name: "default.Table_69575b1a9efd4573a2ee01d3f432800b" modify_external_catalogs: true
I20250901 01:52:33.190078 2113 ts_tablet_manager.cc:1905] T ff013ea889d44cba8797834e4611b489 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:33.190568 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet d10c193452954c198bba686169b8025b (table default.Table_03fb48eaab9b42858ab7cefb722e43a8 [id=6c0576410d7340d6aa493a747e95565b]) successfully deleted
I20250901 01:52:33.196007 1151 tablet_bootstrap.cc:654] T 4fca1435d82447f289fda4035aa2ed89 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:33.196175 1483 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: default.Table_ce2f3b515c8f4c1bb557c5ed84b48dba [id=0baa4993b72e4a8d94ee3059c9d2db6c] alter complete (version 2)
I20250901 01:52:33.197539 2608 raft_consensus.cc:491] T 5d68ea97c71942c7b55b967ba870e0d7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:33.197693 2608 raft_consensus.cc:513] T 5d68ea97c71942c7b55b967ba870e0d7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.197854 1492 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: default.Table_a2a8011bb7624d9ea71e7b3169762e43 [id=78cd71321e9a480ca2d7d88327f157ee] alter complete (version 3)
I20250901 01:52:33.198392 2608 leader_election.cc:290] T 5d68ea97c71942c7b55b967ba870e0d7 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:33.198783 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a29fbe99693247d3a46732cd38cb41e1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.199107 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d68ea97c71942c7b55b967ba870e0d7" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:33.199659 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_396e4d6d66ac4996b45a0cbd6b9a044c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:33.199934 859 leader_election.cc:343] T 5d68ea97c71942c7b55b967ba870e0d7 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
W20250901 01:52:33.200337 1486 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 315c15cf0afc44d4a79cb29294281d60 consensus state has no leader
I20250901 01:52:33.200428 1490 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 3239e58531d445fa80165cf678bc277c
W20250901 01:52:33.200609 1490 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 3239e58531d445fa80165cf678bc277c: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:33.201071 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_98244ec9ac62429081da0667165323b3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.201749 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c536f1db280e483c96694828e4becf98"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:33.201975 858 leader_election.cc:336] T 5d68ea97c71942c7b55b967ba870e0d7 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:33.202070 858 leader_election.cc:304] T 5d68ea97c71942c7b55b967ba870e0d7 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:33.202442 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ec688eb2f1ab4e1fb660ffa3a2da8677"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.202590 2608 raft_consensus.cc:2747] T 5d68ea97c71942c7b55b967ba870e0d7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:33.203121 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_19e3e31c2cf348d187865d08696ca689"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.203774 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a6cb269c244242dc94bc04b179ac6ad5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.204474 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_00380e3a517b4c7fbc7275cda3e2b482"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.204752 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e10aa8ede3924addb2a04f03ccf5fbe7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.205054 1486 master_service.cc:948] ReplaceTablet: received request to replace tablet e6c10426f60f43e8bde69d9b1b9c8eb6 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:33.205404 1490 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_66ad08ce0b6840d4848bcddb304cf88b" } new_table_name: "default.Table_85b90ce84a904990b15be24c0acea21a" modify_external_catalogs: true
I20250901 01:52:33.207765 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 3239e58531d445fa80165cf678bc277c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:32 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:33.207784 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_cf85525bfde84419843461fec291bc20"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.208655 1485 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_f9e2a9182f6340bfa551fa2fc2eb2429" } new_table_name: "default.Table_d24e3a18ef4843fdb7b74ce23d47cf9e" modify_external_catalogs: true
I20250901 01:52:33.210595 1489 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet b009cc95c819431ea9827a3e453b8da3
W20250901 01:52:33.210772 1489 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet b009cc95c819431ea9827a3e453b8da3: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:33.211303 1489 master_service.cc:948] ReplaceTablet: received request to replace tablet 268439ff8d3a4f7a89f3d6395f2079c5 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:33.211851 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet b009cc95c819431ea9827a3e453b8da3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:32 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:33.214056 1484 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet d27c8d99528a42ed8dd1c5a7f373bbae
W20250901 01:52:33.214217 1484 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet d27c8d99528a42ed8dd1c5a7f373bbae: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:33.214701 1484 master_service.cc:948] ReplaceTablet: received request to replace tablet d0463a8d655d4db4800b4e1b221c8315 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:33.214923 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet d27c8d99528a42ed8dd1c5a7f373bbae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:32 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:33.215016 1482 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 49aaaec611784e6bac131efd58dbca51
W20250901 01:52:33.215183 1482 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 49aaaec611784e6bac131efd58dbca51: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:33.215823 1482 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_bcd502e1360b48b1953fa97eadce7006" } new_table_name: "default.Table_9f86b15283e34dd88287f72985404935" modify_external_catalogs: true
I20250901 01:52:33.215899 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 49aaaec611784e6bac131efd58dbca51 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:32 UTC) from {username='slave'} at 127.0.0.1:35946
W20250901 01:52:33.216001 992 consensus_peers.cc:489] T 85322a864d154f299f2d670fdd2a7bfb P fd27ab6665304aaea53adbd102dd4f58 -> Peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Couldn't send request to peer 00c282f779c1477d920f24dc0dc16628. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 85322a864d154f299f2d670fdd2a7bfb. This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:33.217249 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 3239e58531d445fa80165cf678bc277c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:32 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:33.217694 886 tablet_service.cc:1515] Processing DeleteTablet for tablet b009cc95c819431ea9827a3e453b8da3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:32 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:33.218057 886 tablet_service.cc:1515] Processing DeleteTablet for tablet d27c8d99528a42ed8dd1c5a7f373bbae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:32 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:33.218431 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 49aaaec611784e6bac131efd58dbca51 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:32 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:33.218834 1150 tablet_bootstrap.cc:492] T bd9b59f0864a497faf281edd6077668d P 00c282f779c1477d920f24dc0dc16628: No bootstrap required, opened a new log
I20250901 01:52:33.218973 1150 ts_tablet_manager.cc:1397] T bd9b59f0864a497faf281edd6077668d P 00c282f779c1477d920f24dc0dc16628: Time spent bootstrapping tablet: real 0.049s user 0.003s sys 0.001s
I20250901 01:52:33.219691 1150 raft_consensus.cc:357] T bd9b59f0864a497faf281edd6077668d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.219848 1150 raft_consensus.cc:383] T bd9b59f0864a497faf281edd6077668d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:33.219893 1150 raft_consensus.cc:738] T bd9b59f0864a497faf281edd6077668d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Initialized, Role: FOLLOWER
I20250901 01:52:33.220046 1150 consensus_queue.cc:260] T bd9b59f0864a497faf281edd6077668d P 00c282f779c1477d920f24dc0dc16628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.220535 1150 ts_tablet_manager.cc:1428] T bd9b59f0864a497faf281edd6077668d P 00c282f779c1477d920f24dc0dc16628: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250901 01:52:33.220639 1151 tablet_bootstrap.cc:492] T 4fca1435d82447f289fda4035aa2ed89 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:33.220749 1151 ts_tablet_manager.cc:1397] T 4fca1435d82447f289fda4035aa2ed89 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.033s user 0.008s sys 0.000s
I20250901 01:52:33.221482 1151 raft_consensus.cc:357] T 4fca1435d82447f289fda4035aa2ed89 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.221632 1151 raft_consensus.cc:383] T 4fca1435d82447f289fda4035aa2ed89 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:33.221675 1151 raft_consensus.cc:738] T 4fca1435d82447f289fda4035aa2ed89 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:33.221875 1151 consensus_queue.cc:260] T 4fca1435d82447f289fda4035aa2ed89 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.222311 1151 ts_tablet_manager.cc:1428] T 4fca1435d82447f289fda4035aa2ed89 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250901 01:52:33.222911 1151 tablet_bootstrap.cc:492] T f98d1d8d660443cb8066a547e5d9b77f P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:33.224339 2113 ts_tablet_manager.cc:1918] T ff013ea889d44cba8797834e4611b489 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:33.224460 2113 log.cc:1199] T ff013ea889d44cba8797834e4611b489 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/ff013ea889d44cba8797834e4611b489
I20250901 01:52:33.224871 2113 ts_tablet_manager.cc:1939] T ff013ea889d44cba8797834e4611b489 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:33.225126 1491 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: default.Table_685b7ca5d6e14076ad261d156b30858b [id=033c76e0598342c495ce2be42a0df5dd] alter complete (version 2)
I20250901 01:52:33.226711 1151 tablet_bootstrap.cc:654] T f98d1d8d660443cb8066a547e5d9b77f P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:33.227823 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 232) took 6037 ms. Trace:
I20250901 01:52:33.227936 2113 rpcz_store.cc:276] 0901 01:52:27.190850 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:27.190910 (+ 60us) service_pool.cc:225] Handling call
0901 01:52:33.189072 (+5998162us) ts_tablet_manager.cc:1145] Deleting tablet ff013ea889d44cba8797834e4611b489
0901 01:52:33.189082 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:33.202990 (+ 13908us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:33.224231 (+ 21241us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:33.227797 (+ 3566us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5997977}
I20250901 01:52:33.228327 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet ff013ea889d44cba8797834e4611b489 (table default.Table_0f91070385df441b96af4d5406a94724 [id=4747cdd47a954e0ab19609c9526b688d]) successfully deleted
I20250901 01:52:33.229794 2113 tablet_replica.cc:331] T 85322a864d154f299f2d670fdd2a7bfb P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:33.229965 1486 master_service.cc:948] ReplaceTablet: received request to replace tablet 6ba88af177874544b764cc32420e7ac9 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:33.229992 2113 raft_consensus.cc:2241] T 85322a864d154f299f2d670fdd2a7bfb P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:33.235548 2113 raft_consensus.cc:2270] T 85322a864d154f299f2d670fdd2a7bfb P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:33.236243 2113 ts_tablet_manager.cc:1905] T 85322a864d154f299f2d670fdd2a7bfb P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:33.245187 858 consensus_peers.cc:489] T 8ca5cf4e99014c0d84a3acad4ef713af P 00c282f779c1477d920f24dc0dc16628 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250901 01:52:33.245464 2608 raft_consensus.cc:491] T d27c8d99528a42ed8dd1c5a7f373bbae P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:33.247895 2608 raft_consensus.cc:513] T d27c8d99528a42ed8dd1c5a7f373bbae P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.248826 2608 leader_election.cc:290] T d27c8d99528a42ed8dd1c5a7f373bbae P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:33.249164 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d27c8d99528a42ed8dd1c5a7f373bbae" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:33.249650 859 leader_election.cc:343] T d27c8d99528a42ed8dd1c5a7f373bbae P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
W20250901 01:52:33.256079 858 leader_election.cc:336] T d27c8d99528a42ed8dd1c5a7f373bbae P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:33.256590 858 leader_election.cc:304] T d27c8d99528a42ed8dd1c5a7f373bbae P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:33.256911 1485 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5a7fa68d29644ae8afd2711e39dde59a has no consensus state
I20250901 01:52:33.257414 1492 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: default.Table_76438e5616384140b57dafdc896f11c5 [id=cecdd8d6ca6d445c9fd82decd2a95366] alter complete (version 3)
I20250901 01:52:33.261662 2357 raft_consensus.cc:2747] T d27c8d99528a42ed8dd1c5a7f373bbae P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:33.262516 2608 raft_consensus.cc:491] T 61ddcf86c9f34726939cc116434f0822 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:33.262660 2608 raft_consensus.cc:513] T 61ddcf86c9f34726939cc116434f0822 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.263882 2608 leader_election.cc:290] T 61ddcf86c9f34726939cc116434f0822 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:33.264088 2608 raft_consensus.cc:491] T 2973b59bb1df41d18d9a5b6a29ae336f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:33.264117 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "61ddcf86c9f34726939cc116434f0822" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:33.264201 2608 raft_consensus.cc:513] T 2973b59bb1df41d18d9a5b6a29ae336f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
W20250901 01:52:33.264660 859 leader_election.cc:343] T 61ddcf86c9f34726939cc116434f0822 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:33.265125 2608 leader_election.cc:290] T 2973b59bb1df41d18d9a5b6a29ae336f P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:33.265355 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2973b59bb1df41d18d9a5b6a29ae336f" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:33.265796 859 leader_election.cc:343] T 2973b59bb1df41d18d9a5b6a29ae336f P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:33.267061 1151 tablet_bootstrap.cc:492] T f98d1d8d660443cb8066a547e5d9b77f P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:33.267199 1151 ts_tablet_manager.cc:1397] T f98d1d8d660443cb8066a547e5d9b77f P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.044s user 0.006s sys 0.008s
I20250901 01:52:33.267912 1151 raft_consensus.cc:357] T f98d1d8d660443cb8066a547e5d9b77f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.268082 1151 raft_consensus.cc:383] T f98d1d8d660443cb8066a547e5d9b77f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:33.268131 1151 raft_consensus.cc:738] T f98d1d8d660443cb8066a547e5d9b77f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:33.269389 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 172) took 5053 ms. Trace:
I20250901 01:52:33.270139 1151 consensus_queue.cc:260] T f98d1d8d660443cb8066a547e5d9b77f P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.271045 1151 ts_tablet_manager.cc:1428] T f98d1d8d660443cb8066a547e5d9b77f P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250901 01:52:33.271311 1151 tablet_bootstrap.cc:492] T 93bcac1fd95d44dc927abfca81692921 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
W20250901 01:52:33.270056 858 leader_election.cc:336] T 61ddcf86c9f34726939cc116434f0822 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:33.271543 858 leader_election.cc:304] T 61ddcf86c9f34726939cc116434f0822 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:33.271991 858 leader_election.cc:336] T 2973b59bb1df41d18d9a5b6a29ae336f P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:33.272487 858 leader_election.cc:304] T 2973b59bb1df41d18d9a5b6a29ae336f P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:33.273253 1151 tablet_bootstrap.cc:654] T 93bcac1fd95d44dc927abfca81692921 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:33.272208 2608 raft_consensus.cc:2747] T 61ddcf86c9f34726939cc116434f0822 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:33.271183 2118 rpcz_store.cc:276] 0901 01:52:28.215682 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.215891 (+ 209us) service_pool.cc:225] Handling call
0901 01:52:33.067213 (+4851322us) ts_tablet_manager.cc:1145] Deleting tablet 9fbd44641d9e43c3b592e388e4adf350
0901 01:52:33.067221 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:33.149944 (+ 82723us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:33.156266 (+ 6322us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:33.269369 (+113103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4851134}
I20250901 01:52:33.275708 2113 ts_tablet_manager.cc:1918] T 85322a864d154f299f2d670fdd2a7bfb P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20250901 01:52:33.275818 2113 log.cc:1199] T 85322a864d154f299f2d670fdd2a7bfb P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/85322a864d154f299f2d670fdd2a7bfb
I20250901 01:52:33.276074 2118 tablet_replica.cc:331] T 0cba5329ab31440999faec599d704633 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:33.276226 2113 ts_tablet_manager.cc:1939] T 85322a864d154f299f2d670fdd2a7bfb P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:33.276934 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 9fbd44641d9e43c3b592e388e4adf350 (table default.Table_2fbca8463d6546c9b8f5b018fc60251d [id=6fdd3d817dc74551923021bd716ba58b]) successfully deleted
I20250901 01:52:33.276329 2357 raft_consensus.cc:2747] T 2973b59bb1df41d18d9a5b6a29ae336f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:33.277448 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 240) took 5356 ms. Trace:
I20250901 01:52:33.277540 2113 rpcz_store.cc:276] 0901 01:52:27.920925 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:27.920979 (+ 54us) service_pool.cc:225] Handling call
0901 01:52:33.229735 (+5308756us) ts_tablet_manager.cc:1145] Deleting tablet 85322a864d154f299f2d670fdd2a7bfb
0901 01:52:33.229752 (+ 17us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:33.252994 (+ 23242us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:33.275593 (+ 22599us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:33.277310 (+ 1717us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5308596}
I20250901 01:52:33.278064 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 85322a864d154f299f2d670fdd2a7bfb (table default.Table_5a72e7206f604795915f3a577d37f874 [id=1d8f34cc14cb4e90a562ff4424b8577a]) successfully deleted
I20250901 01:52:33.278214 2118 raft_consensus.cc:2241] T 0cba5329ab31440999faec599d704633 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:33.278327 2113 tablet_replica.cc:331] T cb7d7e1796c0456f8de98fb812b9e374 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:33.279217 2113 raft_consensus.cc:2241] T cb7d7e1796c0456f8de98fb812b9e374 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:33.280018 2113 raft_consensus.cc:2270] T cb7d7e1796c0456f8de98fb812b9e374 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:33.289754 1151 tablet_bootstrap.cc:492] T 93bcac1fd95d44dc927abfca81692921 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:33.290047 1151 ts_tablet_manager.cc:1397] T 93bcac1fd95d44dc927abfca81692921 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
I20250901 01:52:33.291208 1151 raft_consensus.cc:357] T 93bcac1fd95d44dc927abfca81692921 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:33.291461 1151 raft_consensus.cc:383] T 93bcac1fd95d44dc927abfca81692921 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:33.291564 1151 raft_consensus.cc:738] T 93bcac1fd95d44dc927abfca81692921 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:33.291853 1151 consensus_queue.cc:260] T 93bcac1fd95d44dc927abfca81692921 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:33.292595 1151 ts_tablet_manager.cc:1428] T 93bcac1fd95d44dc927abfca81692921 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250901 01:52:33.292933 1151 tablet_bootstrap.cc:492] T f195b040a6ae4b7b83a8308ab0a12b37 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:33.294739 1151 tablet_bootstrap.cc:654] T f195b040a6ae4b7b83a8308ab0a12b37 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:33.281901 2113 ts_tablet_manager.cc:1905] T cb7d7e1796c0456f8de98fb812b9e374 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:33.304370 2118 raft_consensus.cc:2270] T 0cba5329ab31440999faec599d704633 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:33.311638 1151 tablet_bootstrap.cc:492] T f195b040a6ae4b7b83a8308ab0a12b37 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:33.311825 2118 ts_tablet_manager.cc:1905] T 0cba5329ab31440999faec599d704633 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:33.311868 1151 ts_tablet_manager.cc:1397] T f195b040a6ae4b7b83a8308ab0a12b37 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.001s
I20250901 01:52:33.312965 1151 raft_consensus.cc:357] T f195b040a6ae4b7b83a8308ab0a12b37 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.313217 1151 raft_consensus.cc:383] T f195b040a6ae4b7b83a8308ab0a12b37 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:33.313311 1151 raft_consensus.cc:738] T f195b040a6ae4b7b83a8308ab0a12b37 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:33.313589 1151 consensus_queue.cc:260] T f195b040a6ae4b7b83a8308ab0a12b37 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.313808 2113 ts_tablet_manager.cc:1918] T cb7d7e1796c0456f8de98fb812b9e374 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:33.313908 2113 log.cc:1199] T cb7d7e1796c0456f8de98fb812b9e374 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/cb7d7e1796c0456f8de98fb812b9e374
I20250901 01:52:33.314312 2113 ts_tablet_manager.cc:1939] T cb7d7e1796c0456f8de98fb812b9e374 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:33.314493 1151 ts_tablet_manager.cc:1428] T f195b040a6ae4b7b83a8308ab0a12b37 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250901 01:52:33.314795 1151 tablet_bootstrap.cc:492] T d0b7c84ec9394a20ad77add3808e555a P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:33.315073 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 241) took 5389 ms. Trace:
I20250901 01:52:33.315155 2113 rpcz_store.cc:276] 0901 01:52:27.925176 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:27.925229 (+ 53us) service_pool.cc:225] Handling call
0901 01:52:33.278286 (+5353057us) ts_tablet_manager.cc:1145] Deleting tablet cb7d7e1796c0456f8de98fb812b9e374
0901 01:52:33.278294 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:33.303768 (+ 25474us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:33.313709 (+ 9941us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:33.315055 (+ 1346us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5352864}
I20250901 01:52:33.315651 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet cb7d7e1796c0456f8de98fb812b9e374 (table default.Table_c4148158ed7b48a3946559ba978e72fd [id=327c5a63904540508142ab600c308757]) successfully deleted
I20250901 01:52:33.316841 1151 tablet_bootstrap.cc:654] T d0b7c84ec9394a20ad77add3808e555a P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:33.319816 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 243) took 5378 ms. Trace:
I20250901 01:52:33.319912 2113 rpcz_store.cc:276] 0901 01:52:27.941250 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:27.941301 (+ 51us) service_pool.cc:225] Handling call
0901 01:52:33.319688 (+5378387us) ts_tablet_manager.cc:1145] Deleting tablet 3697821cd29446bb908b020599df0db9
0901 01:52:33.319697 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:33.319800 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5378183}
I20250901 01:52:33.319988 2778 tablet.cc:1722] T aa424ac6ef1441e6b1cc2867b1c48186 P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250901 01:52:33.320439 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 3697821cd29446bb908b020599df0db9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3697821cd29446bb908b020599df0db9 already in progress: creating tablet
I20250901 01:52:33.321372 2779 tablet.cc:1722] T aa424ac6ef1441e6b1cc2867b1c48186 P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250901 01:52:33.325852 2778 tablet.cc:1722] T 3c1fd0820d8a4995abe027f685da7f7b P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250901 01:52:33.327270 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 249) took 5278 ms. Trace:
I20250901 01:52:33.327368 2113 rpcz_store.cc:276] 0901 01:52:28.048624 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.048779 (+ 155us) service_pool.cc:225] Handling call
0901 01:52:33.327156 (+5278377us) ts_tablet_manager.cc:1145] Deleting tablet 8dd587304e6342edbe96c9897b40e5c9
0901 01:52:33.327165 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:33.327250 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5278198}
W20250901 01:52:33.327811 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 8dd587304e6342edbe96c9897b40e5c9 because the tablet was not found. No further retry: Not found: Tablet not found: 8dd587304e6342edbe96c9897b40e5c9
I20250901 01:52:33.328730 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 251) took 5279 ms. Trace:
I20250901 01:52:33.329090 2113 rpcz_store.cc:276] 0901 01:52:28.048854 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.048895 (+ 41us) service_pool.cc:225] Handling call
0901 01:52:33.328645 (+5279750us) ts_tablet_manager.cc:1145] Deleting tablet f50f33de1b654663b9ef8a91e3436f11
0901 01:52:33.328655 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:33.328714 (+ 59us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5279269}
W20250901 01:52:33.329803 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet f50f33de1b654663b9ef8a91e3436f11 because the tablet was not found. No further retry: Not found: Tablet not found: f50f33de1b654663b9ef8a91e3436f11
I20250901 01:52:33.330513 2113 tablet_replica.cc:331] T 74b709cb5ec64006b2f12bead8217d80 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:33.330844 2113 raft_consensus.cc:2241] T 74b709cb5ec64006b2f12bead8217d80 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:33.331279 2113 raft_consensus.cc:2270] T 74b709cb5ec64006b2f12bead8217d80 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:33.331944 2113 ts_tablet_manager.cc:1905] T 74b709cb5ec64006b2f12bead8217d80 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:33.332361 2779 tablet.cc:1722] T 3c1fd0820d8a4995abe027f685da7f7b P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250901 01:52:33.342602 1151 tablet_bootstrap.cc:492] T d0b7c84ec9394a20ad77add3808e555a P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:33.343055 1151 ts_tablet_manager.cc:1397] T d0b7c84ec9394a20ad77add3808e555a P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.028s user 0.006s sys 0.000s
I20250901 01:52:33.343868 1151 raft_consensus.cc:357] T d0b7c84ec9394a20ad77add3808e555a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.344226 1151 raft_consensus.cc:383] T d0b7c84ec9394a20ad77add3808e555a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:33.344334 1151 raft_consensus.cc:738] T d0b7c84ec9394a20ad77add3808e555a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:33.344858 1151 consensus_queue.cc:260] T d0b7c84ec9394a20ad77add3808e555a P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.345606 1151 ts_tablet_manager.cc:1428] T d0b7c84ec9394a20ad77add3808e555a P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250901 01:52:33.345911 1151 tablet_bootstrap.cc:492] T 81ce448b148447ddbc41875d863e0518 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:33.347702 1151 tablet_bootstrap.cc:654] T 81ce448b148447ddbc41875d863e0518 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:33.357272 2357 raft_consensus.cc:491] T 2dacb608897d47d28512df87dc320927 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:33.357587 2113 ts_tablet_manager.cc:1918] T 74b709cb5ec64006b2f12bead8217d80 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:33.357461 2357 raft_consensus.cc:513] T 2dacb608897d47d28512df87dc320927 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.357694 2113 log.cc:1199] T 74b709cb5ec64006b2f12bead8217d80 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/74b709cb5ec64006b2f12bead8217d80
I20250901 01:52:33.358119 2113 ts_tablet_manager.cc:1939] T 74b709cb5ec64006b2f12bead8217d80 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:33.358167 2357 leader_election.cc:290] T 2dacb608897d47d28512df87dc320927 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:33.358987 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2dacb608897d47d28512df87dc320927" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:33.359589 859 leader_election.cc:343] T 2dacb608897d47d28512df87dc320927 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
W20250901 01:52:33.360539 556 rpcz_store.cc:267] Call kudu.server.GenericService.CheckLeaks from 127.0.0.1:37674 (request call id 30) took 906 ms (client timeout 1000 ms). Trace:
W20250901 01:52:33.360693 556 rpcz_store.cc:269] 0901 01:52:32.454141 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:32.455069 (+ 928us) service_pool.cc:225] Handling call
0901 01:52:33.360487 (+905418us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250901 01:52:33.362015 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskddMNdm/build/asan/bin/kudu with pid 492
I20250901 01:52:33.364055 1151 tablet_bootstrap.cc:492] T 81ce448b148447ddbc41875d863e0518 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:33.364257 1151 ts_tablet_manager.cc:1397] T 81ce448b148447ddbc41875d863e0518 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.002s
I20250901 01:52:33.365044 1151 raft_consensus.cc:357] T 81ce448b148447ddbc41875d863e0518 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.365258 1151 raft_consensus.cc:383] T 81ce448b148447ddbc41875d863e0518 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:33.365338 1151 raft_consensus.cc:738] T 81ce448b148447ddbc41875d863e0518 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:33.365641 1151 consensus_queue.cc:260] T 81ce448b148447ddbc41875d863e0518 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.366521 1151 ts_tablet_manager.cc:1428] T 81ce448b148447ddbc41875d863e0518 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250901 01:52:33.366845 1151 tablet_bootstrap.cc:492] T 5f5396045979450193d461ba1334a5c9 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:33.368602 1151 tablet_bootstrap.cc:654] T 5f5396045979450193d461ba1334a5c9 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:33.372699 2775 raft_consensus.cc:491] T da5f69c5b0ca4257ad2ff7918ee2af2e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:33.372848 2775 raft_consensus.cc:513] T da5f69c5b0ca4257ad2ff7918ee2af2e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.373466 2775 leader_election.cc:290] T da5f69c5b0ca4257ad2ff7918ee2af2e P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:33.374845 992 leader_election.cc:336] T da5f69c5b0ca4257ad2ff7918ee2af2e P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:33.375092 992 leader_election.cc:343] T da5f69c5b0ca4257ad2ff7918ee2af2e P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: da5f69c5b0ca4257ad2ff7918ee2af2e
I20250901 01:52:33.375207 992 leader_election.cc:304] T da5f69c5b0ca4257ad2ff7918ee2af2e P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:33.375422 2775 raft_consensus.cc:2747] T da5f69c5b0ca4257ad2ff7918ee2af2e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:33.376187 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 250) took 5327 ms. Trace:
I20250901 01:52:33.377081 2113 rpcz_store.cc:276] 0901 01:52:28.048745 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.049050 (+ 305us) service_pool.cc:225] Handling call
0901 01:52:33.330321 (+5281271us) ts_tablet_manager.cc:1145] Deleting tablet 74b709cb5ec64006b2f12bead8217d80
0901 01:52:33.330474 (+ 153us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:33.345644 (+ 15170us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:33.357491 (+ 11847us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:33.376170 (+ 18679us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5274220}
I20250901 01:52:33.377588 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 74b709cb5ec64006b2f12bead8217d80 (table default.Table_95884295301d476fb3e03054727f9cf1 [id=1aa18a2513db4e99a72572846527663c]) successfully deleted
I20250901 01:52:33.376904 1151 tablet_bootstrap.cc:492] T 5f5396045979450193d461ba1334a5c9 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:33.377959 1151 ts_tablet_manager.cc:1397] T 5f5396045979450193d461ba1334a5c9 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.005s
W20250901 01:52:33.378270 858 leader_election.cc:336] T 2dacb608897d47d28512df87dc320927 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:33.378463 858 leader_election.cc:304] T 2dacb608897d47d28512df87dc320927 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:33.378732 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 252) took 5321 ms. Trace:
I20250901 01:52:33.378820 2113 rpcz_store.cc:276] 0901 01:52:28.057767 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.057861 (+ 94us) service_pool.cc:225] Handling call
0901 01:52:33.378642 (+5320781us) ts_tablet_manager.cc:1145] Deleting tablet 64fb46b942ff4e4c8749986ecc3dfe65
0901 01:52:33.378652 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:33.378718 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5318198}
I20250901 01:52:33.379168 2357 raft_consensus.cc:2747] T 2dacb608897d47d28512df87dc320927 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:33.379246 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 64fb46b942ff4e4c8749986ecc3dfe65 because the tablet was not found. No further retry: Not found: Tablet not found: 64fb46b942ff4e4c8749986ecc3dfe65
I20250901 01:52:33.379819 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 253) took 5321 ms. Trace:
I20250901 01:52:33.379909 2113 rpcz_store.cc:276] 0901 01:52:28.060515 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.060559 (+ 44us) service_pool.cc:225] Handling call
0901 01:52:33.379732 (+5319173us) ts_tablet_manager.cc:1145] Deleting tablet 66a48a8ebe154b2cac89c234eaa378ff
0901 01:52:33.379742 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:33.379805 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5318942}
W20250901 01:52:33.380276 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 66a48a8ebe154b2cac89c234eaa378ff because the tablet was not found. No further retry: Not found: Tablet not found: 66a48a8ebe154b2cac89c234eaa378ff
I20250901 01:52:33.380786 2778 tablet.cc:1722] T f65ec84958eb488d8b0f812b59b5b3d5 P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250901 01:52:33.380928 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4a7f866079ea4e76b11cccaae84614d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.381153 1151 raft_consensus.cc:357] T 5f5396045979450193d461ba1334a5c9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.381302 1151 raft_consensus.cc:383] T 5f5396045979450193d461ba1334a5c9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:33.381350 1151 raft_consensus.cc:738] T 5f5396045979450193d461ba1334a5c9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:33.382054 2779 tablet.cc:1722] T f65ec84958eb488d8b0f812b59b5b3d5 P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250901 01:52:33.382054 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f0875e176094453dadcd2ed9aa37cee2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.382426 2113 tablet_replica.cc:331] T f3962b471bdb49aea6e9fa95f1f9dfe2 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:33.382632 2113 raft_consensus.cc:2241] T f3962b471bdb49aea6e9fa95f1f9dfe2 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:33.382771 2113 raft_consensus.cc:2270] T f3962b471bdb49aea6e9fa95f1f9dfe2 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:33.382925 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9f4c10f9e87b438b953150eba57b3c08"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.383239 2113 ts_tablet_manager.cc:1905] T f3962b471bdb49aea6e9fa95f1f9dfe2 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:33.383726 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1ca9eebea03a43b9840f80960f6a4113"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.384680 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_62c32a2e033f476bb5f34b0af5413283"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.385468 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9bb81305792a414d8559d70bec4606ab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.386209 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f346690e21bc4ff98158e0846f2bb943"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.386269 1151 consensus_queue.cc:260] T 5f5396045979450193d461ba1334a5c9 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.386961 1151 ts_tablet_manager.cc:1428] T 5f5396045979450193d461ba1334a5c9 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.009s user 0.000s sys 0.001s
I20250901 01:52:33.386914 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4e7b1fcb3b704a06be8b8d04961be939"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.387396 2613 raft_consensus.cc:491] T dabebc77f72e4bb19e264be2f496202a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:33.387493 1490 master_service.cc:948] ReplaceTablet: received request to replace tablet b13afacd37474781877b9f2cd20b313a from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:33.387506 2613 raft_consensus.cc:513] T dabebc77f72e4bb19e264be2f496202a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.388023 2613 leader_election.cc:290] T dabebc77f72e4bb19e264be2f496202a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:33.389209 992 leader_election.cc:336] T dabebc77f72e4bb19e264be2f496202a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:33.389324 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dabebc77f72e4bb19e264be2f496202a" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:33.389487 914 raft_consensus.cc:2466] T dabebc77f72e4bb19e264be2f496202a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:33.389762 992 leader_election.cc:304] T dabebc77f72e4bb19e264be2f496202a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:33.389978 2613 raft_consensus.cc:2802] T dabebc77f72e4bb19e264be2f496202a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:33.390043 2613 raft_consensus.cc:491] T dabebc77f72e4bb19e264be2f496202a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:33.390101 2613 raft_consensus.cc:3058] T dabebc77f72e4bb19e264be2f496202a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:33.390153 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_495df301fee9477393c65737742eb54c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.393162 2613 raft_consensus.cc:513] T dabebc77f72e4bb19e264be2f496202a P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.393729 2613 leader_election.cc:290] T dabebc77f72e4bb19e264be2f496202a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:33.395242 992 leader_election.cc:336] T dabebc77f72e4bb19e264be2f496202a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:33.395967 921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dabebc77f72e4bb19e264be2f496202a" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:33.396078 1482 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_85b90ce84a904990b15be24c0acea21a" } modify_external_catalogs: true
I20250901 01:52:33.396163 921 raft_consensus.cc:3058] T dabebc77f72e4bb19e264be2f496202a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:33.396257 1482 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_85b90ce84a904990b15be24c0acea21a" } modify_external_catalogs: true
I20250901 01:52:33.398092 1151 tablet_bootstrap.cc:492] T 61ddcf86c9f34726939cc116434f0822 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:33.400481 1151 tablet_bootstrap.cc:654] T 61ddcf86c9f34726939cc116434f0822 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:33.401131 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_653a1fad8db342f384e34337176cb4e3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.402120 921 raft_consensus.cc:2466] T dabebc77f72e4bb19e264be2f496202a P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:33.402128 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_689cd9e5419f43c88c5f03f0d38ef963"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.402595 992 leader_election.cc:304] T dabebc77f72e4bb19e264be2f496202a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:33.402829 2613 raft_consensus.cc:2802] T dabebc77f72e4bb19e264be2f496202a P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:33.402930 2613 raft_consensus.cc:695] T dabebc77f72e4bb19e264be2f496202a P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:33.403137 2613 consensus_queue.cc:237] T dabebc77f72e4bb19e264be2f496202a P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.403910 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a168e4c72cba4396b52b0ae45a97cbbb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.404729 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_855aac7c1a0e45a4918316ac9e0ac9a4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.407017 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f423a032a0094a9e9873b18bfe13b01e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:33.408721 992 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111) [suppressed 363 similar messages]
I20250901 01:52:33.409592 2608 raft_consensus.cc:491] T d0463a8d655d4db4800b4e1b221c8315 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:33.409741 2608 raft_consensus.cc:513] T d0463a8d655d4db4800b4e1b221c8315 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.410387 2608 leader_election.cc:290] T d0463a8d655d4db4800b4e1b221c8315 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:33.411471 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0463a8d655d4db4800b4e1b221c8315" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:33.411978 2613 raft_consensus.cc:491] T 26c8ed083cea4dc0acca80d00b11b29b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250901 01:52:33.411979 859 leader_election.cc:343] T d0463a8d655d4db4800b4e1b221c8315 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:33.412106 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_533a252a01ae4586ae54db48fe7cccf5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.412189 2613 raft_consensus.cc:513] T 26c8ed083cea4dc0acca80d00b11b29b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.412739 2613 leader_election.cc:290] T 26c8ed083cea4dc0acca80d00b11b29b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:33.413911 992 leader_election.cc:336] T 26c8ed083cea4dc0acca80d00b11b29b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:33.414727 2113 ts_tablet_manager.cc:1918] T f3962b471bdb49aea6e9fa95f1f9dfe2 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:33.415580 2113 log.cc:1199] T f3962b471bdb49aea6e9fa95f1f9dfe2 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/f3962b471bdb49aea6e9fa95f1f9dfe2
I20250901 01:52:33.416330 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26c8ed083cea4dc0acca80d00b11b29b" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:33.416721 914 raft_consensus.cc:2466] T 26c8ed083cea4dc0acca80d00b11b29b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:33.416775 1489 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 268439ff8d3a4f7a89f3d6395f2079c5
W20250901 01:52:33.416997 1489 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 268439ff8d3a4f7a89f3d6395f2079c5: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:33.417275 2113 ts_tablet_manager.cc:1939] T f3962b471bdb49aea6e9fa95f1f9dfe2 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:33.417373 1489 catalog_manager.cc:6652] ReplaceTablet: tablet 268439ff8d3a4f7a89f3d6395f2079c5 deleted and replaced by tablet 086aa948f93644da8e5835b0fac87ddd
I20250901 01:52:33.417932 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 254) took 5279 ms. Trace:
I20250901 01:52:33.418043 1484 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet d0463a8d655d4db4800b4e1b221c8315
I20250901 01:52:33.417999 2113 rpcz_store.cc:276] 0901 01:52:28.138637 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.140824 (+ 2187us) service_pool.cc:225] Handling call
0901 01:52:33.381539 (+5240715us) ts_tablet_manager.cc:1145] Deleting tablet f3962b471bdb49aea6e9fa95f1f9dfe2
0901 01:52:33.381549 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:33.402465 (+ 20916us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:33.414627 (+ 12162us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:33.417914 (+ 3287us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5240506}
W20250901 01:52:33.418223 1484 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet d0463a8d655d4db4800b4e1b221c8315: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:33.418373 992 leader_election.cc:304] T 26c8ed083cea4dc0acca80d00b11b29b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:33.418633 1484 catalog_manager.cc:6652] ReplaceTablet: tablet d0463a8d655d4db4800b4e1b221c8315 deleted and replaced by tablet 1429a6a7a20e4f7c995e9acbaa8d495c
I20250901 01:52:33.418637 2613 raft_consensus.cc:2802] T 26c8ed083cea4dc0acca80d00b11b29b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:33.418709 2613 raft_consensus.cc:491] T 26c8ed083cea4dc0acca80d00b11b29b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:33.418771 2613 raft_consensus.cc:3058] T 26c8ed083cea4dc0acca80d00b11b29b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
W20250901 01:52:33.419545 858 leader_election.cc:336] T d0463a8d655d4db4800b4e1b221c8315 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:33.419679 858 leader_election.cc:304] T d0463a8d655d4db4800b4e1b221c8315 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:33.419970 2608 raft_consensus.cc:2747] T d0463a8d655d4db4800b4e1b221c8315 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:33.421689 2613 raft_consensus.cc:513] T 26c8ed083cea4dc0acca80d00b11b29b P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.422263 2613 leader_election.cc:290] T 26c8ed083cea4dc0acca80d00b11b29b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
E20250901 01:52:33.422905 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 0c88c339e59a4c1da21756cba0645bdd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d25639c7008f4e29af6226039e4b976b'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:33.422886 921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26c8ed083cea4dc0acca80d00b11b29b" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
E20250901 01:52:33.423044 1696 catalog_manager.cc:6205] Error selecting replicas for tablet a656adbf1b29431687e0989f0db3459c, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2d3d2227e6a24bd2a86f457e553e698b'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:33.423060 921 raft_consensus.cc:3058] T 26c8ed083cea4dc0acca80d00b11b29b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
E20250901 01:52:33.423168 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 494b6488aa0d4355b4caa46004267465, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_3ef97ecacc3042099ec1cc5c3a7a0763'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.423290 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 0ee3f35392954817b350e5f48653be78, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_442e11b9211c4b209c68f653d7bfc313'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.423393 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae4844183a3c4a04a775e6fd878391a3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_aee74198b51d4f089538c4f2bf3ce4dc'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.423521 1696 catalog_manager.cc:6205] Error selecting replicas for tablet be8dfbc466564842a2fdb4953eee939b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_75ba3ba70ca24a94920e7bce96190ff2'. Need at least 3 replicas, but only 2 tablet servers are available
W20250901 01:52:33.423460 992 leader_election.cc:336] T 26c8ed083cea4dc0acca80d00b11b29b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
E20250901 01:52:33.423605 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9745582cebfe47b58c21d0191ab4cb9b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77728af9eee44f29a1e2131d5b75b7ca'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.423713 1696 catalog_manager.cc:6205] Error selecting replicas for tablet e2ed32b3292141278af28026528e96c2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b5077f55951d48be92a67407ffe26a08'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.423810 1696 catalog_manager.cc:6205] Error selecting replicas for tablet eea6bfabe6cc4f83905c24f0f7a5dfc9, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_e0902dfbdd984f85b3fdac8d283731f7'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.423914 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9f62a9d97f514747b940b488907369f3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d53a456dbca64daf92aeb5fb4517b417'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.424008 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5a7fa68d29644ae8afd2711e39dde59a, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d24e3a18ef4843fdb7b74ce23d47cf9e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.424121 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ff2bd017417c41b5b8225b8f00283e63, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_97526c9a6c9f4acbb6dedd78475571e0'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.424213 1696 catalog_manager.cc:6205] Error selecting replicas for tablet dcc60bb000f440f1aca9133928b5ce20, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2fbca8463d6546c9b8f5b018fc60251d'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.424312 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f8f923ddcd2740bcb7e521f4f57c4034, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_5e65c5f068b84995a2d7a76fa65dda0f'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:33.425184 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 268439ff8d3a4f7a89f3d6395f2079c5 with delete_type TABLET_DATA_DELETED (replaced by tablet 086aa948f93644da8e5835b0fac87ddd) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:33.425405 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet d0463a8d655d4db4800b4e1b221c8315 with delete_type TABLET_DATA_DELETED (replaced by tablet 1429a6a7a20e4f7c995e9acbaa8d495c) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:33.426672 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet f3962b471bdb49aea6e9fa95f1f9dfe2 (table default.Table_e2ea7aa90231448888c6cfead7e3d9c1 [id=c264ed0bf1d84bc9bd5c2bb0d1e8a45b]) successfully deleted
E20250901 01:52:33.427099 1696 catalog_manager.cc:6205] Error selecting replicas for tablet e4d8330a4e454d1b8d643e17a6996741, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ce16a1b94f714c2699ee6cd916bb0215'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:33.427771 2613 raft_consensus.cc:491] T d0b7c84ec9394a20ad77add3808e555a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:33.427944 2613 raft_consensus.cc:513] T d0b7c84ec9394a20ad77add3808e555a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.429855 2613 leader_election.cc:290] T d0b7c84ec9394a20ad77add3808e555a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:33.430470 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0b7c84ec9394a20ad77add3808e555a" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:33.430670 914 raft_consensus.cc:2466] T d0b7c84ec9394a20ad77add3808e555a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:33.431005 992 leader_election.cc:304] T d0b7c84ec9394a20ad77add3808e555a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:33.431221 2613 raft_consensus.cc:2802] T d0b7c84ec9394a20ad77add3808e555a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:33.431269 2118 ts_tablet_manager.cc:1918] T 0cba5329ab31440999faec599d704633 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250901 01:52:33.431217 992 leader_election.cc:336] T d0b7c84ec9394a20ad77add3808e555a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:33.431381 2118 log.cc:1199] T 0cba5329ab31440999faec599d704633 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/0cba5329ab31440999faec599d704633
I20250901 01:52:33.431476 2613 raft_consensus.cc:491] T d0b7c84ec9394a20ad77add3808e555a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:33.431614 2613 raft_consensus.cc:3058] T d0b7c84ec9394a20ad77add3808e555a P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:33.431717 2118 ts_tablet_manager.cc:1939] T 0cba5329ab31440999faec599d704633 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:33.433866 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 268439ff8d3a4f7a89f3d6395f2079c5 with delete_type TABLET_DATA_DELETED (replaced by tablet 086aa948f93644da8e5835b0fac87ddd) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:33.434377 886 tablet_service.cc:1515] Processing DeleteTablet for tablet d0463a8d655d4db4800b4e1b221c8315 with delete_type TABLET_DATA_DELETED (replaced by tablet 1429a6a7a20e4f7c995e9acbaa8d495c) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:33.434355 2613 raft_consensus.cc:513] T d0b7c84ec9394a20ad77add3808e555a P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
E20250901 01:52:33.434829 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 0813cc68d4b0488ba60a523e4479971b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d1302673d68f424ebb54b220911ce385'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.435016 1696 catalog_manager.cc:6205] Error selecting replicas for tablet b1ba0e19dc0b462e80c537bf2d183f5b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_66b111271c15410aab23682f8f6081ac'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:33.436759 2613 leader_election.cc:290] T d0b7c84ec9394a20ad77add3808e555a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:33.437134 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0b7c84ec9394a20ad77add3808e555a" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:33.437292 914 raft_consensus.cc:3058] T d0b7c84ec9394a20ad77add3808e555a P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
E20250901 01:52:33.437742 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 8e10a01e77d14cf28c2c5e0205e595b9, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_83cf2d86a1284544ba2e7aa9ebb66bb7'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.437885 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 05a547a8866149cb95d23e960a737237, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_583e53c2e86544e9be68eb27ab6068a3'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.438753 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5f0e054f62de4cf6bc6f69ad6664b192, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ef6832d0845044ed9f165ca3541b4c3f'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.438876 1696 catalog_manager.cc:6205] Error selecting replicas for tablet fd2ce82a4ee24d0faa44980a3967e187, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_63d0f2ead9d946d39ce40162ebd769d6'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.439400 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae9a4ad4fdfb4771ba23a91f7007447e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_5e84ae9a9c2a4084a1f9c5ffb4594de2'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.439540 1696 catalog_manager.cc:6205] Error selecting replicas for tablet d602c8986c75492ead9936ac04e2622d, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_82d91a1ba2eb4265896824028dec8d3a'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:33.439709 914 raft_consensus.cc:2466] T d0b7c84ec9394a20ad77add3808e555a P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
E20250901 01:52:33.439797 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 93aa22afc0ad4d7fa24807ea3c5d5c21, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0f4b4094d2974b7591875375eda6bed0'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.439913 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9074543dea9949e7a085717b40913348, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_53e30205c5fd488eb3b48c685a159d04'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:33.440055 1486 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 6ba88af177874544b764cc32420e7ac9
E20250901 01:52:33.440243 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 3a62034e688348ed9e2ac9de009ed3a2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_409dad98d7eb455a8e16edfb4acedb34'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.440352 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 18498c9cb09141d08c1c5c0854b63ebd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77f4810bc83c4d948531da2a45a3792e'. Need at least 3 replicas, but only 2 tablet servers are available
W20250901 01:52:33.440533 992 leader_election.cc:336] T d0b7c84ec9394a20ad77add3808e555a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:33.440645 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 274) took 5300 ms. Trace:
E20250901 01:52:33.440714 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f5e3316adeae47cd848833dd84b972bb, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7570c3012f8a4ae7bcfc7346eadac92a'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:33.440709 2113 rpcz_store.cc:276] 0901 01:52:28.140158 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.141048 (+ 890us) service_pool.cc:225] Handling call
0901 01:52:33.423992 (+5282944us) ts_tablet_manager.cc:1145] Deleting tablet b6daeff0070a4db08e90020752d21c29
0901 01:52:33.440532 (+ 16540us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:33.440631 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5282827}
I20250901 01:52:33.441037 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 275) took 5300 ms. Trace:
I20250901 01:52:33.441093 2113 rpcz_store.cc:276] 0901 01:52:28.140243 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.141171 (+ 928us) service_pool.cc:225] Handling call
0901 01:52:33.440966 (+5299795us) ts_tablet_manager.cc:1145] Deleting tablet 4410da9e95454ecf95faf80221b0a71c
0901 01:52:33.440975 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:33.441025 (+ 50us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5299708}
I20250901 01:52:33.441308 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 276) took 5300 ms. Trace:
I20250901 01:52:33.441363 2113 rpcz_store.cc:276] 0901 01:52:28.140329 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.141265 (+ 936us) service_pool.cc:225] Handling call
0901 01:52:33.441256 (+5299991us) ts_tablet_manager.cc:1145] Deleting tablet 0cb3262973934c8689b851adad60afe0
0901 01:52:33.441262 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:33.441299 (+ 37us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5299886}
I20250901 01:52:33.441562 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 255) took 5302 ms. Trace:
I20250901 01:52:33.441612 2113 rpcz_store.cc:276] 0901 01:52:28.138731 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.141398 (+ 2667us) service_pool.cc:225] Handling call
0901 01:52:33.441506 (+5300108us) ts_tablet_manager.cc:1145] Deleting tablet 3fda93767e494a4684978c8522b829dd
0901 01:52:33.441513 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:33.441552 (+ 39us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5300002}
I20250901 01:52:33.441831 2113 tablet_replica.cc:331] T b85d64e02cc646e380581ac9f59dee9e P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:33.442040 2113 raft_consensus.cc:2241] T b85d64e02cc646e380581ac9f59dee9e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:33.442236 2113 raft_consensus.cc:2270] T b85d64e02cc646e380581ac9f59dee9e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
E20250901 01:52:33.442401 1696 catalog_manager.cc:6205] Error selecting replicas for tablet a6b539e160194da3a849d9e2f1ec9859, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_03fb48eaab9b42858ab7cefb722e43a8'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:33.442572 1482 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet f65ec84958eb488d8b0f812b59b5b3d5
W20250901 01:52:33.442740 1482 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet f65ec84958eb488d8b0f812b59b5b3d5: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:33.442791 2113 ts_tablet_manager.cc:1905] T b85d64e02cc646e380581ac9f59dee9e P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:33.443723 992 leader_election.cc:304] T d0b7c84ec9394a20ad77add3808e555a P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:33.443926 2775 raft_consensus.cc:2802] T d0b7c84ec9394a20ad77add3808e555a P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:33.444000 886 tablet_service.cc:1515] Processing DeleteTablet for tablet f65ec84958eb488d8b0f812b59b5b3d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:33 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:33.444031 2775 raft_consensus.cc:695] T d0b7c84ec9394a20ad77add3808e555a P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:33.443970 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_450413993093444ea9946ab808cac59d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.444308 2775 consensus_queue.cc:237] T d0b7c84ec9394a20ad77add3808e555a P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.444851 921 raft_consensus.cc:2466] T 26c8ed083cea4dc0acca80d00b11b29b P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:33.445034 1482 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_608d053d56f7424dbba42177d5e72fe3" } new_table_name: "default.Table_62eb2015c3dc4feda347dae9fb970abd" modify_external_catalogs: true
I20250901 01:52:33.445631 1483 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: default.Table_608d053d56f7424dbba42177d5e72fe3 [id=18aecbf3fe4643c4b589ee47706161bd] alter complete (version 4)
I20250901 01:52:33.446383 2780 consensus_queue.cc:1035] T eb19e3a424324a24824f6caaf5848952 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:33.447041 992 leader_election.cc:304] T 26c8ed083cea4dc0acca80d00b11b29b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:33.447614 2781 raft_consensus.cc:2802] T 26c8ed083cea4dc0acca80d00b11b29b P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:33.447719 2781 raft_consensus.cc:695] T 26c8ed083cea4dc0acca80d00b11b29b P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:33.447961 2781 consensus_queue.cc:237] T 26c8ed083cea4dc0acca80d00b11b29b P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:33.448434 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet b6daeff0070a4db08e90020752d21c29 because the tablet was not found. No further retry: Not found: Tablet not found: b6daeff0070a4db08e90020752d21c29
W20250901 01:52:33.449131 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 4410da9e95454ecf95faf80221b0a71c because the tablet was not found. No further retry: Not found: Tablet not found: 4410da9e95454ecf95faf80221b0a71c
I20250901 01:52:33.449348 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet f65ec84958eb488d8b0f812b59b5b3d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:33 UTC) from {username='slave'} at 127.0.0.1:35946
W20250901 01:52:33.449388 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 0cb3262973934c8689b851adad60afe0 because the tablet was not found. No further retry: Not found: Tablet not found: 0cb3262973934c8689b851adad60afe0
W20250901 01:52:33.449745 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 3fda93767e494a4684978c8522b829dd because the tablet was not found. No further retry: Not found: Tablet not found: 3fda93767e494a4684978c8522b829dd
W20250901 01:52:33.450920 992 consensus_peers.cc:489] T 081e2630e284498aa819f85bf8326315 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250901 01:52:33.455722 2608 raft_consensus.cc:491] T 5f5396045979450193d461ba1334a5c9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:33.455920 2608 raft_consensus.cc:513] T 5f5396045979450193d461ba1334a5c9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.456614 2608 leader_election.cc:290] T 5f5396045979450193d461ba1334a5c9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:33.457265 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f5396045979450193d461ba1334a5c9" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:33.457464 1052 raft_consensus.cc:2466] T 5f5396045979450193d461ba1334a5c9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00c282f779c1477d920f24dc0dc16628 in term 0.
I20250901 01:52:33.458323 859 leader_election.cc:304] T 5f5396045979450193d461ba1334a5c9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
E20250901 01:52:33.458913 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 6ce9521b925b4f97bb4a810ce77e147e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_9e04f976a93f4d30baccf90fa3622dfe'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.459070 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2f61edb50d7a42198bfd70639c7bf4de, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_a5c783cbdb694af69047b31d99ff8def'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:33.459551 2608 raft_consensus.cc:2802] T 5f5396045979450193d461ba1334a5c9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:33.459637 2608 raft_consensus.cc:491] T 5f5396045979450193d461ba1334a5c9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:33.459710 2608 raft_consensus.cc:3058] T 5f5396045979450193d461ba1334a5c9 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:33.460453 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_83a5ae89ddca4c5f9f40b738e98699e1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.461424 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3c97ce6dab4541dc8f149a10a487c0d7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.462188 1484 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_4aa01c0e60364804a5e99d552ca3956c" } modify_external_catalogs: true
I20250901 01:52:33.462347 1484 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_4aa01c0e60364804a5e99d552ca3956c" } modify_external_catalogs: true
I20250901 01:52:33.462386 2608 raft_consensus.cc:513] T 5f5396045979450193d461ba1334a5c9 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.463467 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f5396045979450193d461ba1334a5c9" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58"
I20250901 01:52:33.463469 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_eadd4dd2bc5a4f70a36bbc5629fa4fd7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.463688 1052 raft_consensus.cc:3058] T 5f5396045979450193d461ba1334a5c9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:33.463740 2608 leader_election.cc:290] T 5f5396045979450193d461ba1334a5c9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:33.464218 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f2fe0ef6b5784dd9a6046aabf0cc2472"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:33.465096 858 leader_election.cc:336] T 5f5396045979450193d461ba1334a5c9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:33.464996 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8a8adaa9c6134c98b9614cd6bdfe7677"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.465811 1489 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_d25639c7008f4e29af6226039e4b976b" } new_table_name: "default.Table_0927e3ea788d43bda5afeda483d87176" modify_external_catalogs: true
W20250901 01:52:33.466284 858 leader_election.cc:336] T 5f5396045979450193d461ba1334a5c9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:33.466921 1486 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 6ba88af177874544b764cc32420e7ac9: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:33.467329 1486 catalog_manager.cc:6652] ReplaceTablet: tablet 6ba88af177874544b764cc32420e7ac9 deleted and replaced by tablet 415c9d7b552b42c384b444a20491f65c
I20250901 01:52:33.467446 1052 raft_consensus.cc:2466] T 5f5396045979450193d461ba1334a5c9 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c282f779c1477d920f24dc0dc16628 in term 1.
I20250901 01:52:33.467713 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 6ba88af177874544b764cc32420e7ac9 with delete_type TABLET_DATA_DELETED (replaced by tablet 415c9d7b552b42c384b444a20491f65c) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:33.467679 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b51a069dc3d1477c85091d4025edbada"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.467994 859 leader_election.cc:304] T 5f5396045979450193d461ba1334a5c9 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:33.468425 2357 raft_consensus.cc:2802] T 5f5396045979450193d461ba1334a5c9 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:33.468402 1486 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_583e53c2e86544e9be68eb27ab6068a3" } modify_external_catalogs: true
I20250901 01:52:33.468572 2357 raft_consensus.cc:695] T 5f5396045979450193d461ba1334a5c9 P 00c282f779c1477d920f24dc0dc16628 [term 1 LEADER]: Becoming Leader. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Running, Role: LEADER
I20250901 01:52:33.468626 1486 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_583e53c2e86544e9be68eb27ab6068a3" } modify_external_catalogs: true
I20250901 01:52:33.468843 2357 consensus_queue.cc:237] T 5f5396045979450193d461ba1334a5c9 P 00c282f779c1477d920f24dc0dc16628 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.469591 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 6ba88af177874544b764cc32420e7ac9 with delete_type TABLET_DATA_DELETED (replaced by tablet 415c9d7b552b42c384b444a20491f65c) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:33.472349 1488 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: default.Table_a4f42e5055e6416caf95b93122822c6f [id=6a01b6c6977546be9c6128ca5f200a92] alter complete (version 1)
W20250901 01:52:33.475718 992 consensus_peers.cc:489] T eb19e3a424324a24824f6caaf5848952 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:33.476212 2113 ts_tablet_manager.cc:1918] T b85d64e02cc646e380581ac9f59dee9e P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:33.476317 2113 log.cc:1199] T b85d64e02cc646e380581ac9f59dee9e P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/b85d64e02cc646e380581ac9f59dee9e
I20250901 01:52:33.476639 2113 ts_tablet_manager.cc:1939] T b85d64e02cc646e380581ac9f59dee9e P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:33.477421 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 256) took 5338 ms. Trace:
I20250901 01:52:33.477500 2113 rpcz_store.cc:276] 0901 01:52:28.138832 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.141537 (+ 2705us) service_pool.cc:225] Handling call
0901 01:52:33.441781 (+5300244us) ts_tablet_manager.cc:1145] Deleting tablet b85d64e02cc646e380581ac9f59dee9e
0901 01:52:33.441788 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:33.470418 (+ 28630us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:33.476092 (+ 5674us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:33.477403 (+ 1311us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5300148}
I20250901 01:52:33.477952 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 257) took 5339 ms. Trace:
I20250901 01:52:33.478018 2113 rpcz_store.cc:276] 0901 01:52:28.138918 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.141643 (+ 2725us) service_pool.cc:225] Handling call
0901 01:52:33.477833 (+5336190us) ts_tablet_manager.cc:1145] Deleting tablet 088ef4861dc14f81ae470c3aaee5b4e1
0901 01:52:33.477842 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:33.477936 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5336107}
I20250901 01:52:33.478268 2113 tablet_replica.cc:331] T e2aa32450a584395ae07b49f76a7b26d P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:33.478789 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet b85d64e02cc646e380581ac9f59dee9e (table default.Table_7be95f26c04746828f38a19ddb67afc0 [id=2ec1e340c6924cefa540916968475e1f]) successfully deleted
W20250901 01:52:33.479156 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 088ef4861dc14f81ae470c3aaee5b4e1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 088ef4861dc14f81ae470c3aaee5b4e1 already in progress: creating tablet
I20250901 01:52:33.479943 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_40498dd46dc04f37ba90556ee5b34002"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.480911 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ab7ca2360d2d4fd6acc4015a629a47fa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.483741 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_899edccae7a74d63868b16296bd6f7aa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.484177 1485 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_41032b3c7d7e484a9e10e0a9edf47dd6" } modify_external_catalogs: true
I20250901 01:52:33.484359 1485 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_41032b3c7d7e484a9e10e0a9edf47dd6" } modify_external_catalogs: true
I20250901 01:52:33.484586 1482 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_ce16a1b94f714c2699ee6cd916bb0215" } modify_external_catalogs: true
I20250901 01:52:33.484797 1482 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_ce16a1b94f714c2699ee6cd916bb0215" } modify_external_catalogs: true
I20250901 01:52:33.485412 1490 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet b13afacd37474781877b9f2cd20b313a
W20250901 01:52:33.485673 1490 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet b13afacd37474781877b9f2cd20b313a: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:33.486153 1490 catalog_manager.cc:6652] ReplaceTablet: tablet b13afacd37474781877b9f2cd20b313a deleted and replaced by tablet bc826f7cb9d24fc59640d2ff794bb50a
I20250901 01:52:33.486518 1490 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_3f9422b08b0349b9a5466218f0fe1b06" } new_table_name: "default.Table_cca52da43da6439eb88ea9715e4a0d50" modify_external_catalogs: true
I20250901 01:52:33.491286 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet b13afacd37474781877b9f2cd20b313a with delete_type TABLET_DATA_DELETED (replaced by tablet bc826f7cb9d24fc59640d2ff794bb50a) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:33.491948 2113 raft_consensus.cc:2241] T e2aa32450a584395ae07b49f76a7b26d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:33.492113 2113 raft_consensus.cc:2270] T e2aa32450a584395ae07b49f76a7b26d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:33.493273 893 tablet_service.cc:1515] Processing DeleteTablet for tablet b13afacd37474781877b9f2cd20b313a with delete_type TABLET_DATA_DELETED (replaced by tablet bc826f7cb9d24fc59640d2ff794bb50a) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:33.494478 1484 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 92f0ad59fbbb435ea872c669a52f854a
W20250901 01:52:33.494731 1484 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 92f0ad59fbbb435ea872c669a52f854a: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:33.495489 1484 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_aa1bfa4e8b59486f96c587bc9cf84d73" } new_table_name: "default.Table_297737979dae4534a0eef6ca9e87e98d" modify_external_catalogs: true
W20250901 01:52:33.497056 1489 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0c88c339e59a4c1da21756cba0645bdd has no consensus state
I20250901 01:52:33.497377 1489 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_8825dd28afbe464193eb1b668cc9326a" } new_table_name: "default.Table_fa623571fcac4341935b9cb46ef4818c" modify_external_catalogs: true
I20250901 01:52:33.499842 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 92f0ad59fbbb435ea872c669a52f854a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:33 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:33.500694 2113 ts_tablet_manager.cc:1905] T e2aa32450a584395ae07b49f76a7b26d P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:33.505918 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9ceaf024000e41498e53d9c7d8d4dfe9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.506954 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c3111fc9333b4b93b2aae4f663e3416a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.507767 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c9b3c42ab1d7429b965e7f142a6c19b2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:33.509649 1484 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b4278a013ae84082bc7a8497b04ac147 consensus state has no leader
I20250901 01:52:33.509686 1482 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet e4d8330a4e454d1b8d643e17a6996741
I20250901 01:52:33.510149 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a11c58e5c2de4a07b2b3de191052f8d5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.510246 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_114c7bb0a03149428b4ad0c7522db710"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.511122 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7dbbc2a4bd48407c805ab2a508529ed0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.511179 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ab8e244d2522493c8d937d474e32d50d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.511999 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_061c160e47db4a8384c82e65f3f33831"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.512162 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_6e4c122ad1a0427d98cf3e69a50ca972"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:33.513034 1489 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e9f1f892658b4736859ff507536920a1 consensus state has no leader
E20250901 01:52:33.515501 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 0c88c339e59a4c1da21756cba0645bdd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0927e3ea788d43bda5afeda483d87176'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.515666 1696 catalog_manager.cc:6205] Error selecting replicas for tablet a656adbf1b29431687e0989f0db3459c, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2d3d2227e6a24bd2a86f457e553e698b'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:33.515683 2113 ts_tablet_manager.cc:1918] T e2aa32450a584395ae07b49f76a7b26d P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
E20250901 01:52:33.515803 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 494b6488aa0d4355b4caa46004267465, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_3ef97ecacc3042099ec1cc5c3a7a0763'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.515975 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 0ee3f35392954817b350e5f48653be78, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_442e11b9211c4b209c68f653d7bfc313'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:33.515995 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 92f0ad59fbbb435ea872c669a52f854a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:33 UTC) from {username='slave'} at 127.0.0.1:34472
E20250901 01:52:33.516109 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae4844183a3c4a04a775e6fd878391a3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_aee74198b51d4f089538c4f2bf3ce4dc'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.516230 1696 catalog_manager.cc:6205] Error selecting replicas for tablet be8dfbc466564842a2fdb4953eee939b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_75ba3ba70ca24a94920e7bce96190ff2'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.516378 1696 catalog_manager.cc:6205] Error selecting replicas for tablet bc826f7cb9d24fc59640d2ff794bb50a, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_36f5f8854ed14d1ea48eb18865d00659'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.516487 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9745582cebfe47b58c21d0191ab4cb9b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77728af9eee44f29a1e2131d5b75b7ca'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.516608 1696 catalog_manager.cc:6205] Error selecting replicas for tablet e2ed32b3292141278af28026528e96c2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b5077f55951d48be92a67407ffe26a08'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.516736 1696 catalog_manager.cc:6205] Error selecting replicas for tablet eea6bfabe6cc4f83905c24f0f7a5dfc9, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_e0902dfbdd984f85b3fdac8d283731f7'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.516889 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9f62a9d97f514747b940b488907369f3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d53a456dbca64daf92aeb5fb4517b417'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.517009 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5a7fa68d29644ae8afd2711e39dde59a, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d24e3a18ef4843fdb7b74ce23d47cf9e'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:33.517089 2113 log.cc:1199] T e2aa32450a584395ae07b49f76a7b26d P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/e2aa32450a584395ae07b49f76a7b26d
E20250901 01:52:33.517127 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 415c9d7b552b42c384b444a20491f65c, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_97c44ea002094df8a6af3dccd0099c39'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.517238 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ff2bd017417c41b5b8225b8f00283e63, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_97526c9a6c9f4acbb6dedd78475571e0'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.517362 1696 catalog_manager.cc:6205] Error selecting replicas for tablet dcc60bb000f440f1aca9133928b5ce20, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2fbca8463d6546c9b8f5b018fc60251d'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:33.517427 2113 ts_tablet_manager.cc:1939] T e2aa32450a584395ae07b49f76a7b26d P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
E20250901 01:52:33.517459 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f8f923ddcd2740bcb7e521f4f57c4034, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_5e65c5f068b84995a2d7a76fa65dda0f'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.517589 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 0813cc68d4b0488ba60a523e4479971b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d1302673d68f424ebb54b220911ce385'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.517699 1696 catalog_manager.cc:6205] Error selecting replicas for tablet b1ba0e19dc0b462e80c537bf2d183f5b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_66b111271c15410aab23682f8f6081ac'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.517825 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 1429a6a7a20e4f7c995e9acbaa8d495c, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0b769136b0664331945b30cfe0482fc6'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:33.518798 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a40c6c07280c4790b5e5dd5e0f600a45"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.519368 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 173) took 5302 ms. Trace:
I20250901 01:52:33.519488 2118 rpcz_store.cc:276] 0901 01:52:28.215835 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.215892 (+ 57us) service_pool.cc:225] Handling call
0901 01:52:33.276005 (+5060113us) ts_tablet_manager.cc:1145] Deleting tablet 0cba5329ab31440999faec599d704633
0901 01:52:33.276021 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:33.315327 (+ 39306us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:33.431152 (+115825us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:33.518081 (+ 86929us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5059878}
I20250901 01:52:33.519922 2118 tablet_replica.cc:331] T 7f46e9ddadaf4ff986d8c991178daa0c P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:33.520154 2118 raft_consensus.cc:2241] T 7f46e9ddadaf4ff986d8c991178daa0c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:33.520179 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 0cba5329ab31440999faec599d704633 (table default.Table_a5c783cbdb694af69047b31d99ff8def [id=3debd2adcd05421ea96dc426f739f8bd]) successfully deleted
I20250901 01:52:33.520346 2118 raft_consensus.cc:2270] T 7f46e9ddadaf4ff986d8c991178daa0c P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:33.528051 1151 tablet_bootstrap.cc:492] T 61ddcf86c9f34726939cc116434f0822 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:33.528204 1151 ts_tablet_manager.cc:1397] T 61ddcf86c9f34726939cc116434f0822 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.130s user 0.000s sys 0.009s
I20250901 01:52:33.528760 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_82207580e9c94bd5a89bf57dd95e32bb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.528950 1151 raft_consensus.cc:357] T 61ddcf86c9f34726939cc116434f0822 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.529115 1151 raft_consensus.cc:383] T 61ddcf86c9f34726939cc116434f0822 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:33.529165 1151 raft_consensus.cc:738] T 61ddcf86c9f34726939cc116434f0822 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:33.529356 1151 consensus_queue.cc:260] T 61ddcf86c9f34726939cc116434f0822 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.529879 1151 ts_tablet_manager.cc:1428] T 61ddcf86c9f34726939cc116434f0822 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250901 01:52:33.530234 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8ad0b9c171734cb58620c4f785de591f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.530733 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 258) took 5391 ms. Trace:
I20250901 01:52:33.530843 2113 rpcz_store.cc:276] 0901 01:52:28.138982 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.141746 (+ 2764us) service_pool.cc:225] Handling call
0901 01:52:33.478220 (+5336474us) ts_tablet_manager.cc:1145] Deleting tablet e2aa32450a584395ae07b49f76a7b26d
0901 01:52:33.478232 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:33.508103 (+ 29871us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:33.515554 (+ 7451us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:33.530717 (+ 15163us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13468,"tablet-delete.queue_time_us":5336351}
I20250901 01:52:33.531309 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_886dd84709f84e27bdc3121e7bf36e90"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.532460 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_235df6bce60b4d34ab8195b2da900b23"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.533558 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d1909ec8dda74907a087131b2f56c7d1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.534740 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_46118bbb1e044ffa920dd32754933730"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.537134 2118 ts_tablet_manager.cc:1905] T 7f46e9ddadaf4ff986d8c991178daa0c P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:33.536391 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4b67ab85a0d54c03a1fec3f4c21251c2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.537156 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_45ed7c7e34044609a57168aa0eba5344"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.541385 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_234152ab19c54642bb497611681c4b00"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.542361 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ece4a9ff9b8943539a54b97d5e905f7f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.544430 2113 tablet_replica.cc:331] T e8e7d2a72f704f52a8f8d46911a8a9e4 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:33.544713 2113 raft_consensus.cc:2241] T e8e7d2a72f704f52a8f8d46911a8a9e4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:33.544907 2113 raft_consensus.cc:2270] T e8e7d2a72f704f52a8f8d46911a8a9e4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:33.545608 2113 ts_tablet_manager.cc:1905] T e8e7d2a72f704f52a8f8d46911a8a9e4 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:33.552011 2113 ts_tablet_manager.cc:1918] T e8e7d2a72f704f52a8f8d46911a8a9e4 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:33.552132 2113 log.cc:1199] T e8e7d2a72f704f52a8f8d46911a8a9e4 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/e8e7d2a72f704f52a8f8d46911a8a9e4
I20250901 01:52:33.552721 1151 tablet_bootstrap.cc:492] T 56f2c4d2e6ab4bb4bdd6d025843589b6 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:33.554147 1151 tablet_bootstrap.cc:654] T 56f2c4d2e6ab4bb4bdd6d025843589b6 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
E20250901 01:52:33.555687 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 8e10a01e77d14cf28c2c5e0205e595b9, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_83cf2d86a1284544ba2e7aa9ebb66bb7'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.558369 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 05a547a8866149cb95d23e960a737237, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_583e53c2e86544e9be68eb27ab6068a3'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.558535 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5f0e054f62de4cf6bc6f69ad6664b192, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ef6832d0845044ed9f165ca3541b4c3f'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.558678 1696 catalog_manager.cc:6205] Error selecting replicas for tablet fd2ce82a4ee24d0faa44980a3967e187, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_63d0f2ead9d946d39ce40162ebd769d6'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.558789 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae9a4ad4fdfb4771ba23a91f7007447e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_5e84ae9a9c2a4084a1f9c5ffb4594de2'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.558945 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 086aa948f93644da8e5835b0fac87ddd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b4fbc34da806475c912052d0fab3fc53'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:33.558889 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_924ec6c621ad4f5c8fb78f9ec3863682"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.559545 2113 ts_tablet_manager.cc:1939] T e8e7d2a72f704f52a8f8d46911a8a9e4 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:33.559998 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_30764f19a75d4dfc81c172d2a4135f84"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:33.559062 1696 catalog_manager.cc:6205] Error selecting replicas for tablet d602c8986c75492ead9936ac04e2622d, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_82d91a1ba2eb4265896824028dec8d3a'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.560950 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 93aa22afc0ad4d7fa24807ea3c5d5c21, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0f4b4094d2974b7591875375eda6bed0'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.561076 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9074543dea9949e7a085717b40913348, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_53e30205c5fd488eb3b48c685a159d04'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.561194 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 3a62034e688348ed9e2ac9de009ed3a2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_409dad98d7eb455a8e16edfb4acedb34'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.561291 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 18498c9cb09141d08c1c5c0854b63ebd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77f4810bc83c4d948531da2a45a3792e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.561404 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f5e3316adeae47cd848833dd84b972bb, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7570c3012f8a4ae7bcfc7346eadac92a'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.561513 1696 catalog_manager.cc:6205] Error selecting replicas for tablet a6b539e160194da3a849d9e2f1ec9859, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_03fb48eaab9b42858ab7cefb722e43a8'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.561625 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 6ce9521b925b4f97bb4a810ce77e147e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_9e04f976a93f4d30baccf90fa3622dfe'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.561717 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2f61edb50d7a42198bfd70639c7bf4de, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_a5c783cbdb694af69047b31d99ff8def'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:33.562026 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 259) took 5422 ms. Trace:
I20250901 01:52:33.562614 2113 rpcz_store.cc:276] 0901 01:52:28.139048 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.141889 (+ 2841us) service_pool.cc:225] Handling call
0901 01:52:33.531122 (+5389233us) ts_tablet_manager.cc:1145] Deleting tablet e8e7d2a72f704f52a8f8d46911a8a9e4
0901 01:52:33.544333 (+ 13211us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:33.549201 (+ 4868us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:33.551925 (+ 2724us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:33.562009 (+ 10084us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5389103}
I20250901 01:52:33.563122 2779 tablet.cc:1722] T 24ddc13d1bb04db4a7850acbff091577 P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250901 01:52:33.563313 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d08957060f6847148f2d7e739e80bd89"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.563412 2113 tablet_replica.cc:331] T 75345ea2a3dc40fc8c9f8090d8839d2d P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:33.563622 2113 raft_consensus.cc:2241] T 75345ea2a3dc40fc8c9f8090d8839d2d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:33.563758 2113 raft_consensus.cc:2270] T 75345ea2a3dc40fc8c9f8090d8839d2d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:33.564039 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet e8e7d2a72f704f52a8f8d46911a8a9e4 (table default.Table_0ece0b3619db43da9d3bb2b0a72a7ef9 [id=623e11cd89c94142beb5c1a46e1adcd9]) successfully deleted
I20250901 01:52:33.564281 2778 tablet.cc:1722] T 24ddc13d1bb04db4a7850acbff091577 P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250901 01:52:33.564479 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2d961e49c904479ba5b0bc46d2262940"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.564271 2113 ts_tablet_manager.cc:1905] T 75345ea2a3dc40fc8c9f8090d8839d2d P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:33.565407 1151 tablet_bootstrap.cc:492] T 56f2c4d2e6ab4bb4bdd6d025843589b6 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:33.565357 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1ea734ef4f84425192143100a1168c80"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.565682 1151 ts_tablet_manager.cc:1397] T 56f2c4d2e6ab4bb4bdd6d025843589b6 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.003s
I20250901 01:52:33.566138 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_63fae338a04b4c039b759c02431005a4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.566581 1151 raft_consensus.cc:357] T 56f2c4d2e6ab4bb4bdd6d025843589b6 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.566800 1151 raft_consensus.cc:383] T 56f2c4d2e6ab4bb4bdd6d025843589b6 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:33.566900 1151 raft_consensus.cc:738] T 56f2c4d2e6ab4bb4bdd6d025843589b6 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:33.567159 1151 consensus_queue.cc:260] T 56f2c4d2e6ab4bb4bdd6d025843589b6 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.567811 1151 ts_tablet_manager.cc:1428] T 56f2c4d2e6ab4bb4bdd6d025843589b6 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250901 01:52:33.572311 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f76612c49d454c93a8a024fd2b82958d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.572733 2118 ts_tablet_manager.cc:1918] T 7f46e9ddadaf4ff986d8c991178daa0c P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:33.572849 2118 log.cc:1199] T 7f46e9ddadaf4ff986d8c991178daa0c P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/7f46e9ddadaf4ff986d8c991178daa0c
I20250901 01:52:33.573132 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_32e6a713f21249798c1b839df4bef326"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.573884 1151 tablet_bootstrap.cc:492] T 8b9d79a25452410891c09fd3f7c4be6d P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:33.574111 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d43a575716dd475991133afefc206b8d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.575357 1151 tablet_bootstrap.cc:654] T 8b9d79a25452410891c09fd3f7c4be6d P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:33.575439 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8e007d6946254e7bbe9a327a84d6b7fa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.582091 2118 ts_tablet_manager.cc:1939] T 7f46e9ddadaf4ff986d8c991178daa0c P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:33.583009 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c693b15958a74980a57bfe9677d4ca3c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.584290 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_995f861dbf5a4becba7ac40fea14ead6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:33.584712 992 consensus_peers.cc:489] T 5d794471dbed4477b9d52fd411fa4d10 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250901 01:52:33.585247 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8f616c8b2c21427f898ecd1a5b443340"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.586287 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_890cde4d14024a71a2057fa7f7a36381"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.587235 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_240a8b768bbc424dbf8c7eaff8aca427"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.587528 1151 tablet_bootstrap.cc:492] T 8b9d79a25452410891c09fd3f7c4be6d P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:33.587678 1151 ts_tablet_manager.cc:1397] T 8b9d79a25452410891c09fd3f7c4be6d P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.001s
I20250901 01:52:33.588454 1151 raft_consensus.cc:357] T 8b9d79a25452410891c09fd3f7c4be6d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.588666 1151 raft_consensus.cc:383] T 8b9d79a25452410891c09fd3f7c4be6d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:33.588779 1151 raft_consensus.cc:738] T 8b9d79a25452410891c09fd3f7c4be6d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:33.589120 1151 consensus_queue.cc:260] T 8b9d79a25452410891c09fd3f7c4be6d P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.590627 1151 ts_tablet_manager.cc:1428] T 8b9d79a25452410891c09fd3f7c4be6d P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250901 01:52:33.591167 1151 tablet_bootstrap.cc:492] T d000ba05e12843b4925e622ab55471c0 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:33.594319 1151 tablet_bootstrap.cc:654] T d000ba05e12843b4925e622ab55471c0 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:33.597298 2113 ts_tablet_manager.cc:1918] T 75345ea2a3dc40fc8c9f8090d8839d2d P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:33.597484 2113 log.cc:1199] T 75345ea2a3dc40fc8c9f8090d8839d2d P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/75345ea2a3dc40fc8c9f8090d8839d2d
I20250901 01:52:33.598008 2113 ts_tablet_manager.cc:1939] T 75345ea2a3dc40fc8c9f8090d8839d2d P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:33.597781 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_315d4e0b52084c73ba81a386190577fc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.598970 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 260) took 5459 ms. Trace:
I20250901 01:52:33.599095 2113 rpcz_store.cc:276] 0901 01:52:28.139131 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.142039 (+ 2908us) service_pool.cc:225] Handling call
0901 01:52:33.563368 (+5421329us) ts_tablet_manager.cc:1145] Deleting tablet 75345ea2a3dc40fc8c9f8090d8839d2d
0901 01:52:33.563378 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:33.584876 (+ 21498us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:33.597195 (+ 12319us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:33.598952 (+ 1757us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5421233}
I20250901 01:52:33.599748 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5a5569d8275046e1b76811743611da1c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.600240 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 75345ea2a3dc40fc8c9f8090d8839d2d (table default.Table_9734ae6c72e240c698b66ea83c536249 [id=f6d2701cc218463a9d1847e617bfec46]) successfully deleted
I20250901 01:52:33.600339 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 261) took 5461 ms. Trace:
I20250901 01:52:33.600455 2113 rpcz_store.cc:276] 0901 01:52:28.139197 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.142147 (+ 2950us) service_pool.cc:225] Handling call
0901 01:52:33.600247 (+5458100us) ts_tablet_manager.cc:1145] Deleting tablet 5a8c21aee9f84216b6f90538a5bf4c17
0901 01:52:33.600256 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:33.600324 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5457981}
I20250901 01:52:33.600669 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a36bec0027324bc997ed50f57a5c0ae0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:33.601022 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 5a8c21aee9f84216b6f90538a5bf4c17 because the tablet was not found. No further retry: Not found: Tablet not found: 5a8c21aee9f84216b6f90538a5bf4c17
I20250901 01:52:33.601085 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 174) took 5382 ms. Trace:
I20250901 01:52:33.601195 2118 rpcz_store.cc:276] 0901 01:52:28.218629 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.218698 (+ 69us) service_pool.cc:225] Handling call
0901 01:52:33.519856 (+5301158us) ts_tablet_manager.cc:1145] Deleting tablet 7f46e9ddadaf4ff986d8c991178daa0c
0901 01:52:33.519872 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:33.563355 (+ 43483us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:33.572640 (+ 9285us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:33.601057 (+ 28417us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3328,"tablet-delete.queue_time_us":5300915}
I20250901 01:52:33.601415 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 262) took 5462 ms. Trace:
I20250901 01:52:33.601483 2113 rpcz_store.cc:276] 0901 01:52:28.139260 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.142298 (+ 3038us) service_pool.cc:225] Handling call
0901 01:52:33.601335 (+5459037us) ts_tablet_manager.cc:1145] Deleting tablet b541a6e14f114ab29a864cca11806316
0901 01:52:33.601345 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:33.601401 (+ 56us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5458946}
I20250901 01:52:33.601747 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 263) took 5462 ms. Trace:
I20250901 01:52:33.601822 2113 rpcz_store.cc:276] 0901 01:52:28.139321 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.142404 (+ 3083us) service_pool.cc:225] Handling call
0901 01:52:33.601674 (+5459270us) ts_tablet_manager.cc:1145] Deleting tablet b633b461dbf34e13bc6e1cc08e6d20c1
0901 01:52:33.601684 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:33.601735 (+ 51us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5459181}
I20250901 01:52:33.602065 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 264) took 5462 ms. Trace:
I20250901 01:52:33.602018 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ea2c31e122d1436d95084c348d64fb3d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.602115 2113 rpcz_store.cc:276] 0901 01:52:28.139413 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.142511 (+ 3098us) service_pool.cc:225] Handling call
0901 01:52:33.601986 (+5459475us) ts_tablet_manager.cc:1145] Deleting tablet 50472a7aa2f04e0fbd65ac6c86c34b3e
0901 01:52:33.601992 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:33.602055 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5459388}
I20250901 01:52:33.602315 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 265) took 5462 ms. Trace:
I20250901 01:52:33.603386 1486 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 05a547a8866149cb95d23e960a737237
I20250901 01:52:33.603396 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7f65be9a22c04dc58e44199211542c97"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.602365 2113 rpcz_store.cc:276] 0901 01:52:28.139479 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.142615 (+ 3136us) service_pool.cc:225] Handling call
0901 01:52:33.602260 (+5459645us) ts_tablet_manager.cc:1145] Deleting tablet 20f7bfab18134e079e91c52c89468cc3
0901 01:52:33.602267 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:33.602305 (+ 38us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5459520}
I20250901 01:52:33.604008 1485 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 8151bad707c04ac687c5bddfea573e3f
I20250901 01:52:33.604141 2113 tablet_replica.cc:331] T 56f2c4d2e6ab4bb4bdd6d025843589b6 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
W20250901 01:52:33.604221 1485 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 8151bad707c04ac687c5bddfea573e3f: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:33.604314 2113 raft_consensus.cc:2241] T 56f2c4d2e6ab4bb4bdd6d025843589b6 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:33.604456 2113 raft_consensus.cc:2270] T 56f2c4d2e6ab4bb4bdd6d025843589b6 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:33.605700 2113 ts_tablet_manager.cc:1905] T 56f2c4d2e6ab4bb4bdd6d025843589b6 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:33.605803 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet b541a6e14f114ab29a864cca11806316 because the tablet was not found. No further retry: Not found: Tablet not found: b541a6e14f114ab29a864cca11806316
W20250901 01:52:33.606045 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet b633b461dbf34e13bc6e1cc08e6d20c1 because the tablet was not found. No further retry: Not found: Tablet not found: b633b461dbf34e13bc6e1cc08e6d20c1
W20250901 01:52:33.606297 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 50472a7aa2f04e0fbd65ac6c86c34b3e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 50472a7aa2f04e0fbd65ac6c86c34b3e already in progress: creating tablet
W20250901 01:52:33.606523 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 20f7bfab18134e079e91c52c89468cc3 because the tablet was not found. No further retry: Not found: Tablet not found: 20f7bfab18134e079e91c52c89468cc3
I20250901 01:52:33.607049 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 8151bad707c04ac687c5bddfea573e3f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:33 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:33.608386 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 7f46e9ddadaf4ff986d8c991178daa0c with delete_type TABLET_DATA_DELETED (replaced by tablet 7d7346ef9a2e4a35874605256ce41c6a) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:33.608640 901 tablet_service.cc:1515] Processing DeleteTablet for tablet 0cba5329ab31440999faec599d704633 with delete_type TABLET_DATA_DELETED (replaced by tablet 2f61edb50d7a42198bfd70639c7bf4de) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:33.608705 2118 tablet_replica.cc:331] T c422881b40f54971a68530abfbd1945b P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:33.608814 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 9fbd44641d9e43c3b592e388e4adf350 with delete_type TABLET_DATA_DELETED (replaced by tablet dcc60bb000f440f1aca9133928b5ce20) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:33.608995 2118 raft_consensus.cc:2241] T c422881b40f54971a68530abfbd1945b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:33.609026 891 tablet_service.cc:1515] Processing DeleteTablet for tablet cb7d7e1796c0456f8de98fb812b9e374 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:27 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:33.609153 2118 raft_consensus.cc:2270] T c422881b40f54971a68530abfbd1945b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:33.609210 898 tablet_service.cc:1515] Processing DeleteTablet for tablet 3697821cd29446bb908b020599df0db9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:27 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:33.608690 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 8151bad707c04ac687c5bddfea573e3f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:33 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:33.609690 2118 ts_tablet_manager.cc:1905] T c422881b40f54971a68530abfbd1945b P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:33.609943 1151 tablet_bootstrap.cc:492] T d000ba05e12843b4925e622ab55471c0 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:33.610056 1151 ts_tablet_manager.cc:1397] T d000ba05e12843b4925e622ab55471c0 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.000s
I20250901 01:52:33.610944 1151 raft_consensus.cc:357] T d000ba05e12843b4925e622ab55471c0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.611163 1151 raft_consensus.cc:383] T d000ba05e12843b4925e622ab55471c0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:33.611258 1151 raft_consensus.cc:738] T d000ba05e12843b4925e622ab55471c0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:33.611513 1151 consensus_queue.cc:260] T d000ba05e12843b4925e622ab55471c0 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.612125 1151 ts_tablet_manager.cc:1428] T d000ba05e12843b4925e622ab55471c0 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250901 01:52:33.612475 1151 tablet_bootstrap.cc:492] T 36e10328c34242ebb4dbd8f294162696 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:33.613191 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 7442c6ff699141e1aff208ebdc9d192a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:26 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:33.613857 1151 tablet_bootstrap.cc:654] T 36e10328c34242ebb4dbd8f294162696 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:33.608819 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 85322a864d154f299f2d670fdd2a7bfb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:27 UTC) from {username='slave'} at 127.0.0.1:34472
W20250901 01:52:33.617328 1488 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:54603 (request call id 187) took 910 ms (client timeout 1000 ms). Trace:
I20250901 01:52:33.619261 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 641a83a3e50d451189c33c7b7b4ca070 with delete_type TABLET_DATA_DELETED (replaced by tablet 5f0e054f62de4cf6bc6f69ad6664b192) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:33.619576 891 tablet_service.cc:1515] Processing DeleteTablet for tablet f23df6f253a943fa8e146bb8c811ef51 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:27 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:33.619653 885 tablet_service.cc:1515] Processing DeleteTablet for tablet d10c193452954c198bba686169b8025b with delete_type TABLET_DATA_DELETED (replaced by tablet a6b539e160194da3a849d9e2f1ec9859) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:33.619820 888 tablet_service.cc:1515] Processing DeleteTablet for tablet ff013ea889d44cba8797834e4611b489 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:26 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:33.619850 898 tablet_service.cc:1515] Processing DeleteTablet for tablet b85d64e02cc646e380581ac9f59dee9e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:33.619946 2113 ts_tablet_manager.cc:1918] T 56f2c4d2e6ab4bb4bdd6d025843589b6 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:33.620059 2113 log.cc:1199] T 56f2c4d2e6ab4bb4bdd6d025843589b6 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/56f2c4d2e6ab4bb4bdd6d025843589b6
I20250901 01:52:33.620087 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 75345ea2a3dc40fc8c9f8090d8839d2d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:34472
W20250901 01:52:33.617559 1488 rpcz_store.cc:269] 0901 01:52:32.707198 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:32.720480 (+ 13282us) service_pool.cc:225] Handling call
0901 01:52:33.617277 (+896797us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0901 01:52:33.454310 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:33.454482 (+ 172us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:33.454490 (+ 8us) write_op.cc:435] Acquired schema lock
0901 01:52:33.454493 (+ 3us) tablet.cc:660] Decoding operations
0901 01:52:33.454625 (+ 132us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:33.454641 (+ 16us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:33.454649 (+ 8us) tablet.cc:683] Acquiring locks for 1 operations
0901 01:52:33.454725 (+ 76us) tablet.cc:699] Row locks acquired
0901 01:52:33.454729 (+ 4us) write_op.cc:260] PREPARE: finished
0901 01:52:33.454770 (+ 41us) write_op.cc:270] Start()
0901 01:52:33.454812 (+ 42us) write_op.cc:276] Timestamp: P: 1756691553454720 usec, L: 0
0901 01:52:33.454818 (+ 6us) op_driver.cc:348] REPLICATION: starting
0901 01:52:33.455027 (+ 209us) log.cc:844] Serialized 356 byte log entry
0901 01:52:33.471525 (+ 16498us) op_driver.cc:464] REPLICATION: finished
0901 01:52:33.471658 (+ 133us) write_op.cc:301] APPLY: starting
0901 01:52:33.471677 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:33.471723 (+ 46us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:33.471775 (+ 52us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:33.471834 (+ 59us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:33.471875 (+ 41us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0901 01:52:33.471884 (+ 9us) write_op.cc:312] APPLY: finished
0901 01:52:33.471966 (+ 82us) log.cc:844] Serialized 23 byte log entry
0901 01:52:33.472025 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:33.472041 (+ 16us) write_op.cc:454] Released schema lock
0901 01:52:33.472048 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0901 01:52:33.472126 (+ 78us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:33.472142 (+ 16us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":14766,"reported_tablets":124,"child_traces":[["op",{"apply.queue_time_us":73,"apply.run_cpu_time_us":463,"apply.run_wall_time_us":463,"num_ops":1,"prepare.queue_time_us":57,"prepare.run_cpu_time_us":832,"prepare.run_wall_time_us":832,"raft.queue_time_us":38,"raft.run_cpu_time_us":368,"raft.run_wall_time_us":367,"replication_time_us":16660}]]}
I20250901 01:52:33.620313 2113 ts_tablet_manager.cc:1939] T 56f2c4d2e6ab4bb4bdd6d025843589b6 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:33.620608 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 088ef4861dc14f81ae470c3aaee5b4e1 with delete_type TABLET_DATA_DELETED (replaced by tablet 6ce9521b925b4f97bb4a810ce77e147e) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:33.621047 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 266) took 5481 ms. Trace:
I20250901 01:52:33.621137 2113 rpcz_store.cc:276] 0901 01:52:28.139542 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.142769 (+ 3227us) service_pool.cc:225] Handling call
0901 01:52:33.604098 (+5461329us) ts_tablet_manager.cc:1145] Deleting tablet 56f2c4d2e6ab4bb4bdd6d025843589b6
0901 01:52:33.604106 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:33.614789 (+ 10683us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:33.619856 (+ 5067us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:33.621030 (+ 1174us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5461210}
I20250901 01:52:33.621405 2113 tablet_replica.cc:331] T 879fa66e55544118aafe504587393c66 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:33.621579 2113 raft_consensus.cc:2241] T 879fa66e55544118aafe504587393c66 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:33.623232 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet f3962b471bdb49aea6e9fa95f1f9dfe2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:17 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:33.623575 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 008d23f236c348b6be9aee718f8698af with delete_type TABLET_DATA_DELETED (replaced by tablet b1ba0e19dc0b462e80c537bf2d183f5b) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:33.623915 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet b85d64e02cc646e380581ac9f59dee9e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:33.624627 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 088ef4861dc14f81ae470c3aaee5b4e1 with delete_type TABLET_DATA_DELETED (replaced by tablet 6ce9521b925b4f97bb4a810ce77e147e) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:33.625859 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet e2aa32450a584395ae07b49f76a7b26d with delete_type TABLET_DATA_DELETED (replaced by tablet 7304937048ca48a7bd153f3cdcfaeff0) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:33.626096 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 74b709cb5ec64006b2f12bead8217d80 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:33.626591 1037 tablet_service.cc:1515] Processing DeleteTablet for tablet e8e7d2a72f704f52a8f8d46911a8a9e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:33.626773 1035 tablet_service.cc:1515] Processing DeleteTablet for tablet 75345ea2a3dc40fc8c9f8090d8839d2d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:33.626927 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet cb7d7e1796c0456f8de98fb812b9e374 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:27 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:33.630829 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 56f2c4d2e6ab4bb4bdd6d025843589b6 (table default.Table_c0977985d6ee4b21a086dc892c860d6a [id=f2ff184777494b45a70f9bc4f66a448b]) successfully deleted
I20250901 01:52:33.632088 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 50472a7aa2f04e0fbd65ac6c86c34b3e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:20 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:33.632556 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet f23df6f253a943fa8e146bb8c811ef51 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:27 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:33.632951 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 3697821cd29446bb908b020599df0db9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:27 UTC) from {username='slave'} at 127.0.0.1:35946
W20250901 01:52:33.633025 1491 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:54603 (request call id 186) took 1872 ms (client timeout 1000 ms). Trace:
I20250901 01:52:33.633328 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 3dae00aa38204ee3ac77325eb334be42 with delete_type TABLET_DATA_DELETED (replaced by tablet 76ba642c16734cdf969dbd3130639fda) from {username='slave'} at 127.0.0.1:35946
W20250901 01:52:33.633131 1491 rpcz_store.cc:269] 0901 01:52:31.760492 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:31.776576 (+ 16084us) service_pool.cc:225] Handling call
0901 01:52:32.090620 (+314044us) spinlock_profiling.cc:243] Waited 80.6 ms on lock 0x616000040c54. stack: 00007f0d64ec57d6 00007f0d64ec5590 00007f0d63eb83ad 0000000000972d94 0000000000972d5d 00007f0d73c9014d 00007f0d73c90243 00007f0d7094f941 00007f0d784daa5d 00007f0d7865e6d8 00007f0d6f498c86 00007f0d6f4989b5 00007f0d6bc732ce 00007f0d6bc721ae 00007f0d6bc783e5 00007f0d6bc78d0f
0901 01:52:33.632957 (+1542337us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0901 01:52:33.202732 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:33.202847 (+ 115us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:33.202854 (+ 7us) write_op.cc:435] Acquired schema lock
0901 01:52:33.202859 (+ 5us) tablet.cc:660] Decoding operations
0901 01:52:33.202964 (+ 105us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:33.202981 (+ 17us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:33.202987 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0901 01:52:33.203043 (+ 56us) tablet.cc:699] Row locks acquired
0901 01:52:33.203046 (+ 3us) write_op.cc:260] PREPARE: finished
0901 01:52:33.203094 (+ 48us) write_op.cc:270] Start()
0901 01:52:33.203132 (+ 38us) write_op.cc:276] Timestamp: P: 1756691553203072 usec, L: 0
0901 01:52:33.203135 (+ 3us) op_driver.cc:348] REPLICATION: starting
0901 01:52:33.203359 (+ 224us) log.cc:844] Serialized 356 byte log entry
0901 01:52:33.224384 (+ 21025us) op_driver.cc:464] REPLICATION: finished
0901 01:52:33.224513 (+ 129us) write_op.cc:301] APPLY: starting
0901 01:52:33.224527 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:33.224571 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:33.224613 (+ 42us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:33.224665 (+ 52us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:33.224700 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0901 01:52:33.224708 (+ 8us) write_op.cc:312] APPLY: finished
0901 01:52:33.224786 (+ 78us) log.cc:844] Serialized 23 byte log entry
0901 01:52:33.224862 (+ 76us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:33.224877 (+ 15us) write_op.cc:454] Released schema lock
0901 01:52:33.224883 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0901 01:52:33.224954 (+ 71us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:33.224960 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":11246,"reported_tablets":131,"spinlock_wait_cycles":185367424,"child_traces":[["op",{"apply.queue_time_us":54,"apply.run_cpu_time_us":429,"apply.run_wall_time_us":428,"num_ops":1,"prepare.queue_time_us":1159,"prepare.run_cpu_time_us":731,"prepare.run_wall_time_us":732,"raft.queue_time_us":29,"raft.run_cpu_time_us":236,"raft.run_wall_time_us":235,"replication_time_us":21225}]]}
I20250901 01:52:33.634199 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 85322a864d154f299f2d670fdd2a7bfb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:27 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:33.634573 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 42c1156e8dd84c6fa1396b46de13eb6f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:26 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:33.635064 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 061b760d8e6647c6ab0b5d538840a230 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:26 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:33.635423 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 641a83a3e50d451189c33c7b7b4ca070 with delete_type TABLET_DATA_DELETED (replaced by tablet 5f0e054f62de4cf6bc6f69ad6664b192) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:33.635939 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet ff013ea889d44cba8797834e4611b489 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:26 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:33.636328 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 435b1cca5a7d4d49959a310cb250b2a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:26 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:33.636765 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet d10c193452954c198bba686169b8025b with delete_type TABLET_DATA_DELETED (replaced by tablet a6b539e160194da3a849d9e2f1ec9859) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:33.638527 2781 raft_consensus.cc:491] T f98d1d8d660443cb8066a547e5d9b77f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:33.638717 2781 raft_consensus.cc:513] T f98d1d8d660443cb8066a547e5d9b77f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.639465 2781 leader_election.cc:290] T f98d1d8d660443cb8066a547e5d9b77f P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:33.640182 917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f98d1d8d660443cb8066a547e5d9b77f" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:33.640434 917 raft_consensus.cc:2466] T f98d1d8d660443cb8066a547e5d9b77f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:33.640825 992 leader_election.cc:304] T f98d1d8d660443cb8066a547e5d9b77f P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:33.641052 2613 raft_consensus.cc:2802] T f98d1d8d660443cb8066a547e5d9b77f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20250901 01:52:33.641062 992 leader_election.cc:336] T f98d1d8d660443cb8066a547e5d9b77f P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:33.641141 2613 raft_consensus.cc:491] T f98d1d8d660443cb8066a547e5d9b77f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:33.641199 2613 raft_consensus.cc:3058] T f98d1d8d660443cb8066a547e5d9b77f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
W20250901 01:52:33.642699 1483 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.3:51295 (request call id 187) took 979 ms (client timeout 1000 ms). Trace:
W20250901 01:52:33.642843 1483 rpcz_store.cc:269] 0901 01:52:32.663129 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:32.664940 (+ 1811us) service_pool.cc:225] Handling call
0901 01:52:33.642644 (+977704us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0901 01:52:33.108443 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:33.108594 (+ 151us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:33.108602 (+ 8us) write_op.cc:435] Acquired schema lock
0901 01:52:33.108604 (+ 2us) tablet.cc:660] Decoding operations
0901 01:52:33.108831 (+ 227us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:33.108846 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:33.108851 (+ 5us) tablet.cc:683] Acquiring locks for 5 operations
0901 01:52:33.109011 (+ 160us) tablet.cc:699] Row locks acquired
0901 01:52:33.109013 (+ 2us) write_op.cc:260] PREPARE: finished
0901 01:52:33.109052 (+ 39us) write_op.cc:270] Start()
0901 01:52:33.109089 (+ 37us) write_op.cc:276] Timestamp: P: 1756691553108992 usec, L: 0
0901 01:52:33.109112 (+ 23us) op_driver.cc:348] REPLICATION: starting
0901 01:52:33.109312 (+ 200us) log.cc:844] Serialized 2024 byte log entry
0901 01:52:33.113231 (+ 3919us) op_driver.cc:464] REPLICATION: finished
0901 01:52:33.113334 (+ 103us) write_op.cc:301] APPLY: starting
0901 01:52:33.113349 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:33.113449 (+ 100us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:33.113480 (+ 31us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:33.113634 (+ 154us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:33.113672 (+ 38us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=5
0901 01:52:33.113680 (+ 8us) write_op.cc:312] APPLY: finished
0901 01:52:33.113763 (+ 83us) log.cc:844] Serialized 47 byte log entry
0901 01:52:33.113834 (+ 71us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:33.113858 (+ 24us) write_op.cc:454] Released schema lock
0901 01:52:33.113865 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0901 01:52:33.113939 (+ 74us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:33.113943 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0901 01:52:33.161258 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:33.161390 (+ 132us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:33.161397 (+ 7us) write_op.cc:435] Acquired schema lock
0901 01:52:33.161400 (+ 3us) tablet.cc:660] Decoding operations
0901 01:52:33.161516 (+ 116us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:33.161530 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:33.161535 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0901 01:52:33.161648 (+ 113us) tablet.cc:699] Row locks acquired
0901 01:52:33.161651 (+ 3us) write_op.cc:260] PREPARE: finished
0901 01:52:33.161697 (+ 46us) write_op.cc:270] Start()
0901 01:52:33.161738 (+ 41us) write_op.cc:276] Timestamp: P: 1756691553161630 usec, L: 0
0901 01:52:33.161742 (+ 4us) op_driver.cc:348] REPLICATION: starting
0901 01:52:33.161958 (+ 216us) log.cc:844] Serialized 356 byte log entry
0901 01:52:33.171278 (+ 9320us) op_driver.cc:464] REPLICATION: finished
0901 01:52:33.171418 (+ 140us) write_op.cc:301] APPLY: starting
0901 01:52:33.171435 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:33.171475 (+ 40us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:33.171509 (+ 34us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:33.171578 (+ 69us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:33.171612 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0901 01:52:33.171620 (+ 8us) write_op.cc:312] APPLY: finished
0901 01:52:33.171701 (+ 81us) log.cc:844] Serialized 23 byte log entry
0901 01:52:33.171769 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:33.171790 (+ 21us) write_op.cc:454] Released schema lock
0901 01:52:33.171796 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0901 01:52:33.171866 (+ 70us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:33.171870 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0901 01:52:33.228981 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:33.229072 (+ 91us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:33.229078 (+ 6us) write_op.cc:435] Acquired schema lock
0901 01:52:33.229082 (+ 4us) tablet.cc:660] Decoding operations
0901 01:52:33.229159 (+ 77us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:33.229167 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:33.229172 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0901 01:52:33.229228 (+ 56us) tablet.cc:699] Row locks acquired
0901 01:52:33.229231 (+ 3us) write_op.cc:260] PREPARE: finished
0901 01:52:33.229259 (+ 28us) write_op.cc:270] Start()
0901 01:52:33.229283 (+ 24us) write_op.cc:276] Timestamp: P: 1756691553229235 usec, L: 0
0901 01:52:33.229287 (+ 4us) op_driver.cc:348] REPLICATION: starting
0901 01:52:33.229404 (+ 117us) log.cc:844] Serialized 356 byte log entry
0901 01:52:33.427660 (+198256us) op_driver.cc:464] REPLICATION: finished
0901 01:52:33.427776 (+ 116us) write_op.cc:301] APPLY: starting
0901 01:52:33.427795 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:33.427844 (+ 49us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:33.427869 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:33.435199 (+ 7330us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:33.435235 (+ 36us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0901 01:52:33.435245 (+ 10us) write_op.cc:312] APPLY: finished
0901 01:52:33.438112 (+ 2867us) log.cc:844] Serialized 23 byte log entry
0901 01:52:33.438185 (+ 73us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:33.438204 (+ 19us) write_op.cc:454] Released schema lock
0901 01:52:33.438212 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0901 01:52:33.438292 (+ 80us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:33.438296 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":3439,"reported_tablets":148,"child_traces":[["op",{"apply.queue_time_us":56,"apply.run_cpu_time_us":596,"apply.run_wall_time_us":595,"num_ops":5,"prepare.queue_time_us":44,"prepare.run_cpu_time_us":988,"prepare.run_wall_time_us":993,"raft.queue_time_us":36,"raft.run_cpu_time_us":404,"raft.run_wall_time_us":405,"replication_time_us":4090}],["op",{"apply.queue_time_us":88,"apply.run_cpu_time_us":442,"apply.run_wall_time_us":442,"num_ops":1,"prepare.queue_time_us":5533,"prepare.run_cpu_time_us":810,"prepare.run_wall_time_us":810,"raft.queue_time_us":77,"raft.run_cpu_time_us":252,"raft.run_wall_time_us":252,"replication_time_us":9505}],["op",{"apply.queue_time_us":49,"apply.run_cpu_time_us":557,"apply.run_wall_time_us":10508,"num_ops":1,"prepare.queue_time_us":288,"prepare.run_cpu_time_us":484,"prepare.run_wall_time_us":484,"replication_time_us":198331}]]}
I20250901 01:52:33.644172 2778 tablet.cc:1722] T 6c0abbfc8d904dce989ce51961e4237d P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250901 01:52:33.644347 1484 master_service.cc:948] ReplaceTablet: received request to replace tablet 7b16ca1daf394a6da65a4af8da82ec33 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:33.644672 2613 raft_consensus.cc:513] T f98d1d8d660443cb8066a547e5d9b77f P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.645243 2613 leader_election.cc:290] T f98d1d8d660443cb8066a547e5d9b77f P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:33.645629 2753 raft_consensus.cc:491] T b2b172757e50404aae78e818921a7d9d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:33.645773 2753 raft_consensus.cc:513] T b2b172757e50404aae78e818921a7d9d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.646310 2753 leader_election.cc:290] T b2b172757e50404aae78e818921a7d9d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:33.646368 1492 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.3:51295 (request call id 186) took 1998 ms (client timeout 1000 ms). Trace:
I20250901 01:52:33.646817 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2b172757e50404aae78e818921a7d9d" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:33.646950 917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f98d1d8d660443cb8066a547e5d9b77f" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:33.647130 917 raft_consensus.cc:3058] T f98d1d8d660443cb8066a547e5d9b77f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
W20250901 01:52:33.647581 859 leader_election.cc:343] T b2b172757e50404aae78e818921a7d9d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:33.647799 2779 tablet.cc:1722] T 6c0abbfc8d904dce989ce51961e4237d P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250901 01:52:33.649140 858 leader_election.cc:336] T b2b172757e50404aae78e818921a7d9d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:33.649237 858 leader_election.cc:304] T b2b172757e50404aae78e818921a7d9d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:33.649431 2753 raft_consensus.cc:2747] T b2b172757e50404aae78e818921a7d9d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:33.649756 917 raft_consensus.cc:2466] T f98d1d8d660443cb8066a547e5d9b77f P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:33.650161 992 leader_election.cc:304] T f98d1d8d660443cb8066a547e5d9b77f P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:33.650775 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 56f2c4d2e6ab4bb4bdd6d025843589b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:33.651135 2613 raft_consensus.cc:2802] T f98d1d8d660443cb8066a547e5d9b77f P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:33.651243 2613 raft_consensus.cc:695] T f98d1d8d660443cb8066a547e5d9b77f P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:33.651449 2613 consensus_queue.cc:237] T f98d1d8d660443cb8066a547e5d9b77f P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:33.654029 992 leader_election.cc:336] T f98d1d8d660443cb8066a547e5d9b77f P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:33.659646 2113 raft_consensus.cc:2270] T 879fa66e55544118aafe504587393c66 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:33.660283 2113 ts_tablet_manager.cc:1905] T 879fa66e55544118aafe504587393c66 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
E20250901 01:52:33.660871 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 0c88c339e59a4c1da21756cba0645bdd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0927e3ea788d43bda5afeda483d87176'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.661036 1696 catalog_manager.cc:6205] Error selecting replicas for tablet a656adbf1b29431687e0989f0db3459c, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2d3d2227e6a24bd2a86f457e553e698b'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.661194 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 494b6488aa0d4355b4caa46004267465, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_3ef97ecacc3042099ec1cc5c3a7a0763'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.661338 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 0ee3f35392954817b350e5f48653be78, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_442e11b9211c4b209c68f653d7bfc313'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.661484 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae4844183a3c4a04a775e6fd878391a3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_aee74198b51d4f089538c4f2bf3ce4dc'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.661634 1696 catalog_manager.cc:6205] Error selecting replicas for tablet be8dfbc466564842a2fdb4953eee939b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_75ba3ba70ca24a94920e7bce96190ff2'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.661769 1696 catalog_manager.cc:6205] Error selecting replicas for tablet bc826f7cb9d24fc59640d2ff794bb50a, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_36f5f8854ed14d1ea48eb18865d00659'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.661898 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9745582cebfe47b58c21d0191ab4cb9b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77728af9eee44f29a1e2131d5b75b7ca'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.662040 1696 catalog_manager.cc:6205] Error selecting replicas for tablet e2ed32b3292141278af28026528e96c2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b5077f55951d48be92a67407ffe26a08'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.662151 1696 catalog_manager.cc:6205] Error selecting replicas for tablet eea6bfabe6cc4f83905c24f0f7a5dfc9, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_e0902dfbdd984f85b3fdac8d283731f7'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.662281 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9f62a9d97f514747b940b488907369f3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d53a456dbca64daf92aeb5fb4517b417'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.662369 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5a7fa68d29644ae8afd2711e39dde59a, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d24e3a18ef4843fdb7b74ce23d47cf9e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.662482 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 415c9d7b552b42c384b444a20491f65c, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_97c44ea002094df8a6af3dccd0099c39'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.662564 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ff2bd017417c41b5b8225b8f00283e63, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_97526c9a6c9f4acbb6dedd78475571e0'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.662678 1696 catalog_manager.cc:6205] Error selecting replicas for tablet dcc60bb000f440f1aca9133928b5ce20, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2fbca8463d6546c9b8f5b018fc60251d'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.662765 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f8f923ddcd2740bcb7e521f4f57c4034, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_5e65c5f068b84995a2d7a76fa65dda0f'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.662875 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 0813cc68d4b0488ba60a523e4479971b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d1302673d68f424ebb54b220911ce385'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.662954 1696 catalog_manager.cc:6205] Error selecting replicas for tablet b1ba0e19dc0b462e80c537bf2d183f5b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_66b111271c15410aab23682f8f6081ac'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.663053 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 1429a6a7a20e4f7c995e9acbaa8d495c, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0b769136b0664331945b30cfe0482fc6'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.663161 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 8e10a01e77d14cf28c2c5e0205e595b9, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_83cf2d86a1284544ba2e7aa9ebb66bb7'. Need at least 3 replicas, but only 2 tablet servers are available
W20250901 01:52:33.646497 1492 rpcz_store.cc:269] 0901 01:52:31.648277 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:31.662129 (+ 13852us) service_pool.cc:225] Handling call
0901 01:52:32.468487 (+806358us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:32.468493 (+ 6us) write_op.cc:454] Released schema lock
0901 01:52:33.646320 (+1177827us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0901 01:52:32.451006 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:32.451133 (+ 127us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:32.451141 (+ 8us) write_op.cc:435] Acquired schema lock
0901 01:52:32.451143 (+ 2us) tablet.cc:660] Decoding operations
0901 01:52:32.451336 (+ 193us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:32.451350 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:32.451355 (+ 5us) tablet.cc:683] Acquiring locks for 4 operations
0901 01:52:32.451464 (+ 109us) tablet.cc:699] Row locks acquired
0901 01:52:32.451467 (+ 3us) write_op.cc:260] PREPARE: finished
0901 01:52:32.451504 (+ 37us) write_op.cc:270] Start()
0901 01:52:32.451542 (+ 38us) write_op.cc:276] Timestamp: P: 1756691552451440 usec, L: 0
0901 01:52:32.451546 (+ 4us) op_driver.cc:348] REPLICATION: starting
0901 01:52:32.451772 (+ 226us) log.cc:844] Serialized 1650 byte log entry
0901 01:52:32.459052 (+ 7280us) op_driver.cc:464] REPLICATION: finished
0901 01:52:32.459155 (+ 103us) write_op.cc:301] APPLY: starting
0901 01:52:32.459170 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:32.459238 (+ 68us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:32.459269 (+ 31us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:32.459360 (+ 91us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:32.459395 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0901 01:52:32.459402 (+ 7us) write_op.cc:312] APPLY: finished
0901 01:52:32.459489 (+ 87us) log.cc:844] Serialized 41 byte log entry
0901 01:52:32.459557 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:32.459581 (+ 24us) write_op.cc:454] Released schema lock
0901 01:52:32.459588 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0901 01:52:32.657981 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:32.658066 (+ 85us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:32.658071 (+ 5us) write_op.cc:435] Acquired schema lock
0901 01:52:32.658073 (+ 2us) tablet.cc:660] Decoding operations
0901 01:52:32.658199 (+ 126us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:32.658210 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:32.658215 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0901 01:52:32.658272 (+ 57us) tablet.cc:699] Row locks acquired
0901 01:52:32.658275 (+ 3us) write_op.cc:260] PREPARE: finished
0901 01:52:32.658307 (+ 32us) write_op.cc:270] Start()
0901 01:52:32.658335 (+ 28us) write_op.cc:276] Timestamp: P: 1756691552658222 usec, L: 0
0901 01:52:32.658339 (+ 4us) op_driver.cc:348] REPLICATION: starting
0901 01:52:32.658510 (+ 171us) log.cc:844] Serialized 356 byte log entry
0901 01:52:32.669483 (+ 10973us) op_driver.cc:464] REPLICATION: finished
0901 01:52:32.669673 (+ 190us) write_op.cc:301] APPLY: starting
0901 01:52:32.669693 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:32.669747 (+ 54us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:32.669806 (+ 59us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:32.669885 (+ 79us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:32.669930 (+ 45us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0901 01:52:32.669941 (+ 11us) write_op.cc:312] APPLY: finished
0901 01:52:32.670037 (+ 96us) log.cc:844] Serialized 23 byte log entry
0901 01:52:32.670108 (+ 71us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:32.670129 (+ 21us) write_op.cc:454] Released schema lock
0901 01:52:32.670138 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0901 01:52:32.670219 (+ 81us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:32.670225 (+ 6us) write_op.cc:454] Released schema lock
Related trace 'op':
0901 01:52:32.791648 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:32.791774 (+ 126us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:32.791779 (+ 5us) write_op.cc:435] Acquired schema lock
0901 01:52:32.791781 (+ 2us) tablet.cc:660] Decoding operations
0901 01:52:32.791885 (+ 104us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:32.791894 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:32.791898 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0901 01:52:32.791954 (+ 56us) tablet.cc:699] Row locks acquired
0901 01:52:32.791957 (+ 3us) write_op.cc:260] PREPARE: finished
0901 01:52:32.791995 (+ 38us) write_op.cc:270] Start()
0901 01:52:32.792031 (+ 36us) write_op.cc:276] Timestamp: P: 1756691552791965 usec, L: 0
0901 01:52:32.792048 (+ 17us) op_driver.cc:348] REPLICATION: starting
0901 01:52:32.792215 (+ 167us) log.cc:844] Serialized 356 byte log entry
0901 01:52:32.907823 (+115608us) op_driver.cc:464] REPLICATION: finished
0901 01:52:32.915664 (+ 7841us) write_op.cc:301] APPLY: starting
0901 01:52:32.915681 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:32.915720 (+ 39us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:32.915782 (+ 62us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:32.915875 (+ 93us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:32.915908 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0901 01:52:32.915915 (+ 7us) write_op.cc:312] APPLY: finished
0901 01:52:32.916023 (+ 108us) log.cc:844] Serialized 23 byte log entry
0901 01:52:32.916115 (+ 92us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:32.916135 (+ 20us) write_op.cc:454] Released schema lock
0901 01:52:32.916144 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0901 01:52:32.916289 (+ 145us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:32.916294 (+ 5us) write_op.cc:454] Released schema lock
Related trace 'op':
0901 01:52:33.213332 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0901 01:52:33.213418 (+ 86us) write_op.cc:432] Acquiring schema lock in shared mode
0901 01:52:33.213423 (+ 5us) write_op.cc:435] Acquired schema lock
0901 01:52:33.213425 (+ 2us) tablet.cc:660] Decoding operations
0901 01:52:33.213508 (+ 83us) write_op.cc:620] Acquiring the partition lock for write op
0901 01:52:33.213516 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0901 01:52:33.213521 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0901 01:52:33.213569 (+ 48us) tablet.cc:699] Row locks acquired
0901 01:52:33.213571 (+ 2us) write_op.cc:260] PREPARE: finished
0901 01:52:33.213599 (+ 28us) write_op.cc:270] Start()
0901 01:52:33.213626 (+ 27us) write_op.cc:276] Timestamp: P: 1756691553213579 usec, L: 0
0901 01:52:33.213629 (+ 3us) op_driver.cc:348] REPLICATION: starting
0901 01:52:33.213835 (+ 206us) log.cc:844] Serialized 356 byte log entry
0901 01:52:33.254844 (+ 41009us) op_driver.cc:464] REPLICATION: finished
0901 01:52:33.255593 (+ 749us) write_op.cc:301] APPLY: starting
0901 01:52:33.255605 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
0901 01:52:33.255633 (+ 28us) tablet.cc:1368] finished BulkCheckPresence
0901 01:52:33.255672 (+ 39us) tablet.cc:1370] starting ApplyRowOperation cycle
0901 01:52:33.255717 (+ 45us) tablet.cc:1381] finished ApplyRowOperation cycle
0901 01:52:33.255749 (+ 32us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0901 01:52:33.255760 (+ 11us) write_op.cc:312] APPLY: finished
0901 01:52:33.255816 (+ 56us) log.cc:844] Serialized 23 byte log entry
0901 01:52:33.255850 (+ 34us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:33.255860 (+ 10us) write_op.cc:454] Released schema lock
0901 01:52:33.255873 (+ 13us) write_op.cc:341] FINISH: Updating metrics
0901 01:52:33.255923 (+ 50us) write_op.cc:489] Releasing partition, row and schema locks
0901 01:52:33.255926 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":17236,"reported_tablets":169,"child_traces":[["op",{"apply.queue_time_us":55,"apply.run_cpu_time_us":485,"apply.run_wall_time_us":484,"num_ops":4,"prepare.queue_time_us":7248,"prepare.run_cpu_time_us":878,"prepare.run_wall_time_us":878,"raft.queue_time_us":1140,"raft.run_cpu_time_us":448,"raft.run_wall_time_us":447,"replication_time_us":7473}],["op",{"apply.queue_time_us":133,"apply.run_cpu_time_us":534,"apply.run_wall_time_us":533,"num_ops":1,"prepare.queue_time_us":49,"prepare.run_cpu_time_us":621,"prepare.run_wall_time_us":621,"raft.queue_time_us":635,"raft.run_cpu_time_us":203,"raft.run_wall_time_us":203,"replication_time_us":11105}],["op",{"apply.queue_time_us":7755,"apply.run_cpu_time_us":561,"apply.run_wall_time_us":621,"num_ops":1,"prepare.queue_time_us":85,"prepare.run_cpu_time_us":637,"prepare.run_wall_time_us":637,"replication_time_us":115739}],["op",{"apply.queue_time_us":714,"apply.run_cpu_time_us":329,"apply.run_wall_time_us":329,"num_ops":1,"prepare.queue_time_us":42,"prepare.run_cpu_time_us":525,"prepare.run_wall_time_us":567,"replication_time_us":41201}]]}
I20250901 01:52:33.669835 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3c8e3368a866423ab6a867382b6f97fd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.671468 2778 tablet.cc:1722] T 6c0abbfc8d904dce989ce51961e4237d P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250901 01:52:33.673180 2779 tablet.cc:1722] T 6c0abbfc8d904dce989ce51961e4237d P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250901 01:52:33.675740 2778 tablet.cc:1722] T 33f637f6a8134215bb1995ec38440073 P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250901 01:52:33.678234 2613 raft_consensus.cc:491] T 617cc33d0d23439cb5f1e0c31bf11281 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:33.678390 2613 raft_consensus.cc:513] T 617cc33d0d23439cb5f1e0c31bf11281 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.679037 2613 leader_election.cc:290] T 617cc33d0d23439cb5f1e0c31bf11281 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:33.682040 992 leader_election.cc:343] T 617cc33d0d23439cb5f1e0c31bf11281 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 617cc33d0d23439cb5f1e0c31bf11281
W20250901 01:52:33.682978 992 leader_election.cc:336] T 617cc33d0d23439cb5f1e0c31bf11281 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:33.683076 992 leader_election.cc:304] T 617cc33d0d23439cb5f1e0c31bf11281 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:33.683470 2613 raft_consensus.cc:2747] T 617cc33d0d23439cb5f1e0c31bf11281 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:33.685935 2118 ts_tablet_manager.cc:1918] T c422881b40f54971a68530abfbd1945b P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:33.686072 2118 log.cc:1199] T c422881b40f54971a68530abfbd1945b P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/c422881b40f54971a68530abfbd1945b
I20250901 01:52:33.686417 2118 ts_tablet_manager.cc:1939] T c422881b40f54971a68530abfbd1945b P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:33.686780 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 175) took 5200 ms. Trace:
I20250901 01:52:33.686879 2118 rpcz_store.cc:276] 0901 01:52:28.486406 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.487235 (+ 829us) service_pool.cc:225] Handling call
0901 01:52:33.608634 (+5121399us) ts_tablet_manager.cc:1145] Deleting tablet c422881b40f54971a68530abfbd1945b
0901 01:52:33.608651 (+ 17us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:33.636276 (+ 27625us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:33.685818 (+ 49542us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:33.686764 (+ 946us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5121181}
I20250901 01:52:33.687398 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet c422881b40f54971a68530abfbd1945b (table default.Table_f611c1794e564a0496a761da46171d0e [id=dc22dac261644596b2d65e804791462d]) successfully deleted
W20250901 01:52:33.687708 1472 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 0c88c339e59a4c1da21756cba0645bdd (table default.Table_0927e3ea788d43bda5afeda483d87176 [id=ff772d20b287476f94716379b86d2a71], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet 0c88c339e59a4c1da21756cba0645bdd has no consensus state
I20250901 01:52:33.688100 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 177) took 5191 ms. Trace:
I20250901 01:52:33.688235 2118 rpcz_store.cc:276] 0901 01:52:28.496305 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.496355 (+ 50us) service_pool.cc:225] Handling call
0901 01:52:33.687991 (+5191636us) ts_tablet_manager.cc:1145] Deleting tablet e25923b2daf9429683b2d44a75a56189
0901 01:52:33.688003 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:33.688078 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5191428}
W20250901 01:52:33.688766 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet e25923b2daf9429683b2d44a75a56189 because the tablet was not found. No further retry: Not found: Tablet not found: e25923b2daf9429683b2d44a75a56189
I20250901 01:52:33.693313 2779 tablet.cc:1722] T 33f637f6a8134215bb1995ec38440073 P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250901 01:52:33.694015 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_07798246d8764fd19929e9cf252eec65"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.695361 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ced99963c892427aa038d23c116b0f6b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.696956 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_075d2b103d7640a1bebcd9099c104e2a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.699337 2781 raft_consensus.cc:491] T 93bcac1fd95d44dc927abfca81692921 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:33.699465 2781 raft_consensus.cc:513] T 93bcac1fd95d44dc927abfca81692921 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
W20250901 01:52:33.699807 992 consensus_peers.cc:489] T 89d40f0331114490b51de3ba6d773571 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250901 01:52:33.699746 2778 tablet.cc:1722] T 4cde5f4f02044b7bbd7d43b49aac4ae8 P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:33.699983 2781 leader_election.cc:290] T 93bcac1fd95d44dc927abfca81692921 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:33.701071 921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "93bcac1fd95d44dc927abfca81692921" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:33.701285 921 raft_consensus.cc:2466] T 93bcac1fd95d44dc927abfca81692921 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:33.702782 992 leader_election.cc:304] T 93bcac1fd95d44dc927abfca81692921 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
W20250901 01:52:33.703846 992 leader_election.cc:336] T 93bcac1fd95d44dc927abfca81692921 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:33.703843 2775 raft_consensus.cc:2802] T 93bcac1fd95d44dc927abfca81692921 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:33.704064 2775 raft_consensus.cc:491] T 93bcac1fd95d44dc927abfca81692921 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:33.704129 2775 raft_consensus.cc:3058] T 93bcac1fd95d44dc927abfca81692921 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
E20250901 01:52:33.704533 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5f0e054f62de4cf6bc6f69ad6664b192, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ef6832d0845044ed9f165ca3541b4c3f'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.704728 1696 catalog_manager.cc:6205] Error selecting replicas for tablet fd2ce82a4ee24d0faa44980a3967e187, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_63d0f2ead9d946d39ce40162ebd769d6'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.704880 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae9a4ad4fdfb4771ba23a91f7007447e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_5e84ae9a9c2a4084a1f9c5ffb4594de2'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.705029 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 086aa948f93644da8e5835b0fac87ddd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b4fbc34da806475c912052d0fab3fc53'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.705339 1696 catalog_manager.cc:6205] Error selecting replicas for tablet d602c8986c75492ead9936ac04e2622d, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_82d91a1ba2eb4265896824028dec8d3a'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.705538 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 93aa22afc0ad4d7fa24807ea3c5d5c21, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0f4b4094d2974b7591875375eda6bed0'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.705699 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9074543dea9949e7a085717b40913348, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_53e30205c5fd488eb3b48c685a159d04'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:33.705600 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4a11dd94c233437b9e2070694c1493fb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:33.705853 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 3a62034e688348ed9e2ac9de009ed3a2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_409dad98d7eb455a8e16edfb4acedb34'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.705958 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 18498c9cb09141d08c1c5c0854b63ebd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77f4810bc83c4d948531da2a45a3792e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.706044 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f5e3316adeae47cd848833dd84b972bb, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7570c3012f8a4ae7bcfc7346eadac92a'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.706136 1696 catalog_manager.cc:6205] Error selecting replicas for tablet a6b539e160194da3a849d9e2f1ec9859, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_03fb48eaab9b42858ab7cefb722e43a8'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.706228 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 6ce9521b925b4f97bb4a810ce77e147e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_9e04f976a93f4d30baccf90fa3622dfe'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:33.706311 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2f61edb50d7a42198bfd70639c7bf4de, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_a5c783cbdb694af69047b31d99ff8def'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:33.706929 2775 raft_consensus.cc:513] T 93bcac1fd95d44dc927abfca81692921 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:33.707244 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_79d23b8819cc45a39361c61f674d4ad6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.707590 2775 leader_election.cc:290] T 93bcac1fd95d44dc927abfca81692921 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:33.708395 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_52047f12df5b4272aeacad2f191b656b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.709694 921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "93bcac1fd95d44dc927abfca81692921" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:33.709903 921 raft_consensus.cc:3058] T 93bcac1fd95d44dc927abfca81692921 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:33.712724 2824 tablet.cc:1722] T 4cde5f4f02044b7bbd7d43b49aac4ae8 P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:33.712868 921 raft_consensus.cc:2466] T 93bcac1fd95d44dc927abfca81692921 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
W20250901 01:52:33.713330 992 leader_election.cc:336] T 93bcac1fd95d44dc927abfca81692921 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:33.713675 992 leader_election.cc:304] T 93bcac1fd95d44dc927abfca81692921 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:33.714468 2753 raft_consensus.cc:491] T 008d23f236c348b6be9aee718f8698af P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:33.714609 2753 raft_consensus.cc:513] T 008d23f236c348b6be9aee718f8698af P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.715348 2753 leader_election.cc:290] T 008d23f236c348b6be9aee718f8698af P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:33.715644 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2fea1a18108042c48c830e12fe5ae4d0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.716076 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "008d23f236c348b6be9aee718f8698af" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:33.716602 859 leader_election.cc:343] T 008d23f236c348b6be9aee718f8698af P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
W20250901 01:52:33.717028 858 leader_election.cc:336] T 008d23f236c348b6be9aee718f8698af P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:33.717124 858 leader_election.cc:304] T 008d23f236c348b6be9aee718f8698af P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:33.717351 2753 raft_consensus.cc:2747] T 008d23f236c348b6be9aee718f8698af P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:33.717581 2118 tablet_replica.cc:331] T b2e8b2cd10154de08ddd30c214592a4a P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:33.717578 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_bbf0c1f8158d491db77c0f31671c149c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.717856 2118 raft_consensus.cc:2241] T b2e8b2cd10154de08ddd30c214592a4a P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:33.718065 2118 raft_consensus.cc:2270] T b2e8b2cd10154de08ddd30c214592a4a P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:33.719174 2118 ts_tablet_manager.cc:1905] T b2e8b2cd10154de08ddd30c214592a4a P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:33.719398 2775 raft_consensus.cc:2802] T 93bcac1fd95d44dc927abfca81692921 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:33.719549 2775 raft_consensus.cc:695] T 93bcac1fd95d44dc927abfca81692921 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:33.719815 2775 consensus_queue.cc:237] T 93bcac1fd95d44dc927abfca81692921 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:33.721740 1484 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_85b9e2f3f8f7400aa833201daae93dc3" } modify_external_catalogs: true
I20250901 01:52:33.722002 1484 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_85b9e2f3f8f7400aa833201daae93dc3" } modify_external_catalogs: true
I20250901 01:52:33.725080 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a74ade85d504411d8ea9a1f5a1cabef7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.726805 1491 master_service.cc:948] ReplaceTablet: received request to replace tablet 8ca5cf4e99014c0d84a3acad4ef713af from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:33.728288 1490 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_0b769136b0664331945b30cfe0482fc6" } modify_external_catalogs: true
I20250901 01:52:33.728729 1490 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_0b769136b0664331945b30cfe0482fc6" } modify_external_catalogs: true
I20250901 01:52:33.730587 1488 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_c298ecfa989347b0b7a4d7227611539f" } new_table_name: "default.Table_7baa92b46d98454a891222be49a6ca48" modify_external_catalogs: true
I20250901 01:52:33.731704 1485 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_d1302673d68f424ebb54b220911ce385" } modify_external_catalogs: true
I20250901 01:52:33.732465 1486 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_409dad98d7eb455a8e16edfb4acedb34" } new_table_name: "default.Table_e225c82a2d6045f6b16915cdb72f5734" modify_external_catalogs: true
I20250901 01:52:33.734243 1482 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_f610cb75906d4c5cb3b57cfdb80ecefc" } new_table_name: "default.Table_e5753556ca8b4720a2e3cb3c5f99c27f" modify_external_catalogs: true
I20250901 01:52:33.735795 1485 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_d1302673d68f424ebb54b220911ce385" } modify_external_catalogs: true
I20250901 01:52:33.736589 2753 raft_consensus.cc:491] T 98c56eb62ad84f79b04f17a43d44c479 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:33.736768 2753 raft_consensus.cc:513] T 98c56eb62ad84f79b04f17a43d44c479 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.737102 2113 ts_tablet_manager.cc:1918] T 879fa66e55544118aafe504587393c66 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:33.737198 2113 log.cc:1199] T 879fa66e55544118aafe504587393c66 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/879fa66e55544118aafe504587393c66
I20250901 01:52:33.737444 2753 leader_election.cc:290] T 98c56eb62ad84f79b04f17a43d44c479 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:33.737486 2113 ts_tablet_manager.cc:1939] T 879fa66e55544118aafe504587393c66 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
W20250901 01:52:33.740666 858 leader_election.cc:336] T 98c56eb62ad84f79b04f17a43d44c479 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:33.743115 1491 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 8ca5cf4e99014c0d84a3acad4ef713af
W20250901 01:52:33.743356 1491 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 8ca5cf4e99014c0d84a3acad4ef713af: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:33.744000 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 8ca5cf4e99014c0d84a3acad4ef713af with delete_type TABLET_DATA_DELETED (replaced by tablet 465c02a130c54e09b5a74fb4dd2ef762) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:33.734622 1483 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_42422c44ca254d8ca989134c2433391f" } new_table_name: "default.Table_5efd7ee978b74c549a9d9a31d6404db7" modify_external_catalogs: true
I20250901 01:52:33.744098 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 8ca5cf4e99014c0d84a3acad4ef713af with delete_type TABLET_DATA_DELETED (replaced by tablet 465c02a130c54e09b5a74fb4dd2ef762) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:33.749903 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 267) took 5610 ms. Trace:
I20250901 01:52:33.752889 1490 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 1429a6a7a20e4f7c995e9acbaa8d495c
I20250901 01:52:33.754205 1491 catalog_manager.cc:6652] ReplaceTablet: tablet 8ca5cf4e99014c0d84a3acad4ef713af deleted and replaced by tablet 465c02a130c54e09b5a74fb4dd2ef762
I20250901 01:52:33.755456 1491 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_97526c9a6c9f4acbb6dedd78475571e0" } new_table_name: "default.Table_0c7355975d6b44dbbe3a77df942b4d43" modify_external_catalogs: true
I20250901 01:52:33.757079 2753 raft_consensus.cc:491] T 69f61ec283484f9d8b3eebcea27f741b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:33.757228 2753 raft_consensus.cc:513] T 69f61ec283484f9d8b3eebcea27f741b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:33.757951 2753 leader_election.cc:290] T 69f61ec283484f9d8b3eebcea27f741b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:33.758494 1490 master_service.cc:948] ReplaceTablet: received request to replace tablet ad45c90ba0da43c9860745e534664a98 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:33.752151 1151 tablet_bootstrap.cc:492] T 36e10328c34242ebb4dbd8f294162696 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:33.752902 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "98c56eb62ad84f79b04f17a43d44c479" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:33.758706 1151 ts_tablet_manager.cc:1397] T 36e10328c34242ebb4dbd8f294162696 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.146s user 0.013s sys 0.001s
I20250901 01:52:33.759630 1151 raft_consensus.cc:357] T 36e10328c34242ebb4dbd8f294162696 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.759923 1151 raft_consensus.cc:383] T 36e10328c34242ebb4dbd8f294162696 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:33.760039 1151 raft_consensus.cc:738] T 36e10328c34242ebb4dbd8f294162696 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:33.760325 1151 consensus_queue.cc:260] T 36e10328c34242ebb4dbd8f294162696 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.760850 1151 ts_tablet_manager.cc:1428] T 36e10328c34242ebb4dbd8f294162696 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250901 01:52:33.761251 1151 tablet_bootstrap.cc:492] T 435b1cca5a7d4d49959a310cb250b2a0 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
W20250901 01:52:33.762439 1482 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5d68ea97c71942c7b55b967ba870e0d7 consensus state has no leader
I20250901 01:52:33.762522 1485 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 0813cc68d4b0488ba60a523e4479971b
I20250901 01:52:33.762689 1151 tablet_bootstrap.cc:654] T 435b1cca5a7d4d49959a310cb250b2a0 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:33.763615 1486 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3a62034e688348ed9e2ac9de009ed3a2 has no consensus state
I20250901 01:52:33.763541 2113 rpcz_store.cc:276] 0901 01:52:28.139620 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.142909 (+ 3289us) service_pool.cc:225] Handling call
0901 01:52:33.621363 (+5478454us) ts_tablet_manager.cc:1145] Deleting tablet 879fa66e55544118aafe504587393c66
0901 01:52:33.621372 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:33.731622 (+110250us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:33.737007 (+ 5385us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:33.749879 (+ 12872us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5478328}
I20250901 01:52:33.764019 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 879fa66e55544118aafe504587393c66 (table default.Table_b362441111a241e19cc1a09bdeefea49 [id=f1d75be759db40ebb72ff32560f1d038]) successfully deleted
W20250901 01:52:33.764057 858 leader_election.cc:336] T 69f61ec283484f9d8b3eebcea27f741b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:33.764954 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 268) took 5625 ms. Trace:
I20250901 01:52:33.765084 2113 rpcz_store.cc:276] 0901 01:52:28.139696 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.143065 (+ 3369us) service_pool.cc:225] Handling call
0901 01:52:33.764856 (+5621791us) ts_tablet_manager.cc:1145] Deleting tablet ef58f477ebeb49639ed7d071e1a62c10
0901 01:52:33.764865 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:33.764936 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5621707}
W20250901 01:52:33.765445 859 leader_election.cc:343] T 98c56eb62ad84f79b04f17a43d44c479 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:33.765563 859 leader_election.cc:304] T 98c56eb62ad84f79b04f17a43d44c479 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:33.765651 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet ef58f477ebeb49639ed7d071e1a62c10 because the tablet was not found. No further retry: Not found: Tablet not found: ef58f477ebeb49639ed7d071e1a62c10
I20250901 01:52:33.765913 2113 tablet_replica.cc:331] T f98d1d8d660443cb8066a547e5d9b77f P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:33.766062 2753 raft_consensus.cc:2747] T 98c56eb62ad84f79b04f17a43d44c479 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:33.766775 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69f61ec283484f9d8b3eebcea27f741b" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:33.768013 859 leader_election.cc:343] T 69f61ec283484f9d8b3eebcea27f741b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:33.768149 859 leader_election.cc:304] T 69f61ec283484f9d8b3eebcea27f741b P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:33.768461 2608 raft_consensus.cc:2747] T 69f61ec283484f9d8b3eebcea27f741b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:33.768636 2608 raft_consensus.cc:491] T bd9b59f0864a497faf281edd6077668d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:33.768744 2608 raft_consensus.cc:513] T bd9b59f0864a497faf281edd6077668d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.769294 2608 leader_election.cc:290] T bd9b59f0864a497faf281edd6077668d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:33.773031 858 leader_election.cc:336] T bd9b59f0864a497faf281edd6077668d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:33.782809 1151 tablet_bootstrap.cc:492] T 435b1cca5a7d4d49959a310cb250b2a0 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:33.783039 1151 ts_tablet_manager.cc:1397] T 435b1cca5a7d4d49959a310cb250b2a0 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.000s
W20250901 01:52:33.783370 1483 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d0c3c363dda444519a17dca63ad9caa7 consensus state has no leader
I20250901 01:52:33.783870 1151 raft_consensus.cc:357] T 435b1cca5a7d4d49959a310cb250b2a0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.784082 1151 raft_consensus.cc:383] T 435b1cca5a7d4d49959a310cb250b2a0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:33.784163 1151 raft_consensus.cc:738] T 435b1cca5a7d4d49959a310cb250b2a0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:33.784358 1151 consensus_queue.cc:260] T 435b1cca5a7d4d49959a310cb250b2a0 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.784940 1151 ts_tablet_manager.cc:1428] T 435b1cca5a7d4d49959a310cb250b2a0 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250901 01:52:33.784797 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b50b5c594dac4bdb87a2c681fb940ed2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.784855 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_00ae1783091443c9a40ae251f8789c69"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.785316 1151 tablet_bootstrap.cc:492] T 0e4009963cfe42d6ba164fc828897f09 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:33.786450 1490 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet ad45c90ba0da43c9860745e534664a98
W20250901 01:52:33.786422 994 connection.cc:537] client connection to 127.0.104.61:42375 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250901 01:52:33.786695 1490 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet ad45c90ba0da43c9860745e534664a98: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:33.787149 1151 tablet_bootstrap.cc:654] T 0e4009963cfe42d6ba164fc828897f09 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:33.787388 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet ad45c90ba0da43c9860745e534664a98 with delete_type TABLET_DATA_DELETED (replaced by tablet 81e8c6567953484ab000fe2eb4398301) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:33.787947 1490 catalog_manager.cc:6652] ReplaceTablet: tablet ad45c90ba0da43c9860745e534664a98 deleted and replaced by tablet 81e8c6567953484ab000fe2eb4398301
I20250901 01:52:33.788538 2113 raft_consensus.cc:2241] T f98d1d8d660443cb8066a547e5d9b77f P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:33.788776 2113 pending_rounds.cc:70] T f98d1d8d660443cb8066a547e5d9b77f P fd27ab6665304aaea53adbd102dd4f58: Trying to abort 1 pending ops.
I20250901 01:52:33.788826 886 tablet_service.cc:1515] Processing DeleteTablet for tablet ad45c90ba0da43c9860745e534664a98 with delete_type TABLET_DATA_DELETED (replaced by tablet 81e8c6567953484ab000fe2eb4398301) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:33.788861 2113 pending_rounds.cc:77] T f98d1d8d660443cb8066a547e5d9b77f P fd27ab6665304aaea53adbd102dd4f58: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7195408603758309376 op_type: NO_OP noop_request { }
I20250901 01:52:33.788961 2113 raft_consensus.cc:2887] T f98d1d8d660443cb8066a547e5d9b77f P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250901 01:52:33.789048 2113 raft_consensus.cc:2270] T f98d1d8d660443cb8066a547e5d9b77f P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:33.789925 2113 ts_tablet_manager.cc:1905] T f98d1d8d660443cb8066a547e5d9b77f P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:33.790135 860 connection.cc:537] client connection to 127.0.104.61:42375 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250901 01:52:33.791545 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet c6f7c525f59b49938b697de98d4b5835 (table default.Table_db61e055dbd7450898e82c4c0c4763be [id=aabca67f6f754820977ac9c4c20f3e4f]). Expected version 1 got 0
I20250901 01:52:33.791886 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 858218c4427c41e980b2b5c969a5af40 (table default.Table_6d47266af0ed433eb1f2abbc81d18e69 [id=264cc469a73841e79538843479a165ab]). Expected version 1 got 0
I20250901 01:52:33.792151 1492 catalog_manager.cc:5582] T dabebc77f72e4bb19e264be2f496202a P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: UNKNOWN } } }
I20250901 01:52:33.792511 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet dabebc77f72e4bb19e264be2f496202a (table default.Table_2755f7caf53340afbd230181643366db [id=a4d0af97df5949cca940a81a41b8e454]). Expected version 1 got 0
I20250901 01:52:33.793023 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet b164b4a67a9143ceaff69b0ce49d63fe (table default.Table_4505d77e4269443995012fef50c7e7b5 [id=c1b1e9e3876244f68d31c63ce7bdebf1]). Expected version 2 got 0
I20250901 01:52:33.793503 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bd9b59f0864a497faf281edd6077668d" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:33.793701 1474 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.104.61:42375 (error 108) [suppressed 7 similar messages]
I20250901 01:52:33.794071 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_128cc7fec238422cb7a6b0a6601e4aad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.795365 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9dc74b1be75d429283e30615d85a0c43"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.796339 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_17a88d300a284b7987a6ebb13e321539"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.797328 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_cbb906c62cc746d48b9abecd3d75db38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.798310 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_856f7e03e3a94dbcb6c9a0261b012c4d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.799278 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7d374916e9674e87acf07ce75ab4c74e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.800172 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d7aba7528fac41e08ba3b7310f22ae05"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:33.800277 2154 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 01:52:33.789060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:42375 (local address 127.0.104.3:52821)
0901 01:52:33.789406 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0901 01:52:33.789428 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0901 01:52:33.789566 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 01:52:33.800114 (+ 10548us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.61:42375: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":207}
I20250901 01:52:33.801131 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a15d080f58fd4114961e1098a084e3b8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:33.801314 2755 negotiation.cc:337] Failed RPC negotiation. Trace:
0901 01:52:33.792037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:42375 (local address 127.0.104.2:56899)
0901 01:52:33.792287 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0901 01:52:33.792308 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0901 01:52:33.792452 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0901 01:52:33.801161 (+ 8709us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.61:42375: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":146}
I20250901 01:52:33.802587 2113 ts_tablet_manager.cc:1918] T f98d1d8d660443cb8066a547e5d9b77f P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:33.802688 2113 log.cc:1199] T f98d1d8d660443cb8066a547e5d9b77f P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/f98d1d8d660443cb8066a547e5d9b77f
I20250901 01:52:33.803088 2113 ts_tablet_manager.cc:1939] T f98d1d8d660443cb8066a547e5d9b77f P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
W20250901 01:52:33.803087 859 leader_election.cc:343] T bd9b59f0864a497faf281edd6077668d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:33.803207 859 leader_election.cc:304] T bd9b59f0864a497faf281edd6077668d P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:33.803711 2608 raft_consensus.cc:2747] T bd9b59f0864a497faf281edd6077668d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:33.803727 1482 master_service.cc:948] ReplaceTablet: received request to replace tablet 15e4d972e8254cf899c3e0b71b6e2bd9 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:33.804517 2778 tablet.cc:1722] T b164b4a67a9143ceaff69b0ce49d63fe P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250901 01:52:33.806555 2824 tablet.cc:1722] T b164b4a67a9143ceaff69b0ce49d63fe P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250901 01:52:33.811678 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 269) took 5671 ms. Trace:
I20250901 01:52:33.811818 2113 rpcz_store.cc:276] 0901 01:52:28.139803 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.143158 (+ 3355us) service_pool.cc:225] Handling call
0901 01:52:33.765869 (+5622711us) ts_tablet_manager.cc:1145] Deleting tablet f98d1d8d660443cb8066a547e5d9b77f
0901 01:52:33.765878 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:33.795375 (+ 29497us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:33.802488 (+ 7113us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:33.811659 (+ 9171us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":22371,"tablet-delete.queue_time_us":5622635}
I20250901 01:52:33.812305 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet f98d1d8d660443cb8066a547e5d9b77f (table default.Table_583918346a424cb293d2c35bfa4a5b5f [id=1f677c66803b4cb1bf585cb621a4df6e]) successfully deleted
I20250901 01:52:33.812912 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 270) took 5673 ms. Trace:
I20250901 01:52:33.814363 2113 rpcz_store.cc:276] 0901 01:52:28.139873 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.143247 (+ 3374us) service_pool.cc:225] Handling call
0901 01:52:33.812801 (+5669554us) ts_tablet_manager.cc:1145] Deleting tablet e1c8fe96a6ba4edba4f09c00f7add75e
0901 01:52:33.812810 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:33.812895 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5669476}
W20250901 01:52:33.815006 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet e1c8fe96a6ba4edba4f09c00f7add75e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e1c8fe96a6ba4edba4f09c00f7add75e already in progress: creating tablet
I20250901 01:52:33.815407 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 271) took 5675 ms. Trace:
I20250901 01:52:33.815774 2113 rpcz_store.cc:276] 0901 01:52:28.139934 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.143334 (+ 3400us) service_pool.cc:225] Handling call
0901 01:52:33.815332 (+5671998us) ts_tablet_manager.cc:1145] Deleting tablet 60285f2ba037433999f5455fb9cc7acd
0901 01:52:33.815341 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:33.815392 (+ 51us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5671899}
I20250901 01:52:33.816146 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_113bee03bfde4ab8886a675d7d05c7a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:33.816383 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 60285f2ba037433999f5455fb9cc7acd because the tablet was not found. No further retry: Not found: Tablet not found: 60285f2ba037433999f5455fb9cc7acd
I20250901 01:52:33.817045 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 272) took 5677 ms. Trace:
I20250901 01:52:33.817826 2113 rpcz_store.cc:276] 0901 01:52:28.139993 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.143451 (+ 3458us) service_pool.cc:225] Handling call
0901 01:52:33.816977 (+5673526us) ts_tablet_manager.cc:1145] Deleting tablet dd694cfa09564d3da5d279c0861a4c32
0901 01:52:33.816985 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:33.817030 (+ 45us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5673378}
I20250901 01:52:33.818069 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_abf504f59813457f9f92aed044f14b37"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:33.818480 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet dd694cfa09564d3da5d279c0861a4c32 because the tablet was not found. No further retry: Not found: Tablet not found: dd694cfa09564d3da5d279c0861a4c32
I20250901 01:52:33.818809 2113 tablet_replica.cc:331] T 4fca1435d82447f289fda4035aa2ed89 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:33.819298 2113 raft_consensus.cc:2241] T 4fca1435d82447f289fda4035aa2ed89 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:33.819269 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_03351255d46744e183dbcd9be0cb6cd6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.820012 2113 raft_consensus.cc:2270] T 4fca1435d82447f289fda4035aa2ed89 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:33.820571 2113 ts_tablet_manager.cc:1905] T 4fca1435d82447f289fda4035aa2ed89 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:33.820863 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2adcea9a198248e3a69f329bdef1e9f2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.821159 2775 raft_consensus.cc:491] T 61ddcf86c9f34726939cc116434f0822 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:33.821269 2775 raft_consensus.cc:513] T 61ddcf86c9f34726939cc116434f0822 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.821837 2775 leader_election.cc:290] T 61ddcf86c9f34726939cc116434f0822 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:33.822000 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2cc4a73a90b34717879d652efa8fd105"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.822070 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a6d7afeb6c99433d898386f3c195842c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.822166 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3e352025a08c4fe896298dc0c40b2767"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.793251 1492 catalog_manager.cc:5582] T 3eba54e5ddd14c518f4cbd539268ae37 P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: UNKNOWN } } }
I20250901 01:52:33.823853 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 81ce448b148447ddbc41875d863e0518 (table default.Table_6e461d9cb0ae4dd095975bbc21533794 [id=49673fa73f8c436e9b8026efc4f3cbbc]). Expected version 1 got 0
I20250901 01:52:33.824120 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 5f5396045979450193d461ba1334a5c9 (table default.Table_2b946a63bd574ac0971fa3d3b9c9a6a4 [id=9c3930e9fcfe4b49ba4bea67a2b0ad07]). Expected version 2 got 0
I20250901 01:52:33.824458 1151 tablet_bootstrap.cc:492] T 0e4009963cfe42d6ba164fc828897f09 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:33.824658 1151 ts_tablet_manager.cc:1397] T 0e4009963cfe42d6ba164fc828897f09 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.039s user 0.006s sys 0.001s
I20250901 01:52:33.824748 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 93bcac1fd95d44dc927abfca81692921 (table default.Table_b8f8ee74648c406eaf589d3ab80934db [id=b8f69313439543be9c11eea88dfa69c5]). Expected version 1 got 0
I20250901 01:52:33.825189 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 33f637f6a8134215bb1995ec38440073 (table default.Table_3d5afc22c4e14d6990aea31564dfde20 [id=557afe29976d40f6bfb6ae4ca348f887]). Expected version 2 got 0
I20250901 01:52:33.825609 1151 raft_consensus.cc:357] T 0e4009963cfe42d6ba164fc828897f09 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:33.825933 1151 raft_consensus.cc:383] T 0e4009963cfe42d6ba164fc828897f09 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:33.826021 1151 raft_consensus.cc:738] T 0e4009963cfe42d6ba164fc828897f09 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:33.826213 1151 consensus_queue.cc:260] T 0e4009963cfe42d6ba164fc828897f09 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:33.826705 1151 ts_tablet_manager.cc:1428] T 0e4009963cfe42d6ba164fc828897f09 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20250901 01:52:33.826926 992 consensus_peers.cc:489] T b2e8b2cd10154de08ddd30c214592a4a P fd27ab6665304aaea53adbd102dd4f58 -> Peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Couldn't send request to peer 00c282f779c1477d920f24dc0dc16628. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:33.827092 1151 tablet_bootstrap.cc:492] T fecf7d538b4c4c619182d5eccb44dd8b P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
W20250901 01:52:33.827387 992 consensus_peers.cc:489] T a6445b29591548be9f9577db1ba0bfa5 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250901 01:52:33.828088 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4ebf00a4d03d4717a948fe19771aa1a4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.828167 917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "61ddcf86c9f34726939cc116434f0822" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:33.828904 1151 tablet_bootstrap.cc:654] T fecf7d538b4c4c619182d5eccb44dd8b P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:33.828364 917 raft_consensus.cc:2466] T 61ddcf86c9f34726939cc116434f0822 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
W20250901 01:52:33.830847 1474 consensus_peers.cc:489] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f -> Peer d2ae2cb6af0f4b3a8f47ab1cdd6567d3 (127.0.104.61:42375): Couldn't send request to peer d2ae2cb6af0f4b3a8f47ab1cdd6567d3. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.61:42375: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:33.833325 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b1904c4db0974b25800300ac2cdd143a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.834687 2118 ts_tablet_manager.cc:1918] T b2e8b2cd10154de08ddd30c214592a4a P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 01:52:33.834832 2118 log.cc:1199] T b2e8b2cd10154de08ddd30c214592a4a P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/b2e8b2cd10154de08ddd30c214592a4a
I20250901 01:52:33.835225 2118 ts_tablet_manager.cc:1939] T b2e8b2cd10154de08ddd30c214592a4a P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:33.835382 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_483e888bd60d47cfb76d98018dfd4dc7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.836066 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 176) took 5340 ms. Trace:
I20250901 01:52:33.836160 2118 rpcz_store.cc:276] 0901 01:52:28.495980 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.496576 (+ 596us) service_pool.cc:225] Handling call
0901 01:52:33.713242 (+5216666us) ts_tablet_manager.cc:1145] Deleting tablet b2e8b2cd10154de08ddd30c214592a4a
0901 01:52:33.717535 (+ 4293us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:33.831721 (+114186us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:33.834581 (+ 2860us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:33.836044 (+ 1463us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5216499}
I20250901 01:52:33.838532 2113 ts_tablet_manager.cc:1918] T 4fca1435d82447f289fda4035aa2ed89 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:33.838647 2113 log.cc:1199] T 4fca1435d82447f289fda4035aa2ed89 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/4fca1435d82447f289fda4035aa2ed89
I20250901 01:52:33.838923 2113 ts_tablet_manager.cc:1939] T 4fca1435d82447f289fda4035aa2ed89 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:33.841284 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 273) took 5701 ms. Trace:
W20250901 01:52:33.843952 1491 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ff2bd017417c41b5b8225b8f00283e63 has no consensus state
I20250901 01:52:33.842765 992 leader_election.cc:304] T 61ddcf86c9f34726939cc116434f0822 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:33.844384 2781 raft_consensus.cc:2802] T 61ddcf86c9f34726939cc116434f0822 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:33.844489 2781 raft_consensus.cc:491] T 61ddcf86c9f34726939cc116434f0822 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:33.844560 2781 raft_consensus.cc:3058] T 61ddcf86c9f34726939cc116434f0822 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:33.845578 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3f445b3e0bc346bf923d774bd0be4772"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.846747 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2ad28cc169874c8e902b866b36dee351"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.847301 2781 raft_consensus.cc:513] T 61ddcf86c9f34726939cc116434f0822 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.847771 1484 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet d0b7c84ec9394a20ad77add3808e555a
I20250901 01:52:33.847962 2781 leader_election.cc:290] T 61ddcf86c9f34726939cc116434f0822 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:33.847990 1484 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet d0b7c84ec9394a20ad77add3808e555a: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:33.848517 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5e24cacd020c464c87beea1cae27090a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.848628 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet d0b7c84ec9394a20ad77add3808e555a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:33 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:33.849188 886 tablet_service.cc:1515] Processing DeleteTablet for tablet d0b7c84ec9394a20ad77add3808e555a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:33 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:33.849670 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4d9a705be71b4367901d9a67221fa98e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:33.849982 992 consensus_peers.cc:489] T 9127f9c5239e445ba86012951b7c75d9 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250901 01:52:33.853103 1151 tablet_bootstrap.cc:492] T fecf7d538b4c4c619182d5eccb44dd8b P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:33.853700 1151 ts_tablet_manager.cc:1397] T fecf7d538b4c4c619182d5eccb44dd8b P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.002s
I20250901 01:52:33.854146 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3b188a9b5413405aa95af2f7eb0c6cac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.855209 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "61ddcf86c9f34726939cc116434f0822" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:33.855401 914 raft_consensus.cc:3058] T 61ddcf86c9f34726939cc116434f0822 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:33.855795 1151 raft_consensus.cc:357] T fecf7d538b4c4c619182d5eccb44dd8b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.856348 1151 raft_consensus.cc:383] T fecf7d538b4c4c619182d5eccb44dd8b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:33.856593 1151 raft_consensus.cc:738] T fecf7d538b4c4c619182d5eccb44dd8b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:33.856997 1151 consensus_queue.cc:260] T fecf7d538b4c4c619182d5eccb44dd8b P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.858012 914 raft_consensus.cc:2466] T 61ddcf86c9f34726939cc116434f0822 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
W20250901 01:52:33.856004 992 leader_election.cc:336] T 61ddcf86c9f34726939cc116434f0822 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:33.860460 1151 ts_tablet_manager.cc:1428] T fecf7d538b4c4c619182d5eccb44dd8b P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s
I20250901 01:52:33.861138 1151 tablet_bootstrap.cc:492] T f23df6f253a943fa8e146bb8c811ef51 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:33.861397 2347 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "7ce2288f784044eea010664e3def133f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3add7edb97a40ef9f7c6bc011d13227" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 43871 } } peers { permanent_uuid: "d2ae2cb6af0f4b3a8f47ab1cdd6567d3" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 42375 } } peers { permanent_uuid: "7ce2288f784044eea010664e3def133f" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 44231 } } }
I20250901 01:52:33.861654 2347 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f [sys.catalog]: This master's current role is: LEADER
I20250901 01:52:33.862690 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d9f812fd173b4a4eb195d268d82fa70e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.863260 1151 tablet_bootstrap.cc:654] T f23df6f253a943fa8e146bb8c811ef51 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:33.841375 2113 rpcz_store.cc:276] 0901 01:52:28.140075 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.143625 (+ 3550us) service_pool.cc:225] Handling call
0901 01:52:33.818750 (+5675125us) ts_tablet_manager.cc:1145] Deleting tablet 4fca1435d82447f289fda4035aa2ed89
0901 01:52:33.818761 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:33.835584 (+ 16823us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:33.838459 (+ 2875us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:33.841264 (+ 2805us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5674955}
W20250901 01:52:33.865972 858 consensus_peers.cc:489] T 858218c4427c41e980b2b5c969a5af40 P 00c282f779c1477d920f24dc0dc16628 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:33.866102 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 4fca1435d82447f289fda4035aa2ed89 (table default.Table_fbc57d539a2b41e59dfc5369ac8c92e1 [id=8e14f99474a1467baed19f5268f09c5e]) successfully deleted
I20250901 01:52:33.866465 2113 tablet_replica.cc:331] T 9fbd44641d9e43c3b592e388e4adf350 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:33.866693 2113 raft_consensus.cc:2241] T 9fbd44641d9e43c3b592e388e4adf350 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:33.866871 2113 raft_consensus.cc:2270] T 9fbd44641d9e43c3b592e388e4adf350 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:33.867518 2113 ts_tablet_manager.cc:1905] T 9fbd44641d9e43c3b592e388e4adf350 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:33.867658 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9e38cd543b41487583f492253090d893"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.869284 2753 raft_consensus.cc:491] T 0e4009963cfe42d6ba164fc828897f09 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:33.869439 2753 raft_consensus.cc:513] T 0e4009963cfe42d6ba164fc828897f09 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:33.870071 2753 leader_election.cc:290] T 0e4009963cfe42d6ba164fc828897f09 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:33.871810 858 leader_election.cc:336] T 0e4009963cfe42d6ba164fc828897f09 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:33.875016 2113 ts_tablet_manager.cc:1918] T 9fbd44641d9e43c3b592e388e4adf350 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:33.875114 2113 log.cc:1199] T 9fbd44641d9e43c3b592e388e4adf350 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/9fbd44641d9e43c3b592e388e4adf350
I20250901 01:52:33.875411 2113 ts_tablet_manager.cc:1939] T 9fbd44641d9e43c3b592e388e4adf350 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:33.876116 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 277) took 5655 ms. Trace:
I20250901 01:52:33.876647 2113 rpcz_store.cc:276] 0901 01:52:28.220765 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.220833 (+ 68us) service_pool.cc:225] Handling call
0901 01:52:33.866419 (+5645586us) ts_tablet_manager.cc:1145] Deleting tablet 9fbd44641d9e43c3b592e388e4adf350
0901 01:52:33.866428 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:33.872116 (+ 5688us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:33.874939 (+ 2823us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:33.876083 (+ 1144us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5645274}
I20250901 01:52:33.876734 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1eda3ccaf8f8426e81cecd3ef107e3d7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.877128 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 9fbd44641d9e43c3b592e388e4adf350 (table default.Table_2fbca8463d6546c9b8f5b018fc60251d [id=6fdd3d817dc74551923021bd716ba58b]) successfully deleted
I20250901 01:52:33.877543 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 278) took 5655 ms. Trace:
I20250901 01:52:33.877662 2113 rpcz_store.cc:276] 0901 01:52:28.221618 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.221788 (+ 170us) service_pool.cc:225] Handling call
0901 01:52:33.877420 (+5655632us) ts_tablet_manager.cc:1145] Deleting tablet 0cba5329ab31440999faec599d704633
0901 01:52:33.877429 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:33.877524 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5655464}
W20250901 01:52:33.878159 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 0cba5329ab31440999faec599d704633 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0cba5329ab31440999faec599d704633 already in progress: creating tablet
I20250901 01:52:33.878541 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 279) took 5656 ms. Trace:
I20250901 01:52:33.878458 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_91f0c67f0a3d4ef09aabe7cd32b4e611"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.878774 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b26cbcc11049415ea19be98cfa591f91"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.878787 2113 rpcz_store.cc:276] 0901 01:52:28.221744 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.221994 (+ 250us) service_pool.cc:225] Handling call
0901 01:52:33.878443 (+5656449us) ts_tablet_manager.cc:1145] Deleting tablet 7f46e9ddadaf4ff986d8c991178daa0c
0901 01:52:33.878452 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:33.878527 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5656286}
I20250901 01:52:33.878602 1151 tablet_bootstrap.cc:492] T f23df6f253a943fa8e146bb8c811ef51 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:33.879200 1151 ts_tablet_manager.cc:1397] T f23df6f253a943fa8e146bb8c811ef51 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.001s
I20250901 01:52:33.879580 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 282) took 5392 ms. Trace:
I20250901 01:52:33.879644 2113 rpcz_store.cc:276] 0901 01:52:28.487151 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.489174 (+ 2023us) service_pool.cc:225] Handling call
0901 01:52:33.879444 (+5390270us) ts_tablet_manager.cc:1145] Deleting tablet c422881b40f54971a68530abfbd1945b
0901 01:52:33.879451 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:33.879566 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5389760}
W20250901 01:52:33.880013 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet c422881b40f54971a68530abfbd1945b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c422881b40f54971a68530abfbd1945b already in progress: creating tablet
I20250901 01:52:33.879932 1151 raft_consensus.cc:357] T f23df6f253a943fa8e146bb8c811ef51 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.880107 1151 raft_consensus.cc:383] T f23df6f253a943fa8e146bb8c811ef51 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:33.880157 1151 raft_consensus.cc:738] T f23df6f253a943fa8e146bb8c811ef51 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:33.880285 2113 tablet_replica.cc:331] T b2e8b2cd10154de08ddd30c214592a4a P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:33.880348 1151 consensus_queue.cc:260] T f23df6f253a943fa8e146bb8c811ef51 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.880509 2113 raft_consensus.cc:2241] T b2e8b2cd10154de08ddd30c214592a4a P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:33.880823 1151 ts_tablet_manager.cc:1428] T f23df6f253a943fa8e146bb8c811ef51 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250901 01:52:33.880985 2113 raft_consensus.cc:2270] T b2e8b2cd10154de08ddd30c214592a4a P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:33.881063 1151 tablet_bootstrap.cc:492] T 69f61ec283484f9d8b3eebcea27f741b P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:33.881469 2113 ts_tablet_manager.cc:1905] T b2e8b2cd10154de08ddd30c214592a4a P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:33.882354 1484 master_service.cc:948] ReplaceTablet: received request to replace tablet 482a9a2cb81f448bb2823892e35920d4 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:33.882627 1151 tablet_bootstrap.cc:654] T 69f61ec283484f9d8b3eebcea27f741b P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:33.882959 1491 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_6e461d9cb0ae4dd095975bbc21533794" } modify_external_catalogs: true
I20250901 01:52:33.883150 1491 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_6e461d9cb0ae4dd095975bbc21533794" } modify_external_catalogs: true
I20250901 01:52:33.884490 1488 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_75ba3ba70ca24a94920e7bce96190ff2" } modify_external_catalogs: true
I20250901 01:52:33.884809 1488 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_75ba3ba70ca24a94920e7bce96190ff2" } modify_external_catalogs: true
I20250901 01:52:33.888556 2113 ts_tablet_manager.cc:1918] T b2e8b2cd10154de08ddd30c214592a4a P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 01:52:33.888644 2113 log.cc:1199] T b2e8b2cd10154de08ddd30c214592a4a P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/b2e8b2cd10154de08ddd30c214592a4a
I20250901 01:52:33.889230 2113 ts_tablet_manager.cc:1939] T b2e8b2cd10154de08ddd30c214592a4a P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:33.890020 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 283) took 5394 ms. Trace:
I20250901 01:52:33.890095 2113 rpcz_store.cc:276] 0901 01:52:28.495957 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.513177 (+ 17220us) service_pool.cc:225] Handling call
0901 01:52:33.880225 (+5367048us) ts_tablet_manager.cc:1145] Deleting tablet b2e8b2cd10154de08ddd30c214592a4a
0901 01:52:33.880232 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:33.884880 (+ 4648us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:33.888476 (+ 3596us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:33.890003 (+ 1527us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5366449}
I20250901 01:52:33.890528 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet b2e8b2cd10154de08ddd30c214592a4a (table default.Table_4a412a59b6db421eb0187ffd02ecf541 [id=fd72e46c84634b6c9ea74970480ecc62]) successfully deleted
I20250901 01:52:33.891407 1151 tablet_bootstrap.cc:492] T 69f61ec283484f9d8b3eebcea27f741b P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:33.891515 1151 ts_tablet_manager.cc:1397] T 69f61ec283484f9d8b3eebcea27f741b P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20250901 01:52:33.892216 1151 raft_consensus.cc:357] T 69f61ec283484f9d8b3eebcea27f741b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:33.892360 1151 raft_consensus.cc:383] T 69f61ec283484f9d8b3eebcea27f741b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:33.892434 1151 raft_consensus.cc:738] T 69f61ec283484f9d8b3eebcea27f741b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:33.892619 1151 consensus_queue.cc:260] T 69f61ec283484f9d8b3eebcea27f741b P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:33.893097 1151 ts_tablet_manager.cc:1428] T 69f61ec283484f9d8b3eebcea27f741b P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250901 01:52:33.893327 1151 tablet_bootstrap.cc:492] T 4b4aa063ffe948e69e47f03102581899 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
W20250901 01:52:33.868132 992 consensus_peers.cc:489] T 3a8231590ea643b0a1f83eec4d9a8191 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250901 01:52:33.894248 992 leader_election.cc:336] T 61ddcf86c9f34726939cc116434f0822 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:33.894497 992 leader_election.cc:304] T 61ddcf86c9f34726939cc116434f0822 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:33.894832 2775 raft_consensus.cc:2802] T 61ddcf86c9f34726939cc116434f0822 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:33.894959 2775 raft_consensus.cc:695] T 61ddcf86c9f34726939cc116434f0822 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:33.895149 2775 consensus_queue.cc:237] T 61ddcf86c9f34726939cc116434f0822 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.895076 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_cb8bc5cfe57e4a04a6b9a1942d2845e9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.896436 1151 tablet_bootstrap.cc:654] T 4b4aa063ffe948e69e47f03102581899 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:33.903385 2113 tablet_replica.cc:331] T a6445b29591548be9f9577db1ba0bfa5 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:33.903657 2113 raft_consensus.cc:2241] T a6445b29591548be9f9577db1ba0bfa5 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:33.904028 2113 raft_consensus.cc:2270] T a6445b29591548be9f9577db1ba0bfa5 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:33.903973 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_94e7b4382e15478eb67f98297e40734e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.904544 2113 ts_tablet_manager.cc:1905] T a6445b29591548be9f9577db1ba0bfa5 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:33.905702 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_92a690b6856b461f9e9f9286728f3c60"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.907439 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_78ff2973e9b44167afb4b8dad6c2225d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.908113 1151 tablet_bootstrap.cc:492] T 4b4aa063ffe948e69e47f03102581899 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:33.908227 1151 ts_tablet_manager.cc:1397] T 4b4aa063ffe948e69e47f03102581899 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20250901 01:52:33.908973 1151 raft_consensus.cc:357] T 4b4aa063ffe948e69e47f03102581899 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.909155 1151 raft_consensus.cc:383] T 4b4aa063ffe948e69e47f03102581899 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:33.909087 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a7e9aada53cc4f04bed6ffed11c17662"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.909231 1151 raft_consensus.cc:738] T 4b4aa063ffe948e69e47f03102581899 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:33.909272 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a7734f395eaf4ae0933ad37743bde7bd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.909386 1151 consensus_queue.cc:260] T 4b4aa063ffe948e69e47f03102581899 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.909835 1151 ts_tablet_manager.cc:1428] T 4b4aa063ffe948e69e47f03102581899 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250901 01:52:33.910073 1151 tablet_bootstrap.cc:492] T b4278a013ae84082bc7a8497b04ac147 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:33.911067 2113 ts_tablet_manager.cc:1918] T a6445b29591548be9f9577db1ba0bfa5 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:33.911172 2113 log.cc:1199] T a6445b29591548be9f9577db1ba0bfa5 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/a6445b29591548be9f9577db1ba0bfa5
I20250901 01:52:33.911554 2113 ts_tablet_manager.cc:1939] T a6445b29591548be9f9577db1ba0bfa5 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:33.911705 1151 tablet_bootstrap.cc:654] T b4278a013ae84082bc7a8497b04ac147 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:33.912307 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 286) took 5051 ms. Trace:
I20250901 01:52:33.912384 2113 rpcz_store.cc:276] 0901 01:52:28.860896 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.860950 (+ 54us) service_pool.cc:225] Handling call
0901 01:52:33.903337 (+5042387us) ts_tablet_manager.cc:1145] Deleting tablet a6445b29591548be9f9577db1ba0bfa5
0901 01:52:33.903346 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:33.907606 (+ 4260us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:33.910977 (+ 3371us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:33.912289 (+ 1312us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5042179}
I20250901 01:52:33.912881 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet b2e8b2cd10154de08ddd30c214592a4a (table default.Table_4a412a59b6db421eb0187ffd02ecf541 [id=fd72e46c84634b6c9ea74970480ecc62]) successfully deleted
I20250901 01:52:33.912881 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet a6445b29591548be9f9577db1ba0bfa5 (table default.Table_e225c82a2d6045f6b16915cdb72f5734 [id=6f29cdc9974049368af2ae10beadb772]) successfully deleted
I20250901 01:52:33.913164 2113 tablet_replica.cc:331] T 1495de70f0844f82a0eea99c059f076b P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:33.913355 2113 raft_consensus.cc:2241] T 1495de70f0844f82a0eea99c059f076b P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:33.913589 2113 raft_consensus.cc:2270] T 1495de70f0844f82a0eea99c059f076b P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:33.914037 2113 ts_tablet_manager.cc:1905] T 1495de70f0844f82a0eea99c059f076b P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:33.913988 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_21b132e530544aba8526b1cca388b110"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.914070 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f40f671ec6e4498b92990e56f4c1fe1e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.914928 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_75630960e749498395d247dba46de7b9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.916654 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2daa694877d3430eae2fc3525eca04b1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.917300 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 179) took 5420 ms. Trace:
I20250901 01:52:33.917413 2118 rpcz_store.cc:276] 0901 01:52:28.497191 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.497242 (+ 51us) service_pool.cc:225] Handling call
0901 01:52:33.912935 (+5415693us) ts_tablet_manager.cc:1145] Deleting tablet bb0cf5a6a27045a08df6845d7473184d
0901 01:52:33.912949 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:33.917276 (+ 4327us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5415524}
W20250901 01:52:33.918146 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet bb0cf5a6a27045a08df6845d7473184d because the tablet was not found. No further retry: Not found: Tablet not found: bb0cf5a6a27045a08df6845d7473184d
I20250901 01:52:33.918691 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 180) took 5421 ms. Trace:
I20250901 01:52:33.920073 2113 ts_tablet_manager.cc:1918] T 1495de70f0844f82a0eea99c059f076b P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:33.920012 1055 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4009963cfe42d6ba164fc828897f09" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:33.920176 2113 log.cc:1199] T 1495de70f0844f82a0eea99c059f076b P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/1495de70f0844f82a0eea99c059f076b
I20250901 01:52:33.920238 1055 raft_consensus.cc:2466] T 0e4009963cfe42d6ba164fc828897f09 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00c282f779c1477d920f24dc0dc16628 in term 0.
I20250901 01:52:33.920610 2113 ts_tablet_manager.cc:1939] T 1495de70f0844f82a0eea99c059f076b P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:33.921298 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 287) took 5057 ms. Trace:
I20250901 01:52:33.921365 2113 rpcz_store.cc:276] 0901 01:52:28.863789 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.863855 (+ 66us) service_pool.cc:225] Handling call
0901 01:52:33.913124 (+5049269us) ts_tablet_manager.cc:1145] Deleting tablet 1495de70f0844f82a0eea99c059f076b
0901 01:52:33.913133 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:33.916936 (+ 3803us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:33.919979 (+ 3043us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:33.921282 (+ 1303us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5049053}
I20250901 01:52:33.921741 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 1495de70f0844f82a0eea99c059f076b (table default.Table_7570c3012f8a4ae7bcfc7346eadac92a [id=70a5ce71d16b47d4bc6dae97be4548e7]) successfully deleted
I20250901 01:52:33.923795 1151 tablet_bootstrap.cc:492] T b4278a013ae84082bc7a8497b04ac147 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:33.923952 1151 ts_tablet_manager.cc:1397] T b4278a013ae84082bc7a8497b04ac147 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20250901 01:52:33.924695 1151 raft_consensus.cc:357] T b4278a013ae84082bc7a8497b04ac147 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.924893 1151 raft_consensus.cc:383] T b4278a013ae84082bc7a8497b04ac147 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:33.924973 1151 raft_consensus.cc:738] T b4278a013ae84082bc7a8497b04ac147 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:33.925164 1151 consensus_queue.cc:260] T b4278a013ae84082bc7a8497b04ac147 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.925693 1151 ts_tablet_manager.cc:1428] T b4278a013ae84082bc7a8497b04ac147 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250901 01:52:33.925981 1151 tablet_bootstrap.cc:492] T e1c8fe96a6ba4edba4f09c00f7add75e P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:33.927628 1151 tablet_bootstrap.cc:654] T e1c8fe96a6ba4edba4f09c00f7add75e P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:33.928706 2113 tablet_replica.cc:331] T 8b9d79a25452410891c09fd3f7c4be6d P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:33.928970 2113 raft_consensus.cc:2241] T 8b9d79a25452410891c09fd3f7c4be6d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:33.929132 2113 raft_consensus.cc:2270] T 8b9d79a25452410891c09fd3f7c4be6d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250901 01:52:33.925956 858 consensus_peers.cc:489] T 71941b815d6840478c5acc1cbb187cc9 P 00c282f779c1477d920f24dc0dc16628 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250901 01:52:33.929844 2113 ts_tablet_manager.cc:1905] T 8b9d79a25452410891c09fd3f7c4be6d P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:33.921996 859 leader_election.cc:304] T 0e4009963cfe42d6ba164fc828897f09 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:33.930364 2753 raft_consensus.cc:2802] T 0e4009963cfe42d6ba164fc828897f09 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:33.930473 2753 raft_consensus.cc:491] T 0e4009963cfe42d6ba164fc828897f09 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:33.930553 2753 raft_consensus.cc:3058] T 0e4009963cfe42d6ba164fc828897f09 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:33.933648 2753 raft_consensus.cc:513] T 0e4009963cfe42d6ba164fc828897f09 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:33.934592 2753 leader_election.cc:290] T 0e4009963cfe42d6ba164fc828897f09 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:33.935873 2113 ts_tablet_manager.cc:1918] T 8b9d79a25452410891c09fd3f7c4be6d P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250901 01:52:33.936239 858 leader_election.cc:336] T 0e4009963cfe42d6ba164fc828897f09 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:33.918797 2118 rpcz_store.cc:276] 0901 01:52:28.497683 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.497740 (+ 57us) service_pool.cc:225] Handling call
0901 01:52:33.918585 (+5420845us) ts_tablet_manager.cc:1145] Deleting tablet a54af99bc10045e8b0b6868e5aed98b4
0901 01:52:33.918599 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:33.918671 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5420676}
I20250901 01:52:33.936780 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_dccdddc93c9940e1926f20d83f87555d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.937386 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 181) took 5439 ms. Trace:
I20250901 01:52:33.937475 2118 rpcz_store.cc:276] 0901 01:52:28.498264 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.498308 (+ 44us) service_pool.cc:225] Handling call
0901 01:52:33.937263 (+5438955us) ts_tablet_manager.cc:1145] Deleting tablet 7a2f6e2c7ee2402ea310c058e046416c
0901 01:52:33.937276 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:33.937363 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5438806}
W20250901 01:52:33.937532 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet a54af99bc10045e8b0b6868e5aed98b4 because the tablet was not found. No further retry: Not found: Tablet not found: a54af99bc10045e8b0b6868e5aed98b4
I20250901 01:52:33.937896 2118 tablet_replica.cc:331] T 26c8ed083cea4dc0acca80d00b11b29b P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:33.938114 2118 raft_consensus.cc:2241] T 26c8ed083cea4dc0acca80d00b11b29b P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:33.938277 2118 raft_consensus.cc:2270] T 26c8ed083cea4dc0acca80d00b11b29b P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250901 01:52:33.938405 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 7a2f6e2c7ee2402ea310c058e046416c because the tablet was not found. No further retry: Not found: Tablet not found: 7a2f6e2c7ee2402ea310c058e046416c
I20250901 01:52:33.938849 2118 ts_tablet_manager.cc:1905] T 26c8ed083cea4dc0acca80d00b11b29b P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:33.940395 2113 log.cc:1199] T 8b9d79a25452410891c09fd3f7c4be6d P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/8b9d79a25452410891c09fd3f7c4be6d
I20250901 01:52:33.941143 2113 ts_tablet_manager.cc:1939] T 8b9d79a25452410891c09fd3f7c4be6d P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:33.941838 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 288) took 5075 ms. Trace:
I20250901 01:52:33.941921 2113 rpcz_store.cc:276] 0901 01:52:28.866805 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.867170 (+ 365us) service_pool.cc:225] Handling call
0901 01:52:33.928660 (+5061490us) ts_tablet_manager.cc:1145] Deleting tablet 8b9d79a25452410891c09fd3f7c4be6d
0901 01:52:33.928670 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:33.932954 (+ 4284us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:33.935797 (+ 2843us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:33.941817 (+ 6020us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5061343}
I20250901 01:52:33.942317 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 8b9d79a25452410891c09fd3f7c4be6d (table default.Table_442e11b9211c4b209c68f653d7bfc313 [id=af349c3f534e4e8e99fdae0344187c44]) successfully deleted
I20250901 01:52:33.944339 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 289) took 4963 ms. Trace:
I20250901 01:52:33.944778 2113 rpcz_store.cc:276] 0901 01:52:28.980849 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.982705 (+ 1856us) service_pool.cc:225] Handling call
0901 01:52:33.944210 (+4961505us) ts_tablet_manager.cc:1145] Deleting tablet 7304937048ca48a7bd153f3cdcfaeff0
0901 01:52:33.944220 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:33.944322 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4961314}
W20250901 01:52:33.945418 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 7304937048ca48a7bd153f3cdcfaeff0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7304937048ca48a7bd153f3cdcfaeff0 already in progress: creating tablet
I20250901 01:52:33.945755 2113 tablet_replica.cc:331] T 186ec810d02e4748aad15228b5866ac1 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:33.946206 2113 raft_consensus.cc:2241] T 186ec810d02e4748aad15228b5866ac1 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:33.946504 1486 catalog_manager.cc:5582] T 858218c4427c41e980b2b5c969a5af40 P 00c282f779c1477d920f24dc0dc16628 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00c282f779c1477d920f24dc0dc16628 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "00c282f779c1477d920f24dc0dc16628" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: HEALTHY } } }
I20250901 01:52:33.946823 2113 raft_consensus.cc:2270] T 186ec810d02e4748aad15228b5866ac1 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:33.947098 1486 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 858218c4427c41e980b2b5c969a5af40 (table default.Table_6d47266af0ed433eb1f2abbc81d18e69 [id=264cc469a73841e79538843479a165ab]). Expected version 1 got 0
I20250901 01:52:33.947504 1486 catalog_manager.cc:5582] T 5f5396045979450193d461ba1334a5c9 P 00c282f779c1477d920f24dc0dc16628 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00c282f779c1477d920f24dc0dc16628 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "00c282f779c1477d920f24dc0dc16628" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: HEALTHY } } }
I20250901 01:52:33.947798 2113 ts_tablet_manager.cc:1905] T 186ec810d02e4748aad15228b5866ac1 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:33.948040 1486 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 5f5396045979450193d461ba1334a5c9 (table default.Table_2b946a63bd574ac0971fa3d3b9c9a6a4 [id=9c3930e9fcfe4b49ba4bea67a2b0ad07]). Expected version 2 got 0
I20250901 01:52:33.948292 1486 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet b164b4a67a9143ceaff69b0ce49d63fe (table default.Table_4505d77e4269443995012fef50c7e7b5 [id=c1b1e9e3876244f68d31c63ce7bdebf1]). Expected version 2 got 0
I20250901 01:52:33.948639 1486 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 6c0abbfc8d904dce989ce51961e4237d (table default.Table_cca52da43da6439eb88ea9715e4a0d50 [id=f0dfb3811e744bd4ba6d59195cebde6f]). Expected version 3 got 2
I20250901 01:52:33.949198 1486 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 98c56eb62ad84f79b04f17a43d44c479 (table default.Table_50ff85e8f1314f4499e76299441bc110 [id=00e7f7fe6dc44f2787b9dd8e3cbf4a51]). Expected version 1 got 0
I20250901 01:52:33.955066 2113 ts_tablet_manager.cc:1918] T 186ec810d02e4748aad15228b5866ac1 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:33.961591 1151 tablet_bootstrap.cc:492] T e1c8fe96a6ba4edba4f09c00f7add75e P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:33.973451 1151 ts_tablet_manager.cc:1397] T e1c8fe96a6ba4edba4f09c00f7add75e P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.048s user 0.009s sys 0.007s
I20250901 01:52:33.973464 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e40f8aeb767542149c15cdd9e50e30ca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.974205 1151 raft_consensus.cc:357] T e1c8fe96a6ba4edba4f09c00f7add75e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.974359 1151 raft_consensus.cc:383] T e1c8fe96a6ba4edba4f09c00f7add75e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:33.974407 1151 raft_consensus.cc:738] T e1c8fe96a6ba4edba4f09c00f7add75e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:33.974342 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_59b1f9e575bc474eae696f551b0a3fe4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.974579 1151 consensus_queue.cc:260] T e1c8fe96a6ba4edba4f09c00f7add75e P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.974785 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d822d1db56984a5aad5d65cfe4cb83ae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.974862 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3262226ac4614f40a4aff15dd0426387"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.975106 1151 ts_tablet_manager.cc:1428] T e1c8fe96a6ba4edba4f09c00f7add75e P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250901 01:52:33.975193 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1093c6a2ef5f42ea84dcabc8c60a6840"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.975392 1151 tablet_bootstrap.cc:492] T b2b172757e50404aae78e818921a7d9d P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:33.975468 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_10a87dd7f4074942bda452341fecfd54"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.975606 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_be35a735cba448ff86857ceab56bb1ae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.975875 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b67121820c7540fca4ed48cb3a5f439c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.976083 1483 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_36f5f8854ed14d1ea48eb18865d00659" } modify_external_catalogs: true
I20250901 01:52:33.976228 1483 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_36f5f8854ed14d1ea48eb18865d00659" } modify_external_catalogs: true
I20250901 01:52:33.977429 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c812b385bd0d441ab45c615e843bc011"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.978204 1151 tablet_bootstrap.cc:654] T b2b172757e50404aae78e818921a7d9d P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:33.980454 2113 log.cc:1199] T 186ec810d02e4748aad15228b5866ac1 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/186ec810d02e4748aad15228b5866ac1
I20250901 01:52:33.980715 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f075335e64f844dbb0fa3e24789098df"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:33.980897 2113 ts_tablet_manager.cc:1939] T 186ec810d02e4748aad15228b5866ac1 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:33.982403 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 290) took 4956 ms. Trace:
I20250901 01:52:33.982472 2113 rpcz_store.cc:276] 0901 01:52:29.026180 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:29.026234 (+ 54us) service_pool.cc:225] Handling call
0901 01:52:33.945707 (+4919473us) ts_tablet_manager.cc:1145] Deleting tablet 186ec810d02e4748aad15228b5866ac1
0901 01:52:33.945717 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:33.952040 (+ 6323us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:33.954988 (+ 2948us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:33.982387 (+ 27399us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4919282}
I20250901 01:52:33.982865 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 186ec810d02e4748aad15228b5866ac1 (table default.Table_9d0d7a93ff15424092012d4d681b8ae0 [id=aad143eea5c2439299aad87723399c58]) successfully deleted
I20250901 01:52:33.984494 1482 master_service.cc:948] ReplaceTablet: received request to replace tablet 629db80a8ab444e582628578155638fd from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:33.985533 1488 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet be8dfbc466564842a2fdb4953eee939b
I20250901 01:52:33.985915 2113 tablet_replica.cc:331] T 3a8231590ea643b0a1f83eec4d9a8191 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:33.986119 2113 raft_consensus.cc:2241] T 3a8231590ea643b0a1f83eec4d9a8191 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:33.986445 2113 raft_consensus.cc:2270] T 3a8231590ea643b0a1f83eec4d9a8191 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:33.986954 2113 ts_tablet_manager.cc:1905] T 3a8231590ea643b0a1f83eec4d9a8191 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:33.995005 1055 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e4009963cfe42d6ba164fc828897f09" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58"
I20250901 01:52:33.995215 2778 tablet.cc:1722] T 482a9a2cb81f448bb2823892e35920d4 P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:33.995813 2781 raft_consensus.cc:491] T c6f7c525f59b49938b697de98d4b5835 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:33.995954 2781 raft_consensus.cc:513] T c6f7c525f59b49938b697de98d4b5835 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:33.996620 2781 leader_election.cc:290] T c6f7c525f59b49938b697de98d4b5835 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:33.995216 1055 raft_consensus.cc:3058] T 0e4009963cfe42d6ba164fc828897f09 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:33.999358 2829 tablet.cc:1722] T 482a9a2cb81f448bb2823892e35920d4 P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:34.003398 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c6f7c525f59b49938b697de98d4b5835" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:34.003616 914 raft_consensus.cc:2466] T c6f7c525f59b49938b697de98d4b5835 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:34.006317 992 leader_election.cc:304] T c6f7c525f59b49938b697de98d4b5835 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:34.007818 2753 raft_consensus.cc:491] T 438ae684af234c868a419fbab43b0b98 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:34.007966 2753 raft_consensus.cc:513] T 438ae684af234c868a419fbab43b0b98 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:34.008672 2753 leader_election.cc:290] T 438ae684af234c868a419fbab43b0b98 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:34.009049 2780 raft_consensus.cc:2802] T c6f7c525f59b49938b697de98d4b5835 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:34.009145 2780 raft_consensus.cc:491] T c6f7c525f59b49938b697de98d4b5835 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:34.009213 2780 raft_consensus.cc:3058] T c6f7c525f59b49938b697de98d4b5835 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
W20250901 01:52:34.010690 858 leader_election.cc:336] T 438ae684af234c868a419fbab43b0b98 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:34.012035 2780 raft_consensus.cc:513] T c6f7c525f59b49938b697de98d4b5835 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:34.012614 2781 raft_consensus.cc:491] T e1c8fe96a6ba4edba4f09c00f7add75e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:34.012933 2781 raft_consensus.cc:513] T e1c8fe96a6ba4edba4f09c00f7add75e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:34.013521 2781 leader_election.cc:290] T e1c8fe96a6ba4edba4f09c00f7add75e P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:34.013978 2780 leader_election.cc:290] T c6f7c525f59b49938b697de98d4b5835 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:34.014389 992 leader_election.cc:336] T c6f7c525f59b49938b697de98d4b5835 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:34.012367 1055 raft_consensus.cc:2466] T 0e4009963cfe42d6ba164fc828897f09 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c282f779c1477d920f24dc0dc16628 in term 1.
I20250901 01:52:34.017336 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c6f7c525f59b49938b697de98d4b5835" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:34.017551 914 raft_consensus.cc:3058] T c6f7c525f59b49938b697de98d4b5835 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
W20250901 01:52:34.020900 992 leader_election.cc:336] T e1c8fe96a6ba4edba4f09c00f7add75e P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:34.021006 859 leader_election.cc:304] T 0e4009963cfe42d6ba164fc828897f09 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
W20250901 01:52:34.021492 992 leader_election.cc:336] T c6f7c525f59b49938b697de98d4b5835 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:34.021540 2753 raft_consensus.cc:2802] T 0e4009963cfe42d6ba164fc828897f09 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:34.021653 2753 raft_consensus.cc:695] T 0e4009963cfe42d6ba164fc828897f09 P 00c282f779c1477d920f24dc0dc16628 [term 1 LEADER]: Becoming Leader. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Running, Role: LEADER
I20250901 01:52:34.022001 2753 consensus_queue.cc:237] T 0e4009963cfe42d6ba164fc828897f09 P 00c282f779c1477d920f24dc0dc16628 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:34.022907 2118 ts_tablet_manager.cc:1918] T 26c8ed083cea4dc0acca80d00b11b29b P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:34.023011 2118 log.cc:1199] T 26c8ed083cea4dc0acca80d00b11b29b P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/26c8ed083cea4dc0acca80d00b11b29b
I20250901 01:52:34.023097 2113 ts_tablet_manager.cc:1918] T 3a8231590ea643b0a1f83eec4d9a8191 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 01:52:34.023200 2113 log.cc:1199] T 3a8231590ea643b0a1f83eec4d9a8191 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/3a8231590ea643b0a1f83eec4d9a8191
I20250901 01:52:34.023319 2118 ts_tablet_manager.cc:1939] T 26c8ed083cea4dc0acca80d00b11b29b P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:34.023624 2113 ts_tablet_manager.cc:1939] T 3a8231590ea643b0a1f83eec4d9a8191 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:34.024175 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 182) took 5525 ms. Trace:
I20250901 01:52:34.024302 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 291) took 4862 ms. Trace:
I20250901 01:52:34.024253 2118 rpcz_store.cc:276] 0901 01:52:28.498743 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.498803 (+ 60us) service_pool.cc:225] Handling call
0901 01:52:33.937846 (+5439043us) ts_tablet_manager.cc:1145] Deleting tablet 26c8ed083cea4dc0acca80d00b11b29b
0901 01:52:33.937857 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:34.006850 (+ 68993us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:34.022804 (+ 15954us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:34.024155 (+ 1351us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5438881}
I20250901 01:52:34.026937 1055 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "438ae684af234c868a419fbab43b0b98" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:34.027083 2118 tablet_replica.cc:331] T f98d1d8d660443cb8066a547e5d9b77f P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:34.027336 2118 raft_consensus.cc:2241] T f98d1d8d660443cb8066a547e5d9b77f P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
W20250901 01:52:34.028268 859 leader_election.cc:343] T 438ae684af234c868a419fbab43b0b98 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:34.028439 859 leader_election.cc:304] T 438ae684af234c868a419fbab43b0b98 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:34.028465 2118 raft_consensus.cc:2270] T f98d1d8d660443cb8066a547e5d9b77f P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:34.028842 2608 raft_consensus.cc:2747] T 438ae684af234c868a419fbab43b0b98 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:34.029147 2118 ts_tablet_manager.cc:1905] T f98d1d8d660443cb8066a547e5d9b77f P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:34.029887 914 raft_consensus.cc:2466] T c6f7c525f59b49938b697de98d4b5835 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:34.030488 992 leader_election.cc:304] T c6f7c525f59b49938b697de98d4b5835 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:34.030781 2780 raft_consensus.cc:2802] T c6f7c525f59b49938b697de98d4b5835 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:34.030912 2780 raft_consensus.cc:695] T c6f7c525f59b49938b697de98d4b5835 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:34.031252 2780 consensus_queue.cc:237] T c6f7c525f59b49938b697de98d4b5835 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:34.024376 2113 rpcz_store.cc:276] 0901 01:52:29.162095 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:29.162829 (+ 734us) service_pool.cc:225] Handling call
0901 01:52:33.985868 (+4823039us) ts_tablet_manager.cc:1145] Deleting tablet 3a8231590ea643b0a1f83eec4d9a8191
0901 01:52:33.985878 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:34.003910 (+ 18032us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:34.022998 (+ 19088us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:34.024285 (+ 1287us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4822837}
I20250901 01:52:34.037573 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 3a8231590ea643b0a1f83eec4d9a8191 (table default.Table_a4ebd8ab87c842f082da5cd04d1bddac [id=fd6d8e92d6e64a89a7d38aa761376955]) successfully deleted
I20250901 01:52:34.038048 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 293) took 4875 ms. Trace:
I20250901 01:52:34.038422 2113 rpcz_store.cc:276] 0901 01:52:29.162281 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:29.163155 (+ 874us) service_pool.cc:225] Handling call
0901 01:52:34.037930 (+4874775us) ts_tablet_manager.cc:1145] Deleting tablet 438cd35ff3f74c49b908d072560077db
0901 01:52:34.037946 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:34.038031 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4874640}
W20250901 01:52:34.038964 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 438cd35ff3f74c49b908d072560077db because the tablet was not found. No further retry: Not found: Tablet not found: 438cd35ff3f74c49b908d072560077db
I20250901 01:52:34.040195 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 294) took 4877 ms. Trace:
I20250901 01:52:34.040486 2113 rpcz_store.cc:276] 0901 01:52:29.162356 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:29.163309 (+ 953us) service_pool.cc:225] Handling call
0901 01:52:34.040116 (+4876807us) ts_tablet_manager.cc:1145] Deleting tablet 49d30da95b14400b845270122487a2a0
0901 01:52:34.040124 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:34.040181 (+ 57us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4876699}
W20250901 01:52:34.043784 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 49d30da95b14400b845270122487a2a0 because the tablet was not found. No further retry: Not found: Tablet not found: 49d30da95b14400b845270122487a2a0
I20250901 01:52:34.044142 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 295) took 4881 ms. Trace:
I20250901 01:52:34.044433 2113 rpcz_store.cc:276] 0901 01:52:29.162428 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:29.163443 (+ 1015us) service_pool.cc:225] Handling call
0901 01:52:34.044053 (+4880610us) ts_tablet_manager.cc:1145] Deleting tablet a774cc6ae3924d299127fd9ffb01ca57
0901 01:52:34.044061 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:34.044128 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4880473}
W20250901 01:52:34.044930 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet a774cc6ae3924d299127fd9ffb01ca57 because the tablet was not found. No further retry: Not found: Tablet not found: a774cc6ae3924d299127fd9ffb01ca57
I20250901 01:52:34.045305 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 296) took 4882 ms. Trace:
W20250901 01:52:34.045637 992 consensus_peers.cc:489] T ea846f88b3534beebd0001b7f17b5ca9 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250901 01:52:34.047881 992 consensus_peers.cc:489] T 3c1fd0820d8a4995abe027f685da7f7b P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250901 01:52:34.045660 2113 rpcz_store.cc:276] 0901 01:52:29.162560 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:29.163607 (+ 1047us) service_pool.cc:225] Handling call
0901 01:52:34.045192 (+4881585us) ts_tablet_manager.cc:1145] Deleting tablet a7ff996482164776b468cf305989866b
0901 01:52:34.045201 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:34.045289 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4881478}
I20250901 01:52:34.050479 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 297) took 4887 ms. Trace:
W20250901 01:52:34.050570 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet a7ff996482164776b468cf305989866b because the tablet was not found. No further retry: Not found: Tablet not found: a7ff996482164776b468cf305989866b
I20250901 01:52:34.063715 2781 consensus_queue.cc:1035] T 764bfeb2a7d14c40815b5ec1b07bdd62 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250901 01:52:34.071684 992 consensus_peers.cc:489] T 764bfeb2a7d14c40815b5ec1b07bdd62 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:34.075470 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f6ad16b42d2c46debeecdf11a51db32c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.076614 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8f90a825e70d462fadde19b9502f7bc9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.077286 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_04ca9d790d804a6b802e7b1f0f62b3c3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.077432 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_cad47c5528504ed7abccbac0e449d4da"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.078055 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1688c58e48da4b9d9890f1848b2c0ef6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.078051 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5ca31150cab84df2afc0fdf6fb3d8c89"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.078722 1489 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_99f0fb91a6994b2abf5af0c8f701159b" } modify_external_catalogs: true
I20250901 01:52:34.078751 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a4a227a40bd04357adb55ee33aaa8b1a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.078867 1489 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_99f0fb91a6994b2abf5af0c8f701159b" } modify_external_catalogs: true
I20250901 01:52:34.079396 1482 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_3ef97ecacc3042099ec1cc5c3a7a0763" } new_table_name: "default.Table_7c21b68ef2e74136a807c1dd90e17f4e" modify_external_catalogs: true
I20250901 01:52:34.076637 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c45bef040823445d97ee6fc0caf79c22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.081737 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8a860fe74a174393b9865d3f2009c822"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.082435 1488 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_f027ccc695a74d1fb4595d9614f30381" } new_table_name: "default.Table_c903c87140a144d587f6fa78e6d9b11b" modify_external_catalogs: true
I20250901 01:52:34.083243 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f4430093e88c4fa780edd2dc1ed8a5e8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.084679 1485 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_7f273a7efc7c496d8d8f9a49871026f4" } new_table_name: "default.Table_74f5d01b84de4566ae19752b8d366240" modify_external_catalogs: true
I20250901 01:52:34.085520 1490 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_2755f7caf53340afbd230181643366db" } new_table_name: "default.Table_359a26011f684089ac6da2a62e032680" modify_external_catalogs: true
I20250901 01:52:34.088533 2118 ts_tablet_manager.cc:1918] T f98d1d8d660443cb8066a547e5d9b77f P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:34.088673 2118 log.cc:1199] T f98d1d8d660443cb8066a547e5d9b77f P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/f98d1d8d660443cb8066a547e5d9b77f
I20250901 01:52:34.090391 2781 raft_consensus.cc:491] T f195b040a6ae4b7b83a8308ab0a12b37 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:34.090519 2781 raft_consensus.cc:513] T f195b040a6ae4b7b83a8308ab0a12b37 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:34.091069 2781 leader_election.cc:290] T f195b040a6ae4b7b83a8308ab0a12b37 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:34.093688 2118 ts_tablet_manager.cc:1939] T f98d1d8d660443cb8066a547e5d9b77f P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:34.094040 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 183) took 5594 ms. Trace:
I20250901 01:52:34.094131 2118 rpcz_store.cc:276] 0901 01:52:28.499598 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.499658 (+ 60us) service_pool.cc:225] Handling call
0901 01:52:34.027014 (+5527356us) ts_tablet_manager.cc:1145] Deleting tablet f98d1d8d660443cb8066a547e5d9b77f
0901 01:52:34.027024 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:34.063574 (+ 36550us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:34.088359 (+ 24785us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:34.094024 (+ 5665us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5527189}
W20250901 01:52:34.094241 992 leader_election.cc:336] T f195b040a6ae4b7b83a8308ab0a12b37 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:34.096489 2118 tablet_replica.cc:331] T 50472a7aa2f04e0fbd65ac6c86c34b3e P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:34.096712 2118 raft_consensus.cc:2241] T 50472a7aa2f04e0fbd65ac6c86c34b3e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:34.096879 2118 raft_consensus.cc:2270] T 50472a7aa2f04e0fbd65ac6c86c34b3e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:34.102479 917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f195b040a6ae4b7b83a8308ab0a12b37" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:34.102769 917 raft_consensus.cc:2466] T f195b040a6ae4b7b83a8308ab0a12b37 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:34.103448 992 leader_election.cc:304] T f195b040a6ae4b7b83a8308ab0a12b37 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:34.103818 2781 raft_consensus.cc:2802] T f195b040a6ae4b7b83a8308ab0a12b37 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:34.103899 2781 raft_consensus.cc:491] T f195b040a6ae4b7b83a8308ab0a12b37 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:34.103964 2781 raft_consensus.cc:3058] T f195b040a6ae4b7b83a8308ab0a12b37 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:34.104681 2118 ts_tablet_manager.cc:1905] T 50472a7aa2f04e0fbd65ac6c86c34b3e P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:34.106503 2781 raft_consensus.cc:513] T f195b040a6ae4b7b83a8308ab0a12b37 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:34.107172 2781 leader_election.cc:290] T f195b040a6ae4b7b83a8308ab0a12b37 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:34.111928 992 leader_election.cc:336] T f195b040a6ae4b7b83a8308ab0a12b37 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:34.116906 917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f195b040a6ae4b7b83a8308ab0a12b37" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:34.117187 917 raft_consensus.cc:3058] T f195b040a6ae4b7b83a8308ab0a12b37 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:34.120154 2781 raft_consensus.cc:491] T 36e10328c34242ebb4dbd8f294162696 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:34.120416 2781 raft_consensus.cc:513] T 36e10328c34242ebb4dbd8f294162696 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:34.121150 2781 leader_election.cc:290] T 36e10328c34242ebb4dbd8f294162696 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:34.121425 1472 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.62:43871, user_credentials={real_user=slave}} blocked reactor thread for 142356us
I20250901 01:52:34.121899 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 26c8ed083cea4dc0acca80d00b11b29b (table default.Table_5a32379f33204b9e9b2dfe82995134dc [id=5e18d8cdb4fd4399b1a9f8518676f08f]) successfully deleted
I20250901 01:52:34.122579 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet f98d1d8d660443cb8066a547e5d9b77f (table default.Table_583918346a424cb293d2c35bfa4a5b5f [id=1f677c66803b4cb1bf585cb621a4df6e]) successfully deleted
W20250901 01:52:34.122771 992 leader_election.cc:336] T 36e10328c34242ebb4dbd8f294162696 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:34.122958 992 leader_election.cc:343] T 36e10328c34242ebb4dbd8f294162696 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 36e10328c34242ebb4dbd8f294162696
I20250901 01:52:34.123027 992 leader_election.cc:304] T 36e10328c34242ebb4dbd8f294162696 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:34.123354 2781 raft_consensus.cc:2747] T 36e10328c34242ebb4dbd8f294162696 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:34.123445 2118 ts_tablet_manager.cc:1918] T 50472a7aa2f04e0fbd65ac6c86c34b3e P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:34.123584 2118 log.cc:1199] T 50472a7aa2f04e0fbd65ac6c86c34b3e P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/50472a7aa2f04e0fbd65ac6c86c34b3e
I20250901 01:52:34.123908 2118 ts_tablet_manager.cc:1939] T 50472a7aa2f04e0fbd65ac6c86c34b3e P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
W20250901 01:52:34.124120 2846 log.cc:927] Time spent T 0e4009963cfe42d6ba164fc828897f09 P 00c282f779c1477d920f24dc0dc16628: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20250901 01:52:34.124388 2113 rpcz_store.cc:276] 0901 01:52:29.162657 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:29.163737 (+ 1080us) service_pool.cc:225] Handling call
0901 01:52:34.050322 (+4886585us) ts_tablet_manager.cc:1145] Deleting tablet a54af99bc10045e8b0b6868e5aed98b4
0901 01:52:34.050331 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:34.050459 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4886460}
I20250901 01:52:34.124842 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 298) took 4962 ms. Trace:
I20250901 01:52:34.124910 2113 rpcz_store.cc:276] 0901 01:52:29.162750 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:29.163886 (+ 1136us) service_pool.cc:225] Handling call
0901 01:52:34.124729 (+4960843us) ts_tablet_manager.cc:1145] Deleting tablet c95798f0daf4437db18a291c4464522e
0901 01:52:34.124741 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:34.124826 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4960724}
I20250901 01:52:34.125144 2113 tablet_replica.cc:331] T e6c10426f60f43e8bde69d9b1b9c8eb6 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:34.125322 2113 raft_consensus.cc:2241] T e6c10426f60f43e8bde69d9b1b9c8eb6 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:34.125710 2113 raft_consensus.cc:2270] T e6c10426f60f43e8bde69d9b1b9c8eb6 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250901 01:52:34.129199 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet a54af99bc10045e8b0b6868e5aed98b4 because the tablet was not found. No further retry: Not found: Tablet not found: a54af99bc10045e8b0b6868e5aed98b4
W20250901 01:52:34.129464 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet c95798f0daf4437db18a291c4464522e because the tablet was not found. No further retry: Not found: Tablet not found: c95798f0daf4437db18a291c4464522e
I20250901 01:52:34.130110 2113 ts_tablet_manager.cc:1905] T e6c10426f60f43e8bde69d9b1b9c8eb6 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:34.130695 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 184) took 5630 ms. Trace:
I20250901 01:52:34.130793 2118 rpcz_store.cc:276] 0901 01:52:28.500316 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.500379 (+ 63us) service_pool.cc:225] Handling call
0901 01:52:34.096443 (+5596064us) ts_tablet_manager.cc:1145] Deleting tablet 50472a7aa2f04e0fbd65ac6c86c34b3e
0901 01:52:34.096453 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:34.112071 (+ 15618us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:34.123311 (+ 11240us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:34.130672 (+ 7361us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5595851}
I20250901 01:52:34.131327 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 178) took 5634 ms. Trace:
I20250901 01:52:34.131417 2118 rpcz_store.cc:276] 0901 01:52:28.496770 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.500470 (+ 3700us) service_pool.cc:225] Handling call
0901 01:52:34.131233 (+5630763us) ts_tablet_manager.cc:1145] Deleting tablet c763d252598745a088344aa0eb5fa3d3
0901 01:52:34.131242 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:34.131309 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5630596}
I20250901 01:52:34.131832 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 50472a7aa2f04e0fbd65ac6c86c34b3e (table default.Table_489b87a28dac48d093c241f301d5cccc [id=5fdd352d5e5e47a3a2bd8fd608a263dd]) successfully deleted
W20250901 01:52:34.132144 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet c763d252598745a088344aa0eb5fa3d3 because the tablet was not found. No further retry: Not found: Tablet not found: c763d252598745a088344aa0eb5fa3d3
I20250901 01:52:34.134194 1484 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 482a9a2cb81f448bb2823892e35920d4
W20250901 01:52:34.134407 1484 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 482a9a2cb81f448bb2823892e35920d4: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:34.134804 1484 catalog_manager.cc:6652] ReplaceTablet: tablet 482a9a2cb81f448bb2823892e35920d4 deleted and replaced by tablet 01deaae6abb141789f0cd240f3058c73
I20250901 01:52:34.135191 1484 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_76438e5616384140b57dafdc896f11c5" } new_table_name: "default.Table_cebf7994eb8541e7ab9957cc012cb328" modify_external_catalogs: true
I20250901 01:52:34.135363 1483 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet bc826f7cb9d24fc59640d2ff794bb50a
I20250901 01:52:34.135511 1492 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: default.Table_69575b1a9efd4573a2ee01d3f432800b [id=76928d8caa2f4134a014656cf2cc871a] alter complete (version 2)
I20250901 01:52:34.135708 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ffa58c89a8494fc4a01c7edf02953dce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.136494 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_28b013e5e46c4044ac75ab93edda25a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.136623 1489 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 5adf6b492640417ca071c94702b48cb9
W20250901 01:52:34.136829 1489 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 5adf6b492640417ca071c94702b48cb9: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:34.137178 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_6e68dc0b78474bfe93929c30f48f2f8a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.137360 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b2e6e5fb70f54f0295c1f13e1cd19bda"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.137809 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_10e1021a1cf14b009dc3ca74aa4d2d08"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.138136 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_206d170e53d346159d3a1aaa061d3c9e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.138417 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4aacab70ffac40e69619572e6408a32f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.138531 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8eb2ea7d3997497090c4a41f503c1e94"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.138813 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1a16194d94f74e728dd14000d9d65dac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.139025 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_af22652e98fc44d1954f8c87bd827c62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.139217 1490 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_03fb48eaab9b42858ab7cefb722e43a8" } modify_external_catalogs: true
I20250901 01:52:34.139402 1490 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_03fb48eaab9b42858ab7cefb722e43a8" } modify_external_catalogs: true
W20250901 01:52:34.139655 1485 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 26e3ad68393547e5b6e3dd6b9ec1dbbe consensus state has no leader
W20250901 01:52:34.140666 1482 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 494b6488aa0d4355b4caa46004267465 has no consensus state
I20250901 01:52:34.143180 2608 raft_consensus.cc:491] T d000ba05e12843b4925e622ab55471c0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:34.143323 2608 raft_consensus.cc:513] T d000ba05e12843b4925e622ab55471c0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:34.143970 2608 leader_election.cc:290] T d000ba05e12843b4925e622ab55471c0 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:34.145941 1055 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d000ba05e12843b4925e622ab55471c0" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:34.146136 1055 raft_consensus.cc:2466] T d000ba05e12843b4925e622ab55471c0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00c282f779c1477d920f24dc0dc16628 in term 0.
I20250901 01:52:34.146876 1491 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 81ce448b148447ddbc41875d863e0518
W20250901 01:52:34.147068 1491 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 81ce448b148447ddbc41875d863e0518: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:34.147138 859 leader_election.cc:304] T d000ba05e12843b4925e622ab55471c0 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
W20250901 01:52:34.147975 858 leader_election.cc:336] T d000ba05e12843b4925e622ab55471c0 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:34.148347 2608 raft_consensus.cc:2802] T d000ba05e12843b4925e622ab55471c0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:34.148447 2608 raft_consensus.cc:491] T d000ba05e12843b4925e622ab55471c0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:34.148510 2608 raft_consensus.cc:3058] T d000ba05e12843b4925e622ab55471c0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:34.150293 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 482a9a2cb81f448bb2823892e35920d4 with delete_type TABLET_DATA_DELETED (replaced by tablet 01deaae6abb141789f0cd240f3058c73) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:34.150523 1035 tablet_service.cc:1515] Processing DeleteTablet for tablet 5adf6b492640417ca071c94702b48cb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:34 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:34.151109 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 81ce448b148447ddbc41875d863e0518 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:33 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:34.151225 2608 raft_consensus.cc:513] T d000ba05e12843b4925e622ab55471c0 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:34.151991 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_0127cbebd67a4297a3f411c53173d7fb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.152961 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f6de45f3b78b48798515fc3d57000aa6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:34.153497 992 consensus_peers.cc:489] T dabebc77f72e4bb19e264be2f496202a P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:34.154017 2608 leader_election.cc:290] T d000ba05e12843b4925e622ab55471c0 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:34.154268 858 leader_election.cc:336] T d000ba05e12843b4925e622ab55471c0 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:34.154621 1055 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d000ba05e12843b4925e622ab55471c0" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58"
I20250901 01:52:34.154807 1055 raft_consensus.cc:3058] T d000ba05e12843b4925e622ab55471c0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:34.155997 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_44fcaafa256e4dcaaf63516b1c1239f2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.156797 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_14cfbd47ae0045a0a80deb2e18e1318f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.157485 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7dcf39a51b8a4fc0b52a0c5cf4325f20"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.157558 1055 raft_consensus.cc:2466] T d000ba05e12843b4925e622ab55471c0 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c282f779c1477d920f24dc0dc16628 in term 1.
I20250901 01:52:34.158234 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_86de3443c2b0430c91a8b40e55b74c3d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.158468 859 leader_election.cc:304] T d000ba05e12843b4925e622ab55471c0 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:34.159166 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7e91b8511a5e43f98a8dd9d1f8be2258"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.159909 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_256f079d7474412d9fe5d1230f8f9fd4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.160660 914 raft_consensus.cc:1273] T dabebc77f72e4bb19e264be2f496202a P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Refusing update from remote peer fd27ab6665304aaea53adbd102dd4f58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250901 01:52:34.160657 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e71b3f992e414dd99586f302a5da6f62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.162123 2780 consensus_queue.cc:1035] T dabebc77f72e4bb19e264be2f496202a P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250901 01:52:34.162712 992 consensus_peers.cc:489] T 3eba54e5ddd14c518f4cbd539268ae37 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:34.165190 2357 raft_consensus.cc:2802] T d000ba05e12843b4925e622ab55471c0 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:34.165308 2357 raft_consensus.cc:695] T d000ba05e12843b4925e622ab55471c0 P 00c282f779c1477d920f24dc0dc16628 [term 1 LEADER]: Becoming Leader. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Running, Role: LEADER
I20250901 01:52:34.165577 2357 consensus_queue.cc:237] T d000ba05e12843b4925e622ab55471c0 P 00c282f779c1477d920f24dc0dc16628 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:34.167946 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_95749b4f74084576bb174fbb998bfc06"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.169013 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a7189238a43a4f7db9ab2d43c978002e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.170110 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a513076960164afe978f4ac3d87118f0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:34.171510 992 consensus_peers.cc:489] T f13b972e25214f168421439fe3943e8a P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250901 01:52:34.172556 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 482a9a2cb81f448bb2823892e35920d4 with delete_type TABLET_DATA_DELETED (replaced by tablet 01deaae6abb141789f0cd240f3058c73) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:34.177227 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ec859425e61f4b9b94a480233cf90e46"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.177613 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 5adf6b492640417ca071c94702b48cb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:34 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:34.178653 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_31ff7e2ec2774a0f9791045563c0b7de"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.179378 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_16a467c89fa345bab392cbdf247cd03d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.180312 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_822dea1afd97447b9991c6152ac7fd7c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.181241 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_eb98b1ad3e5746d4bd9e490055ab3240"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.182171 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9df21239c4ee461da4bdda4b532b3e02"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.183095 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8b82e217ba274db6aa1a3c61d879dfd2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.184015 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_0b945453ae6e4bacbc85b5f5a78896d0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.184958 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f6bf6fa35c7f45a886930b62b95191a8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.186394 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 81ce448b148447ddbc41875d863e0518 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:33 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:34.188642 921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e1c8fe96a6ba4edba4f09c00f7add75e" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:34.189213 921 raft_consensus.cc:2466] T e1c8fe96a6ba4edba4f09c00f7add75e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:34.194918 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e7cbd002162e4a86ae592413dda7372c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.195966 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e6daa6d731df4984a91ba5a0ef5c6d45"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.196807 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 186) took 5695 ms. Trace:
I20250901 01:52:34.196983 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_aa1e74c6935d4239928a91601ef11352"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.196954 2118 rpcz_store.cc:276] 0901 01:52:28.501727 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.502125 (+ 398us) service_pool.cc:225] Handling call
0901 01:52:34.196670 (+5694545us) ts_tablet_manager.cc:1145] Deleting tablet 20f7bfab18134e079e91c52c89468cc3
0901 01:52:34.196683 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:34.196784 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5694388}
I20250901 01:52:34.197445 2118 tablet_replica.cc:331] T 56f2c4d2e6ab4bb4bdd6d025843589b6 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:34.197649 2118 raft_consensus.cc:2241] T 56f2c4d2e6ab4bb4bdd6d025843589b6 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:34.197835 2118 raft_consensus.cc:2270] T 56f2c4d2e6ab4bb4bdd6d025843589b6 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:34.198096 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_488572eb507a45e78aef099234568c1b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.198832 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_df8dce2e70c045f8bb54fe1f2dbf4800"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.203634 917 raft_consensus.cc:2466] T f195b040a6ae4b7b83a8308ab0a12b37 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:34.205246 1151 tablet_bootstrap.cc:492] T b2b172757e50404aae78e818921a7d9d P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:34.205224 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8e485565e94e479d9f25be51205fe0df"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.205379 1151 ts_tablet_manager.cc:1397] T b2b172757e50404aae78e818921a7d9d P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.230s user 0.028s sys 0.024s
I20250901 01:52:34.206055 1151 raft_consensus.cc:357] T b2b172757e50404aae78e818921a7d9d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:34.206259 1151 raft_consensus.cc:383] T b2b172757e50404aae78e818921a7d9d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:34.206323 1151 raft_consensus.cc:738] T b2b172757e50404aae78e818921a7d9d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:34.206362 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_57717f0f567646b8b1f413d4d5d77086"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.206523 1151 consensus_queue.cc:260] T b2b172757e50404aae78e818921a7d9d P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:34.206691 2118 ts_tablet_manager.cc:1905] T 56f2c4d2e6ab4bb4bdd6d025843589b6 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:34.207160 1151 ts_tablet_manager.cc:1428] T b2b172757e50404aae78e818921a7d9d P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250901 01:52:34.207410 1151 tablet_bootstrap.cc:492] T 3363fd67b1ad44849ecb7b4090b61a72 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:34.208963 1151 tablet_bootstrap.cc:654] T 3363fd67b1ad44849ecb7b4090b61a72 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:34.209034 1482 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_d8e839ac6396439197b8b727af6a31b7" } modify_external_catalogs: true
I20250901 01:52:34.209255 1482 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_d8e839ac6396439197b8b727af6a31b7" } modify_external_catalogs: true
I20250901 01:52:34.210012 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_40f45ce3ad774d3f827ddf54886f9b0f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.197713 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_35f36293307446ccaa164c82d9b600fe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.213785 992 leader_election.cc:304] T e1c8fe96a6ba4edba4f09c00f7add75e P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:34.214242 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2ec3cbac0eda48cabe46e24d82de61fb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.214694 2778 tablet.cc:1722] T dabebc77f72e4bb19e264be2f496202a P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250901 01:52:34.216599 2780 raft_consensus.cc:2802] T e1c8fe96a6ba4edba4f09c00f7add75e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:34.216683 2780 raft_consensus.cc:491] T e1c8fe96a6ba4edba4f09c00f7add75e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:34.216745 2780 raft_consensus.cc:3058] T e1c8fe96a6ba4edba4f09c00f7add75e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:34.217562 1485 master_service.cc:948] ReplaceTablet: received request to replace tablet eee34044c39d4013baf78d3705489626 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:34.217967 992 leader_election.cc:304] T f195b040a6ae4b7b83a8308ab0a12b37 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:34.219488 2780 raft_consensus.cc:513] T e1c8fe96a6ba4edba4f09c00f7add75e P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:34.219918 2775 raft_consensus.cc:2802] T f195b040a6ae4b7b83a8308ab0a12b37 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:34.220037 2775 raft_consensus.cc:695] T f195b040a6ae4b7b83a8308ab0a12b37 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:34.220263 2775 consensus_queue.cc:237] T f195b040a6ae4b7b83a8308ab0a12b37 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:34.222831 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 20f7bfab18134e079e91c52c89468cc3 because the tablet was not found. No further retry: Not found: Tablet not found: 20f7bfab18134e079e91c52c89468cc3
I20250901 01:52:34.224310 2753 consensus_queue.cc:1035] T 858218c4427c41e980b2b5c969a5af40 P 00c282f779c1477d920f24dc0dc16628 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:34.227663 2780 leader_election.cc:290] T e1c8fe96a6ba4edba4f09c00f7add75e P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:34.234462 1490 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet a6b539e160194da3a849d9e2f1ec9859
I20250901 01:52:34.234720 1486 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: default.Table_d8e839ac6396439197b8b727af6a31b7 [id=55c31cd42fac43e6b8eddc048f565d0a] alter complete (version 2)
I20250901 01:52:34.235184 1492 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: default.Table_62eb2015c3dc4feda347dae9fb970abd [id=18aecbf3fe4643c4b589ee47706161bd] alter complete (version 5)
E20250901 01:52:34.237139 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 0c88c339e59a4c1da21756cba0645bdd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0927e3ea788d43bda5afeda483d87176'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.237335 1696 catalog_manager.cc:6205] Error selecting replicas for tablet a656adbf1b29431687e0989f0db3459c, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2d3d2227e6a24bd2a86f457e553e698b'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.237447 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 494b6488aa0d4355b4caa46004267465, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7c21b68ef2e74136a807c1dd90e17f4e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.237608 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 0ee3f35392954817b350e5f48653be78, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_442e11b9211c4b209c68f653d7bfc313'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.237735 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae4844183a3c4a04a775e6fd878391a3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_aee74198b51d4f089538c4f2bf3ce4dc'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.237869 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9745582cebfe47b58c21d0191ab4cb9b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77728af9eee44f29a1e2131d5b75b7ca'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.237998 1696 catalog_manager.cc:6205] Error selecting replicas for tablet e2ed32b3292141278af28026528e96c2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b5077f55951d48be92a67407ffe26a08'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.238121 1696 catalog_manager.cc:6205] Error selecting replicas for tablet eea6bfabe6cc4f83905c24f0f7a5dfc9, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_e0902dfbdd984f85b3fdac8d283731f7'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.238236 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9f62a9d97f514747b940b488907369f3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d53a456dbca64daf92aeb5fb4517b417'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.238333 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5a7fa68d29644ae8afd2711e39dde59a, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d24e3a18ef4843fdb7b74ce23d47cf9e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.238415 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 415c9d7b552b42c384b444a20491f65c, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_97c44ea002094df8a6af3dccd0099c39'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.238539 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ff2bd017417c41b5b8225b8f00283e63, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0c7355975d6b44dbbe3a77df942b4d43'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.238624 1696 catalog_manager.cc:6205] Error selecting replicas for tablet dcc60bb000f440f1aca9133928b5ce20, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2fbca8463d6546c9b8f5b018fc60251d'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.238729 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 465c02a130c54e09b5a74fb4dd2ef762, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ed6fd284be6f4ae3a008e21e7ce17930'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.238813 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f8f923ddcd2740bcb7e521f4f57c4034, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_5e65c5f068b84995a2d7a76fa65dda0f'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.239104 1696 catalog_manager.cc:6205] Error selecting replicas for tablet b1ba0e19dc0b462e80c537bf2d183f5b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_66b111271c15410aab23682f8f6081ac'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.239213 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 8e10a01e77d14cf28c2c5e0205e595b9, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_83cf2d86a1284544ba2e7aa9ebb66bb7'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.239323 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5f0e054f62de4cf6bc6f69ad6664b192, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ef6832d0845044ed9f165ca3541b4c3f'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.239409 1696 catalog_manager.cc:6205] Error selecting replicas for tablet fd2ce82a4ee24d0faa44980a3967e187, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_63d0f2ead9d946d39ce40162ebd769d6'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.239602 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae9a4ad4fdfb4771ba23a91f7007447e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_5e84ae9a9c2a4084a1f9c5ffb4594de2'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.239728 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 086aa948f93644da8e5835b0fac87ddd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b4fbc34da806475c912052d0fab3fc53'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.239991 1696 catalog_manager.cc:6205] Error selecting replicas for tablet d602c8986c75492ead9936ac04e2622d, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_82d91a1ba2eb4265896824028dec8d3a'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.240084 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 93aa22afc0ad4d7fa24807ea3c5d5c21, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0f4b4094d2974b7591875375eda6bed0'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.240182 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9074543dea9949e7a085717b40913348, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_53e30205c5fd488eb3b48c685a159d04'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.240257 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 3a62034e688348ed9e2ac9de009ed3a2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_e225c82a2d6045f6b16915cdb72f5734'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.240345 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 18498c9cb09141d08c1c5c0854b63ebd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77f4810bc83c4d948531da2a45a3792e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.240440 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f5e3316adeae47cd848833dd84b972bb, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7570c3012f8a4ae7bcfc7346eadac92a'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.240550 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 6ce9521b925b4f97bb4a810ce77e147e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_9e04f976a93f4d30baccf90fa3622dfe'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.240654 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2f61edb50d7a42198bfd70639c7bf4de, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_a5c783cbdb694af69047b31d99ff8def'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:34.242805 2829 tablet.cc:1722] T dabebc77f72e4bb19e264be2f496202a P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250901 01:52:34.248868 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d5336e69e69e43a1853796bfe2c3b00e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.249737 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4978d0008900400fa3006645898a1323"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.250113 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_cf35d0e381594cf8b3f43e1bca3776c6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.250958 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_56466f40479645d48ffed9cad9cba5a8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.252218 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f14a207ea7aa435da2459823b8866d91"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.253212 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_625a1677129047a9b1ddc761e57c3011"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.254252 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_af89666c6f6d49e883db056be97a8bbe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.254369 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_6a6fb0b83df04f50b21f2678e92e5948"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.255571 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_036133d4567c4d6db0f0c33bcb08a12d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.256613 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ccabfbcffe574cca91d16b93fdae4937"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.259900 1482 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet aec951f2cc84499b9162b9fa24d5440e
W20250901 01:52:34.260104 1482 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet aec951f2cc84499b9162b9fa24d5440e: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:34.261396 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet aec951f2cc84499b9162b9fa24d5440e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:34 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:34.261668 886 tablet_service.cc:1515] Processing DeleteTablet for tablet aec951f2cc84499b9162b9fa24d5440e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:34 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:34.259546 1486 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: default.Table_9f86b15283e34dd88287f72985404935 [id=b3914f33e856466e8479d0a2263ec56f] alter complete (version 2)
E20250901 01:52:34.267813 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 0c88c339e59a4c1da21756cba0645bdd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0927e3ea788d43bda5afeda483d87176'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.267959 1696 catalog_manager.cc:6205] Error selecting replicas for tablet a656adbf1b29431687e0989f0db3459c, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2d3d2227e6a24bd2a86f457e553e698b'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.268074 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 494b6488aa0d4355b4caa46004267465, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7c21b68ef2e74136a807c1dd90e17f4e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.268172 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 01deaae6abb141789f0cd240f3058c73, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7baa92b46d98454a891222be49a6ca48'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.268282 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 0ee3f35392954817b350e5f48653be78, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_442e11b9211c4b209c68f653d7bfc313'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:34.271365 2780 raft_consensus.cc:491] T b4278a013ae84082bc7a8497b04ac147 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:34.271499 2780 raft_consensus.cc:513] T b4278a013ae84082bc7a8497b04ac147 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:34.272014 2780 leader_election.cc:290] T b4278a013ae84082bc7a8497b04ac147 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:34.273429 918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b4278a013ae84082bc7a8497b04ac147" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:34.273614 918 raft_consensus.cc:2466] T b4278a013ae84082bc7a8497b04ac147 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
W20250901 01:52:34.273301 992 leader_election.cc:336] T e1c8fe96a6ba4edba4f09c00f7add75e P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:34.274338 992 consensus_peers.cc:489] T 268439ff8d3a4f7a89f3d6395f2079c5 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250901 01:52:34.274973 992 leader_election.cc:336] T b4278a013ae84082bc7a8497b04ac147 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:34.275175 992 leader_election.cc:304] T b4278a013ae84082bc7a8497b04ac147 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:34.275379 2775 raft_consensus.cc:2802] T b4278a013ae84082bc7a8497b04ac147 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:34.275458 2775 raft_consensus.cc:491] T b4278a013ae84082bc7a8497b04ac147 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:34.275520 2775 raft_consensus.cc:3058] T b4278a013ae84082bc7a8497b04ac147 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:34.276741 2829 tablet.cc:1722] T 71941b815d6840478c5acc1cbb187cc9 P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:34.277339 1492 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: default.Table_cca52da43da6439eb88ea9715e4a0d50 [id=f0dfb3811e744bd4ba6d59195cebde6f] alter complete (version 3)
I20250901 01:52:34.281922 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 4fca1435d82447f289fda4035aa2ed89 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:20 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:34.282128 1035 tablet_service.cc:1515] Processing DeleteTablet for tablet 1495de70f0844f82a0eea99c059f076b with delete_type TABLET_DATA_DELETED (replaced by tablet f5e3316adeae47cd848833dd84b972bb) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:34.282279 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 8b9d79a25452410891c09fd3f7c4be6d with delete_type TABLET_DATA_DELETED (replaced by tablet 0ee3f35392954817b350e5f48653be78) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:34.282420 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet f98d1d8d660443cb8066a547e5d9b77f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:23 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:34.284632 1492 master_service.cc:948] ReplaceTablet: received request to replace tablet 508406601c994e42a5c5b1f665e74957 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:34.285176 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e552d571c59546df8fa381ea98096c80"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.286279 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_93266ddb3625405da68bd8212aaa2af6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.287171 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_bf3b38b6f7a5494e8163227be8dc1f79"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.288661 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_110b8a01246c4fb1bd6cf71d2e869260"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.288698 917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e1c8fe96a6ba4edba4f09c00f7add75e" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:34.288851 917 raft_consensus.cc:3058] T e1c8fe96a6ba4edba4f09c00f7add75e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:34.289551 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_02933335a4774cc1bed19c562f6e87e8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.290298 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_51e1e4ad00444dbfb5c8154f3842d8c1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.290443 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9e9d1c6186b44caea04c858760e6739a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.291157 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_0c5ce92d550c4f9abc2a510c9e8dcf1a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.291312 917 raft_consensus.cc:2466] T e1c8fe96a6ba4edba4f09c00f7add75e P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:34.292481 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_49d9015a8ce64792aca18ecf9c553d8d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.293131 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9f3887bbb06846bd9f64eb4313227280"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.293247 2113 ts_tablet_manager.cc:1918] T e6c10426f60f43e8bde69d9b1b9c8eb6 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 01:52:34.293346 2113 log.cc:1199] T e6c10426f60f43e8bde69d9b1b9c8eb6 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/e6c10426f60f43e8bde69d9b1b9c8eb6
I20250901 01:52:34.293668 2113 ts_tablet_manager.cc:1939] T e6c10426f60f43e8bde69d9b1b9c8eb6 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:34.295188 2775 raft_consensus.cc:513] T b4278a013ae84082bc7a8497b04ac147 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:34.295794 2775 leader_election.cc:290] T b4278a013ae84082bc7a8497b04ac147 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:34.299751 992 leader_election.cc:304] T e1c8fe96a6ba4edba4f09c00f7add75e P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:34.300103 2781 raft_consensus.cc:2802] T e1c8fe96a6ba4edba4f09c00f7add75e P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:34.300230 2781 raft_consensus.cc:695] T e1c8fe96a6ba4edba4f09c00f7add75e P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:34.300469 2781 consensus_queue.cc:237] T e1c8fe96a6ba4edba4f09c00f7add75e P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:34.301355 2780 raft_consensus.cc:491] T fecf7d538b4c4c619182d5eccb44dd8b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:34.301465 2780 raft_consensus.cc:513] T fecf7d538b4c4c619182d5eccb44dd8b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:34.302201 2780 leader_election.cc:290] T fecf7d538b4c4c619182d5eccb44dd8b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:34.304065 917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b4278a013ae84082bc7a8497b04ac147" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:34.304214 917 raft_consensus.cc:3058] T b4278a013ae84082bc7a8497b04ac147 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:34.304996 2778 tablet.cc:1722] T 71941b815d6840478c5acc1cbb187cc9 P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:34.308187 917 raft_consensus.cc:2466] T b4278a013ae84082bc7a8497b04ac147 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
E20250901 01:52:34.268371 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae4844183a3c4a04a775e6fd878391a3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_aee74198b51d4f089538c4f2bf3ce4dc'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.309240 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9745582cebfe47b58c21d0191ab4cb9b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77728af9eee44f29a1e2131d5b75b7ca'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.309376 1696 catalog_manager.cc:6205] Error selecting replicas for tablet e2ed32b3292141278af28026528e96c2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b5077f55951d48be92a67407ffe26a08'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:34.316298 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_574460aeceb142f8b2fb636963520cba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.317479 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_641fb8991ed04438ae7897ca53dcee00"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.318173 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 299) took 4828 ms. Trace:
I20250901 01:52:34.318713 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4f3bd19167ae40549af64bac59b89b0d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.318799 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_da21d0d25c424abca845d07e204132b9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.319656 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_0d0027f445a94658ab717bf122ced687"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.319818 2118 ts_tablet_manager.cc:1918] T 56f2c4d2e6ab4bb4bdd6d025843589b6 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:34.319918 2118 log.cc:1199] T 56f2c4d2e6ab4bb4bdd6d025843589b6 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/56f2c4d2e6ab4bb4bdd6d025843589b6
I20250901 01:52:34.320191 2118 ts_tablet_manager.cc:1939] T 56f2c4d2e6ab4bb4bdd6d025843589b6 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
W20250901 01:52:34.321837 858 consensus_peers.cc:489] T e87d1281f35648b0b115b67c95e244e7 P 00c282f779c1477d920f24dc0dc16628 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250901 01:52:34.319697 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b59157816c3d4edab1eaaccfbbde8382"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.320494 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7f03379e977a494e97726ef2c14e7461"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.321056 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f80a1ebe39f3459ababb51e0b49663c1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.320883 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1765aee2619c4da9b89a17ca6680c713"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:34.321964 1696 catalog_manager.cc:6205] Error selecting replicas for tablet eea6bfabe6cc4f83905c24f0f7a5dfc9, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_e0902dfbdd984f85b3fdac8d283731f7'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:34.321516 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_fff0641b6fff4da18452521d88747a66"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:34.324340 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9f62a9d97f514747b940b488907369f3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d53a456dbca64daf92aeb5fb4517b417'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.324499 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5a7fa68d29644ae8afd2711e39dde59a, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d24e3a18ef4843fdb7b74ce23d47cf9e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.324604 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 415c9d7b552b42c384b444a20491f65c, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_97c44ea002094df8a6af3dccd0099c39'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.324723 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ff2bd017417c41b5b8225b8f00283e63, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0c7355975d6b44dbbe3a77df942b4d43'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.324820 1696 catalog_manager.cc:6205] Error selecting replicas for tablet dcc60bb000f440f1aca9133928b5ce20, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2fbca8463d6546c9b8f5b018fc60251d'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.324921 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 465c02a130c54e09b5a74fb4dd2ef762, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ed6fd284be6f4ae3a008e21e7ce17930'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.325002 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f8f923ddcd2740bcb7e521f4f57c4034, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_5e65c5f068b84995a2d7a76fa65dda0f'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.325093 1696 catalog_manager.cc:6205] Error selecting replicas for tablet b1ba0e19dc0b462e80c537bf2d183f5b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_66b111271c15410aab23682f8f6081ac'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.325170 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 8e10a01e77d14cf28c2c5e0205e595b9, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_83cf2d86a1284544ba2e7aa9ebb66bb7'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.325263 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5f0e054f62de4cf6bc6f69ad6664b192, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ef6832d0845044ed9f165ca3541b4c3f'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.325351 1696 catalog_manager.cc:6205] Error selecting replicas for tablet fd2ce82a4ee24d0faa44980a3967e187, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_63d0f2ead9d946d39ce40162ebd769d6'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.325505 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae9a4ad4fdfb4771ba23a91f7007447e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_5e84ae9a9c2a4084a1f9c5ffb4594de2'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.325609 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 086aa948f93644da8e5835b0fac87ddd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b4fbc34da806475c912052d0fab3fc53'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.325721 1696 catalog_manager.cc:6205] Error selecting replicas for tablet d602c8986c75492ead9936ac04e2622d, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_82d91a1ba2eb4265896824028dec8d3a'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.325821 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 93aa22afc0ad4d7fa24807ea3c5d5c21, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0f4b4094d2974b7591875375eda6bed0'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.325907 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9074543dea9949e7a085717b40913348, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_53e30205c5fd488eb3b48c685a159d04'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.325991 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 3a62034e688348ed9e2ac9de009ed3a2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_e225c82a2d6045f6b16915cdb72f5734'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.326081 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 18498c9cb09141d08c1c5c0854b63ebd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77f4810bc83c4d948531da2a45a3792e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.326159 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f5e3316adeae47cd848833dd84b972bb, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7570c3012f8a4ae7bcfc7346eadac92a'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.326261 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 81e8c6567953484ab000fe2eb4398301, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_a920a3ffbabf4d83a6563c3423cde12a'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.326344 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 6ce9521b925b4f97bb4a810ce77e147e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_9e04f976a93f4d30baccf90fa3622dfe'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.326431 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2f61edb50d7a42198bfd70639c7bf4de, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_a5c783cbdb694af69047b31d99ff8def'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:34.318261 2113 rpcz_store.cc:276] 0901 01:52:29.489673 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:29.490148 (+ 475us) service_pool.cc:225] Handling call
0901 01:52:34.125097 (+4634949us) ts_tablet_manager.cc:1145] Deleting tablet e6c10426f60f43e8bde69d9b1b9c8eb6
0901 01:52:34.125108 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:34.201415 (+ 76307us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:34.293126 (+ 91711us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:34.318152 (+ 25026us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":23775,"tablet-delete.queue_time_us":4634749}
I20250901 01:52:34.327576 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet e6c10426f60f43e8bde69d9b1b9c8eb6 (table default.Table_7c21b68ef2e74136a807c1dd90e17f4e [id=d3dbeb4258424133844e20b181ee44e4]) successfully deleted
I20250901 01:52:34.327924 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 302) took 4837 ms. Trace:
I20250901 01:52:34.328004 2113 rpcz_store.cc:276] 0901 01:52:29.489947 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:29.490433 (+ 486us) service_pool.cc:225] Handling call
0901 01:52:34.327811 (+4837378us) ts_tablet_manager.cc:1145] Deleting tablet b73e362321284df1b18ef755dfe930a5
0901 01:52:34.327823 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:34.327904 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4837199}
W20250901 01:52:34.328375 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet b73e362321284df1b18ef755dfe930a5 because the tablet was not found. No further retry: Not found: Tablet not found: b73e362321284df1b18ef755dfe930a5
I20250901 01:52:34.328697 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 304) took 4838 ms. Trace:
I20250901 01:52:34.328769 2113 rpcz_store.cc:276] 0901 01:52:29.490071 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:29.490778 (+ 707us) service_pool.cc:225] Handling call
0901 01:52:34.328625 (+4837847us) ts_tablet_manager.cc:1145] Deleting tablet b11e8f77243e4764b96548d1132b8e5a
0901 01:52:34.328634 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:34.328683 (+ 49us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4837715}
W20250901 01:52:34.329113 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet b11e8f77243e4764b96548d1132b8e5a because the tablet was not found. No further retry: Not found: Tablet not found: b11e8f77243e4764b96548d1132b8e5a
I20250901 01:52:34.329385 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 309) took 4837 ms. Trace:
I20250901 01:52:34.329444 2113 rpcz_store.cc:276] 0901 01:52:29.491574 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:29.491636 (+ 62us) service_pool.cc:225] Handling call
0901 01:52:34.329320 (+4837684us) ts_tablet_manager.cc:1145] Deleting tablet e4fdbc8d4bff4173a20db2c182acfd9e
0901 01:52:34.329328 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:34.329372 (+ 44us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4837510}
W20250901 01:52:34.329782 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet e4fdbc8d4bff4173a20db2c182acfd9e because the tablet was not found. No further retry: Not found: Tablet not found: e4fdbc8d4bff4173a20db2c182acfd9e
I20250901 01:52:34.330335 992 leader_election.cc:304] T b4278a013ae84082bc7a8497b04ac147 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:34.330809 2775 raft_consensus.cc:2802] T b4278a013ae84082bc7a8497b04ac147 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:34.330999 2780 raft_consensus.cc:491] T 81ce448b148447ddbc41875d863e0518 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:34.332439 2780 raft_consensus.cc:513] T 81ce448b148447ddbc41875d863e0518 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:34.332664 921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fecf7d538b4c4c619182d5eccb44dd8b" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:34.332834 921 raft_consensus.cc:2466] T fecf7d538b4c4c619182d5eccb44dd8b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:34.332938 2780 leader_election.cc:290] T 81ce448b148447ddbc41875d863e0518 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:34.333176 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 187) took 5830 ms. Trace:
I20250901 01:52:34.333242 2118 rpcz_store.cc:276] 0901 01:52:28.502443 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.502494 (+ 51us) service_pool.cc:225] Handling call
0901 01:52:34.197390 (+5694896us) ts_tablet_manager.cc:1145] Deleting tablet 56f2c4d2e6ab4bb4bdd6d025843589b6
0901 01:52:34.197401 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:34.222765 (+ 25364us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:34.319711 (+ 96946us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:34.333163 (+ 13452us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5694755}
I20250901 01:52:34.333637 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 310) took 4842 ms. Trace:
I20250901 01:52:34.333700 2113 rpcz_store.cc:276] 0901 01:52:29.491666 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:29.491840 (+ 174us) service_pool.cc:225] Handling call
0901 01:52:34.333568 (+4841728us) ts_tablet_manager.cc:1145] Deleting tablet e5e250e6c84d4248b857bf1478d47703
0901 01:52:34.333576 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:34.333624 (+ 48us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4841582}
W20250901 01:52:34.334036 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet e5e250e6c84d4248b857bf1478d47703 because the tablet was not found. No further retry: Not found: Tablet not found: e5e250e6c84d4248b857bf1478d47703
W20250901 01:52:34.335016 992 leader_election.cc:336] T b4278a013ae84082bc7a8497b04ac147 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:34.335754 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 188) took 5832 ms. Trace:
I20250901 01:52:34.335830 2118 rpcz_store.cc:276] 0901 01:52:28.503172 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.503217 (+ 45us) service_pool.cc:225] Handling call
0901 01:52:34.335675 (+5832458us) ts_tablet_manager.cc:1145] Deleting tablet 0dbdcb76eeba4fefa56e7835c23c0316
0901 01:52:34.335683 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:34.335739 (+ 56us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5832320}
I20250901 01:52:34.336207 2118 tablet_replica.cc:331] T e1c8fe96a6ba4edba4f09c00f7add75e P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:34.336374 2118 raft_consensus.cc:2241] T e1c8fe96a6ba4edba4f09c00f7add75e P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:34.336517 2118 raft_consensus.cc:2270] T e1c8fe96a6ba4edba4f09c00f7add75e P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:34.337131 2113 tablet_replica.cc:331] T 5f52698fb56d472d9e91fb10cb10c69a P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:34.337299 2113 raft_consensus.cc:2241] T 5f52698fb56d472d9e91fb10cb10c69a P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:34.337563 2113 raft_consensus.cc:2270] T 5f52698fb56d472d9e91fb10cb10c69a P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:34.338027 2113 ts_tablet_manager.cc:1905] T 5f52698fb56d472d9e91fb10cb10c69a P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:34.337978 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e8147bc85c2f4d0c9d9b8a3f93c4b629"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.339923 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_04a4ef2bd1914e78b840c948d2901257"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.340739 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8fbda6feb5154b17a3a0fabf26491e49"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.341500 2118 ts_tablet_manager.cc:1905] T e1c8fe96a6ba4edba4f09c00f7add75e P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:34.341727 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e43e0529ca7a4cae9a128712d0e8c3d9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.342002 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ebf8cf99d37c4b96a0936e04c80c2f0a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.342563 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f6d728a82f56438b9f5f39b16bbce3a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.343750 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_71fb6e67eb3f4b00a3452ba79892ebf7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.344506 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_bae3aad3a7d34312b359f61475ca7293"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.345363 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 56f2c4d2e6ab4bb4bdd6d025843589b6 (table default.Table_c0977985d6ee4b21a086dc892c860d6a [id=f2ff184777494b45a70f9bc4f66a448b]) successfully deleted
W20250901 01:52:34.345669 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 0dbdcb76eeba4fefa56e7835c23c0316 because the tablet was not found. No further retry: Not found: Tablet not found: 0dbdcb76eeba4fefa56e7835c23c0316
I20250901 01:52:34.347594 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_eca5dc818368423bbfaa1c281238433a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.348913 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d50c7924b16e4075a9d5c38d774056fb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.353331 1488 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_6d47266af0ed433eb1f2abbc81d18e69" } modify_external_catalogs: true
I20250901 01:52:34.353574 1488 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_6d47266af0ed433eb1f2abbc81d18e69" } modify_external_catalogs: true
I20250901 01:52:34.353765 2775 raft_consensus.cc:695] T b4278a013ae84082bc7a8497b04ac147 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:34.354068 2775 consensus_queue.cc:237] T b4278a013ae84082bc7a8497b04ac147 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:34.358899 1490 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_a2a8011bb7624d9ea71e7b3169762e43" } new_table_name: "default.Table_ee3fa2db164c4209b58ffaf8102700ba" modify_external_catalogs: true
I20250901 01:52:34.360483 1489 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_5efd7ee978b74c549a9d9a31d6404db7" } new_table_name: "default.Table_2f51253b29ae46f8ae7b33f230aab3ac" modify_external_catalogs: true
I20250901 01:52:34.360925 1484 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_c903c87140a144d587f6fa78e6d9b11b" } modify_external_catalogs: true
I20250901 01:52:34.361145 1484 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_c903c87140a144d587f6fa78e6d9b11b" } modify_external_catalogs: true
I20250901 01:52:34.362010 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_17759de51ad2405da672a56755a7cd57"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.362807 2113 ts_tablet_manager.cc:1918] T 5f52698fb56d472d9e91fb10cb10c69a P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:34.362911 2113 log.cc:1199] T 5f52698fb56d472d9e91fb10cb10c69a P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/5f52698fb56d472d9e91fb10cb10c69a
I20250901 01:52:34.363215 2113 ts_tablet_manager.cc:1939] T 5f52698fb56d472d9e91fb10cb10c69a P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:34.363916 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 300) took 4874 ms. Trace:
I20250901 01:52:34.363991 2113 rpcz_store.cc:276] 0901 01:52:29.489803 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:29.490382 (+ 579us) service_pool.cc:225] Handling call
0901 01:52:34.337089 (+4846707us) ts_tablet_manager.cc:1145] Deleting tablet 5f52698fb56d472d9e91fb10cb10c69a
0901 01:52:34.337097 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:34.356153 (+ 19056us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:34.362709 (+ 6556us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:34.363895 (+ 1186us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4844263}
I20250901 01:52:34.364424 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 5f52698fb56d472d9e91fb10cb10c69a (table default.Table_83cf2d86a1284544ba2e7aa9ebb66bb7 [id=264d1a223e7d43c8a7a661db6722cb17]) successfully deleted
I20250901 01:52:34.366787 2118 ts_tablet_manager.cc:1918] T e1c8fe96a6ba4edba4f09c00f7add75e P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:34.366894 2118 log.cc:1199] T e1c8fe96a6ba4edba4f09c00f7add75e P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/e1c8fe96a6ba4edba4f09c00f7add75e
I20250901 01:52:34.367174 2118 ts_tablet_manager.cc:1939] T e1c8fe96a6ba4edba4f09c00f7add75e P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:34.369217 1151 tablet_bootstrap.cc:492] T 3363fd67b1ad44849ecb7b4090b61a72 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:34.369339 1151 ts_tablet_manager.cc:1397] T 3363fd67b1ad44849ecb7b4090b61a72 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.162s user 0.004s sys 0.012s
I20250901 01:52:34.369424 1486 catalog_manager.cc:5582] T 0e4009963cfe42d6ba164fc828897f09 P 00c282f779c1477d920f24dc0dc16628 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00c282f779c1477d920f24dc0dc16628 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "00c282f779c1477d920f24dc0dc16628" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: UNKNOWN } } }
I20250901 01:52:34.369813 1486 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 0e4009963cfe42d6ba164fc828897f09 (table default.Table_1adc70cccd2f4cb3a79df96861cb4181 [id=24fffc7ebf5a47bdb37211910301acf7]). Expected version 1 got 0
I20250901 01:52:34.369973 1151 raft_consensus.cc:357] T 3363fd67b1ad44849ecb7b4090b61a72 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:34.370110 1151 raft_consensus.cc:383] T 3363fd67b1ad44849ecb7b4090b61a72 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:34.370153 1151 raft_consensus.cc:738] T 3363fd67b1ad44849ecb7b4090b61a72 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:34.370704 1151 consensus_queue.cc:260] T 3363fd67b1ad44849ecb7b4090b61a72 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:34.371409 1151 ts_tablet_manager.cc:1428] T 3363fd67b1ad44849ecb7b4090b61a72 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250901 01:52:34.371709 1151 tablet_bootstrap.cc:492] T 2973b59bb1df41d18d9a5b6a29ae336f P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:34.376231 1151 tablet_bootstrap.cc:654] T 2973b59bb1df41d18d9a5b6a29ae336f P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:34.377420 1489 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d0c3c363dda444519a17dca63ad9caa7 consensus state has no leader
I20250901 01:52:34.378566 1484 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 71941b815d6840478c5acc1cbb187cc9
W20250901 01:52:34.378741 1484 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 71941b815d6840478c5acc1cbb187cc9: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:34.379313 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 71941b815d6840478c5acc1cbb187cc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:34 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:34.368378 917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "81ce448b148447ddbc41875d863e0518" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:34.379846 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 71941b815d6840478c5acc1cbb187cc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:34 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:34.382210 917 raft_consensus.cc:2466] T 81ce448b148447ddbc41875d863e0518 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:34.380733 992 leader_election.cc:304] T fecf7d538b4c4c619182d5eccb44dd8b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:34.382683 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 189) took 5879 ms. Trace:
I20250901 01:52:34.382764 2118 rpcz_store.cc:276] 0901 01:52:28.503611 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.503663 (+ 52us) service_pool.cc:225] Handling call
0901 01:52:34.336167 (+5832504us) ts_tablet_manager.cc:1145] Deleting tablet e1c8fe96a6ba4edba4f09c00f7add75e
0901 01:52:34.336175 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:34.347372 (+ 11197us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:34.366690 (+ 19318us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:34.382668 (+ 15978us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5832359}
I20250901 01:52:34.383194 2118 tablet_replica.cc:331] T 764bfeb2a7d14c40815b5ec1b07bdd62 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:34.383738 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet e1c8fe96a6ba4edba4f09c00f7add75e (table default.Table_f05ff31088d34354b72333af6d6c7235 [id=1c1a988d4e434fa88e7b5abded4419c4]) successfully deleted
I20250901 01:52:34.383755 2780 raft_consensus.cc:2802] T fecf7d538b4c4c619182d5eccb44dd8b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:34.383841 2780 raft_consensus.cc:491] T fecf7d538b4c4c619182d5eccb44dd8b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:34.383908 2780 raft_consensus.cc:3058] T fecf7d538b4c4c619182d5eccb44dd8b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:34.384078 2118 raft_consensus.cc:2241] T 764bfeb2a7d14c40815b5ec1b07bdd62 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:34.384272 2118 raft_consensus.cc:2270] T 764bfeb2a7d14c40815b5ec1b07bdd62 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:34.384572 2113 tablet_replica.cc:331] T 4fd0e4f9d86c4362af96ce9964b72328 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:34.385252 2118 ts_tablet_manager.cc:1905] T 764bfeb2a7d14c40815b5ec1b07bdd62 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:34.385555 2113 raft_consensus.cc:2241] T 4fd0e4f9d86c4362af96ce9964b72328 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:34.386976 2780 raft_consensus.cc:513] T fecf7d538b4c4c619182d5eccb44dd8b P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:34.388624 2780 leader_election.cc:290] T fecf7d538b4c4c619182d5eccb44dd8b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:34.385855 2113 raft_consensus.cc:2270] T 4fd0e4f9d86c4362af96ce9964b72328 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:34.389870 1151 tablet_bootstrap.cc:492] T 2973b59bb1df41d18d9a5b6a29ae336f P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:34.390033 1151 ts_tablet_manager.cc:1397] T 2973b59bb1df41d18d9a5b6a29ae336f P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.003s
I20250901 01:52:34.390973 1151 raft_consensus.cc:357] T 2973b59bb1df41d18d9a5b6a29ae336f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:34.391183 1151 raft_consensus.cc:383] T 2973b59bb1df41d18d9a5b6a29ae336f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:34.391250 1151 raft_consensus.cc:738] T 2973b59bb1df41d18d9a5b6a29ae336f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:34.391472 1151 consensus_queue.cc:260] T 2973b59bb1df41d18d9a5b6a29ae336f P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:34.392015 1151 ts_tablet_manager.cc:1428] T 2973b59bb1df41d18d9a5b6a29ae336f P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250901 01:52:34.392298 1151 tablet_bootstrap.cc:492] T 088ef4861dc14f81ae470c3aaee5b4e1 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:34.392177 1485 catalog_manager.cc:5582] T c6f7c525f59b49938b697de98d4b5835 P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: UNKNOWN } } }
I20250901 01:52:34.392647 1485 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet c6f7c525f59b49938b697de98d4b5835 (table default.Table_db61e055dbd7450898e82c4c0c4763be [id=aabca67f6f754820977ac9c4c20f3e4f]). Expected version 1 got 0
I20250901 01:52:34.392756 2780 consensus_queue.cc:1035] T 268439ff8d3a4f7a89f3d6395f2079c5 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:34.393002 1485 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 0e4009963cfe42d6ba164fc828897f09 (table default.Table_1adc70cccd2f4cb3a79df96861cb4181 [id=24fffc7ebf5a47bdb37211910301acf7]). Expected version 1 got 0
I20250901 01:52:34.393199 1485 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet b4278a013ae84082bc7a8497b04ac147 (table default.Table_297737979dae4534a0eef6ca9e87e98d [id=6b36921a6764455aa19284d36c3f8d32]). Expected version 1 got 0
I20250901 01:52:34.393340 1485 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet b2b172757e50404aae78e818921a7d9d (table default.Table_2e67ad583a0b413e9190cb55137ccf3a [id=5419e36671ec470a98dff53acdbd4760]). Expected version 1 got 0
I20250901 01:52:34.393638 1485 catalog_manager.cc:5582] T 93bcac1fd95d44dc927abfca81692921 P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } }
I20250901 01:52:34.393962 1151 tablet_bootstrap.cc:654] T 088ef4861dc14f81ae470c3aaee5b4e1 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:34.394050 1485 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 93bcac1fd95d44dc927abfca81692921 (table default.Table_b8f8ee74648c406eaf589d3ab80934db [id=b8f69313439543be9c11eea88dfa69c5]). Expected version 1 got 0
I20250901 01:52:34.394688 1482 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_0cb2185875b646dcac4d1f6f2f4cffd5" } new_table_name: "default.Table_d3fd1940d13e4afba8b3bb674bd4e4ee" modify_external_catalogs: true
I20250901 01:52:34.395437 1488 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 858218c4427c41e980b2b5c969a5af40
W20250901 01:52:34.395642 1488 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 858218c4427c41e980b2b5c969a5af40: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:34.396044 2781 consensus_queue.cc:1035] T 3eba54e5ddd14c518f4cbd539268ae37 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:34.396267 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 858218c4427c41e980b2b5c969a5af40 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:34 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:34.396800 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 858218c4427c41e980b2b5c969a5af40 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:34 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:34.397670 1488 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_14e7f1e7dd314cdcaee09ac2fdf28993" } modify_external_catalogs: true
I20250901 01:52:34.400007 2781 raft_consensus.cc:491] T 3363fd67b1ad44849ecb7b4090b61a72 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:34.400123 2781 raft_consensus.cc:513] T 3363fd67b1ad44849ecb7b4090b61a72 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:34.400777 2781 leader_election.cc:290] T 3363fd67b1ad44849ecb7b4090b61a72 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:34.400825 992 consensus_peers.cc:489] T 26c8ed083cea4dc0acca80d00b11b29b P fd27ab6665304aaea53adbd102dd4f58 -> Peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Couldn't send request to peer 00c282f779c1477d920f24dc0dc16628. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 26c8ed083cea4dc0acca80d00b11b29b. This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:34.402527 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d36d80ef4ae7407fbcbc85cce80a092b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.402870 2113 ts_tablet_manager.cc:1905] T 4fd0e4f9d86c4362af96ce9964b72328 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:34.403261 1484 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_cebf7994eb8541e7ab9957cc012cb328" } new_table_name: "default.Table_3cd96297e5ac445d9fcfb30dd37697da" modify_external_catalogs: true
I20250901 01:52:34.403566 1486 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: default.Table_4505d77e4269443995012fef50c7e7b5 [id=c1b1e9e3876244f68d31c63ce7bdebf1] alter complete (version 2)
I20250901 01:52:34.405702 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_709cde5b8e904a12a465f2659b980037"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.407796 1492 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 508406601c994e42a5c5b1f665e74957
W20250901 01:52:34.407975 1492 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 508406601c994e42a5c5b1f665e74957: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:34.408327 1492 catalog_manager.cc:6652] ReplaceTablet: tablet 508406601c994e42a5c5b1f665e74957 deleted and replaced by tablet 082db24bb3fd4089950504145a20a075
I20250901 01:52:34.408921 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 508406601c994e42a5c5b1f665e74957 with delete_type TABLET_DATA_DELETED (replaced by tablet 082db24bb3fd4089950504145a20a075) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:34.409237 1488 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_14e7f1e7dd314cdcaee09ac2fdf28993" } modify_external_catalogs: true
I20250901 01:52:34.409614 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 508406601c994e42a5c5b1f665e74957 with delete_type TABLET_DATA_DELETED (replaced by tablet 082db24bb3fd4089950504145a20a075) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:34.413146 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4d9a4cdb857b488aa69b583b778168f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.414247 992 leader_election.cc:304] T 81ce448b148447ddbc41875d863e0518 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:34.416000 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_06fda91ba68649af8f64c8e2b67e27c1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.417392 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1a7e8ee501d045c18965e420a1c5f70c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.417357 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_dea75a70aca54f5fb6d22f7a37f73e39"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.418978 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e88b182a708045bbb5d494f0b46aff48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.420061 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3b7901e0fa5d455aa95a5cb2be67e770"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.420217 2118 ts_tablet_manager.cc:1918] T 764bfeb2a7d14c40815b5ec1b07bdd62 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:34.420343 2118 log.cc:1199] T 764bfeb2a7d14c40815b5ec1b07bdd62 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/764bfeb2a7d14c40815b5ec1b07bdd62
I20250901 01:52:34.421204 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4f2f6833581a41dbbcf92dfbbefc805c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.422247 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4dd234ef812f4d2eba3f58fffca57f35"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:34.423539 992 leader_election.cc:336] T fecf7d538b4c4c619182d5eccb44dd8b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:34.424366 2781 raft_consensus.cc:2802] T 81ce448b148447ddbc41875d863e0518 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:34.424487 2781 raft_consensus.cc:491] T 81ce448b148447ddbc41875d863e0518 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:34.424561 2781 raft_consensus.cc:3058] T 81ce448b148447ddbc41875d863e0518 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:34.428144 917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fecf7d538b4c4c619182d5eccb44dd8b" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:34.428323 917 raft_consensus.cc:3058] T fecf7d538b4c4c619182d5eccb44dd8b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:34.430876 917 raft_consensus.cc:2466] T fecf7d538b4c4c619182d5eccb44dd8b P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:34.434095 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1227cf58bd834a0c9a17d8680ba3af28"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.435691 992 leader_election.cc:304] T fecf7d538b4c4c619182d5eccb44dd8b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:34.435842 2113 ts_tablet_manager.cc:1918] T 4fd0e4f9d86c4362af96ce9964b72328 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 01:52:34.435966 2113 log.cc:1199] T 4fd0e4f9d86c4362af96ce9964b72328 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/4fd0e4f9d86c4362af96ce9964b72328
I20250901 01:52:34.436398 2113 ts_tablet_manager.cc:1939] T 4fd0e4f9d86c4362af96ce9964b72328 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:34.436406 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_bcf948b3503a40d49104bf9cff09fce2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.437736 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_24780ce6d7594babb78e6b8de61dc4ad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.439169 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9e540f6293844953a150a53e97cba50b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.439864 2781 raft_consensus.cc:513] T 81ce448b148447ddbc41875d863e0518 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:34.440339 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3ca2d70c39c544a2a05be36975a29c88"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.440480 2781 leader_election.cc:290] T 81ce448b148447ddbc41875d863e0518 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:34.441885 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_0cc765039878453fa85462625dae67a0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.444049 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9cbde5abae924da5bb87c30036968515"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.444945 2118 ts_tablet_manager.cc:1939] T 764bfeb2a7d14c40815b5ec1b07bdd62 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:34.445107 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c9b39de7982a4ffdaf280b8d22667b41"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.445343 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 185) took 5944 ms. Trace:
I20250901 01:52:34.445468 2118 rpcz_store.cc:276] 0901 01:52:28.501112 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.503948 (+ 2836us) service_pool.cc:225] Handling call
0901 01:52:34.383142 (+5879194us) ts_tablet_manager.cc:1145] Deleting tablet 764bfeb2a7d14c40815b5ec1b07bdd62
0901 01:52:34.383151 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:34.415501 (+ 32350us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:34.420093 (+ 4592us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:34.445324 (+ 25231us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5879047}
I20250901 01:52:34.445905 2118 tablet_replica.cc:331] T 4fca1435d82447f289fda4035aa2ed89 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:34.446147 2118 raft_consensus.cc:2241] T 4fca1435d82447f289fda4035aa2ed89 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:34.446319 2118 raft_consensus.cc:2270] T 4fca1435d82447f289fda4035aa2ed89 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:34.445892 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_df98422134f54c9d8e017e0f51b754d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.446907 2357 raft_consensus.cc:491] T d0c3c363dda444519a17dca63ad9caa7 P 00c282f779c1477d920f24dc0dc16628 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:34.447021 2357 raft_consensus.cc:513] T d0c3c363dda444519a17dca63ad9caa7 P 00c282f779c1477d920f24dc0dc16628 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:34.446727 1490 master_service.cc:948] ReplaceTablet: received request to replace tablet 4fca1435d82447f289fda4035aa2ed89 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:34.447633 2118 ts_tablet_manager.cc:1905] T 4fca1435d82447f289fda4035aa2ed89 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:34.448609 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_34ae35bcf18d4569b579fadbf6fc8d74"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.448992 1151 tablet_bootstrap.cc:492] T 088ef4861dc14f81ae470c3aaee5b4e1 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:34.449115 1151 ts_tablet_manager.cc:1397] T 088ef4861dc14f81ae470c3aaee5b4e1 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.057s user 0.020s sys 0.016s
W20250901 01:52:34.449122 858 leader_election.cc:336] T d0c3c363dda444519a17dca63ad9caa7 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:34.449522 2357 leader_election.cc:290] T d0c3c363dda444519a17dca63ad9caa7 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:34.449821 2780 raft_consensus.cc:2802] T fecf7d538b4c4c619182d5eccb44dd8b P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:34.449972 2780 raft_consensus.cc:695] T fecf7d538b4c4c619182d5eccb44dd8b P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:34.450284 2780 consensus_queue.cc:237] T fecf7d538b4c4c619182d5eccb44dd8b P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:34.452935 1151 raft_consensus.cc:357] T 088ef4861dc14f81ae470c3aaee5b4e1 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:34.453112 1151 raft_consensus.cc:383] T 088ef4861dc14f81ae470c3aaee5b4e1 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:34.453177 1151 raft_consensus.cc:738] T 088ef4861dc14f81ae470c3aaee5b4e1 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:34.453385 1151 consensus_queue.cc:260] T 088ef4861dc14f81ae470c3aaee5b4e1 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:34.453935 1151 ts_tablet_manager.cc:1428] T 088ef4861dc14f81ae470c3aaee5b4e1 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250901 01:52:34.454196 1151 tablet_bootstrap.cc:492] T 2dacb608897d47d28512df87dc320927 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:34.455814 1151 tablet_bootstrap.cc:654] T 2dacb608897d47d28512df87dc320927 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:34.456995 992 leader_election.cc:343] T 3363fd67b1ad44849ecb7b4090b61a72 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 3363fd67b1ad44849ecb7b4090b61a72
W20250901 01:52:34.457988 992 leader_election.cc:336] T 81ce448b148447ddbc41875d863e0518 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:34.458200 992 consensus_peers.cc:489] T 26c8ed083cea4dc0acca80d00b11b29b P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250901 01:52:34.458338 992 consensus_peers.cc:489] T 6789505328b1409ea288e8b4eaf94935 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250901 01:52:34.458463 992 leader_election.cc:336] T 3363fd67b1ad44849ecb7b4090b61a72 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:34.458557 992 leader_election.cc:304] T 3363fd67b1ad44849ecb7b4090b61a72 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:34.458791 2781 raft_consensus.cc:2747] T 3363fd67b1ad44849ecb7b4090b61a72 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:34.458873 992 leader_election.cc:336] T fecf7d538b4c4c619182d5eccb44dd8b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:34.460031 2118 ts_tablet_manager.cc:1918] T 4fca1435d82447f289fda4035aa2ed89 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:34.460173 2118 log.cc:1199] T 4fca1435d82447f289fda4035aa2ed89 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/4fca1435d82447f289fda4035aa2ed89
I20250901 01:52:34.460487 2118 ts_tablet_manager.cc:1939] T 4fca1435d82447f289fda4035aa2ed89 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:34.460794 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 190) took 5955 ms. Trace:
I20250901 01:52:34.460896 2118 rpcz_store.cc:276] 0901 01:52:28.505780 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.505844 (+ 64us) service_pool.cc:225] Handling call
0901 01:52:34.445849 (+5940005us) ts_tablet_manager.cc:1145] Deleting tablet 4fca1435d82447f289fda4035aa2ed89
0901 01:52:34.445860 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:34.453476 (+ 7616us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:34.459936 (+ 6460us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:34.460779 (+ 843us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5939835}
I20250901 01:52:34.456373 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 764bfeb2a7d14c40815b5ec1b07bdd62 (table default.Table_4401d83f75f8464da49d5b811d7dd89d [id=16143c8e6f484f9a873437e709d8466d]) successfully deleted
I20250901 01:52:34.461450 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 191) took 5955 ms. Trace:
I20250901 01:52:34.461578 2118 rpcz_store.cc:276] 0901 01:52:28.506411 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.506477 (+ 66us) service_pool.cc:225] Handling call
0901 01:52:34.461363 (+5954886us) ts_tablet_manager.cc:1145] Deleting tablet 46bfbdff3f904715a8c24907b20fcab0
0901 01:52:34.461373 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:34.461437 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5954730}
I20250901 01:52:34.462270 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 192) took 5955 ms. Trace:
I20250901 01:52:34.462399 2118 rpcz_store.cc:276] 0901 01:52:28.507212 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.507273 (+ 61us) service_pool.cc:225] Handling call
0901 01:52:34.462183 (+5954910us) ts_tablet_manager.cc:1145] Deleting tablet 670d90a37b934579b353bb4e4820127b
0901 01:52:34.462193 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:34.462255 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5954765}
I20250901 01:52:34.462795 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 193) took 5955 ms. Trace:
I20250901 01:52:34.462901 2118 rpcz_store.cc:276] 0901 01:52:28.507794 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.507841 (+ 47us) service_pool.cc:225] Handling call
0901 01:52:34.462727 (+5954886us) ts_tablet_manager.cc:1145] Deleting tablet 50e3da53433f4c14b57db8a733fa53bc
0901 01:52:34.462734 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:34.462776 (+ 42us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5954734}
I20250901 01:52:34.463294 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 194) took 5955 ms. Trace:
I20250901 01:52:34.463901 2118 rpcz_store.cc:276] 0901 01:52:28.508278 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.508435 (+ 157us) service_pool.cc:225] Handling call
0901 01:52:34.463232 (+5954797us) ts_tablet_manager.cc:1145] Deleting tablet f8a5886ddb644a86b76df44aab309057
0901 01:52:34.463239 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:34.463283 (+ 44us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5954631}
I20250901 01:52:34.464298 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 195) took 5955 ms. Trace:
I20250901 01:52:34.464668 2118 rpcz_store.cc:276] 0901 01:52:28.508904 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.508959 (+ 55us) service_pool.cc:225] Handling call
0901 01:52:34.464236 (+5955277us) ts_tablet_manager.cc:1145] Deleting tablet 1286c0ab85704837a99b6e5a62184d77
0901 01:52:34.464244 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:34.464287 (+ 43us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5955113}
I20250901 01:52:34.465123 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 196) took 5955 ms. Trace:
I20250901 01:52:34.465237 2118 rpcz_store.cc:276] 0901 01:52:28.509801 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.509865 (+ 64us) service_pool.cc:225] Handling call
0901 01:52:34.465052 (+5955187us) ts_tablet_manager.cc:1145] Deleting tablet 822382a1781b4304bce29355ce83730e
0901 01:52:34.465060 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:34.465109 (+ 49us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5955033}
I20250901 01:52:34.465920 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 197) took 5955 ms. Trace:
I20250901 01:52:34.466737 1151 tablet_bootstrap.cc:492] T 2dacb608897d47d28512df87dc320927 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:34.466909 1151 ts_tablet_manager.cc:1397] T 2dacb608897d47d28512df87dc320927 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20250901 01:52:34.466764 917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "81ce448b148447ddbc41875d863e0518" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:34.467604 2118 rpcz_store.cc:276] 0901 01:52:28.510369 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.510446 (+ 77us) service_pool.cc:225] Handling call
0901 01:52:34.465849 (+5955403us) ts_tablet_manager.cc:1145] Deleting tablet 579b9cdf819f4609a532e96bec093fd5
0901 01:52:34.465857 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:34.465905 (+ 48us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5955262}
I20250901 01:52:34.467805 917 raft_consensus.cc:3058] T 81ce448b148447ddbc41875d863e0518 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:34.467878 1489 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_2b946a63bd574ac0971fa3d3b9c9a6a4" } modify_external_catalogs: true
I20250901 01:52:34.468003 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 198) took 5956 ms. Trace:
I20250901 01:52:34.468148 1489 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_2b946a63bd574ac0971fa3d3b9c9a6a4" } modify_external_catalogs: true
I20250901 01:52:34.468099 2118 rpcz_store.cc:276] 0901 01:52:28.511244 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.511291 (+ 47us) service_pool.cc:225] Handling call
0901 01:52:34.467932 (+5956641us) ts_tablet_manager.cc:1145] Deleting tablet d77aadd7291647a0bae7c421f244c252
0901 01:52:34.467940 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:34.467989 (+ 49us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5956479}
I20250901 01:52:34.469517 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 199) took 5957 ms. Trace:
I20250901 01:52:34.469647 2118 rpcz_store.cc:276] 0901 01:52:28.512034 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.512082 (+ 48us) service_pool.cc:225] Handling call
0901 01:52:34.469449 (+5957367us) ts_tablet_manager.cc:1145] Deleting tablet c49d1143d54b4c319739488906b0e271
0901 01:52:34.469455 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:34.469503 (+ 48us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5957221}
I20250901 01:52:34.470047 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 200) took 5957 ms. Trace:
I20250901 01:52:34.470398 917 raft_consensus.cc:2466] T 81ce448b148447ddbc41875d863e0518 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:34.470146 2118 rpcz_store.cc:276] 0901 01:52:28.512859 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.512901 (+ 42us) service_pool.cc:225] Handling call
0901 01:52:34.469982 (+5957081us) ts_tablet_manager.cc:1145] Deleting tablet 4855448edb4b40c184838aa7f81c112b
0901 01:52:34.469989 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:34.470035 (+ 46us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5956932}
I20250901 01:52:34.470949 2118 tablet_replica.cc:331] T a6445b29591548be9f9577db1ba0bfa5 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:34.471201 2118 raft_consensus.cc:2241] T a6445b29591548be9f9577db1ba0bfa5 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:34.471366 2118 raft_consensus.cc:2270] T a6445b29591548be9f9577db1ba0bfa5 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:34.471405 1491 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_5e65c5f068b84995a2d7a76fa65dda0f" } new_table_name: "default.Table_e09ac1a9050547038400d4b52299d22f" modify_external_catalogs: true
I20250901 01:52:34.472668 1151 raft_consensus.cc:357] T 2dacb608897d47d28512df87dc320927 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:34.473323 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 4fca1435d82447f289fda4035aa2ed89 (table default.Table_fbc57d539a2b41e59dfc5369ac8c92e1 [id=8e14f99474a1467baed19f5268f09c5e]) successfully deleted
W20250901 01:52:34.473685 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 46bfbdff3f904715a8c24907b20fcab0 because the tablet was not found. No further retry: Not found: Tablet not found: 46bfbdff3f904715a8c24907b20fcab0
W20250901 01:52:34.473928 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 670d90a37b934579b353bb4e4820127b because the tablet was not found. No further retry: Not found: Tablet not found: 670d90a37b934579b353bb4e4820127b
W20250901 01:52:34.474177 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 50e3da53433f4c14b57db8a733fa53bc because the tablet was not found. No further retry: Not found: Tablet not found: 50e3da53433f4c14b57db8a733fa53bc
I20250901 01:52:34.474382 1151 raft_consensus.cc:383] T 2dacb608897d47d28512df87dc320927 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:34.474452 1151 raft_consensus.cc:738] T 2dacb608897d47d28512df87dc320927 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:34.474660 1151 consensus_queue.cc:260] T 2dacb608897d47d28512df87dc320927 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:34.475162 1151 ts_tablet_manager.cc:1428] T 2dacb608897d47d28512df87dc320927 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20250901 01:52:34.475418 1151 tablet_bootstrap.cc:492] T 508406601c994e42a5c5b1f665e74957 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:34.476518 2753 raft_consensus.cc:491] T ae21b9ae2e3b472eb67fb5f96388a50f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:34.476709 2753 raft_consensus.cc:513] T ae21b9ae2e3b472eb67fb5f96388a50f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:34.477326 1151 tablet_bootstrap.cc:654] T 508406601c994e42a5c5b1f665e74957 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:34.478500 2118 ts_tablet_manager.cc:1905] T a6445b29591548be9f9577db1ba0bfa5 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:34.478711 858 leader_election.cc:336] T ae21b9ae2e3b472eb67fb5f96388a50f P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:34.479153 2753 leader_election.cc:290] T ae21b9ae2e3b472eb67fb5f96388a50f P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:34.479686 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0c3c363dda444519a17dca63ad9caa7" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:34.479862 1052 raft_consensus.cc:2466] T d0c3c363dda444519a17dca63ad9caa7 P fd27ab6665304aaea53adbd102dd4f58 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00c282f779c1477d920f24dc0dc16628 in term 2.
I20250901 01:52:34.485924 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ae21b9ae2e3b472eb67fb5f96388a50f" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:34.487102 2118 ts_tablet_manager.cc:1918] T a6445b29591548be9f9577db1ba0bfa5 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:34.477667 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5de3c5864ede4426bc39bb86e895c98d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.479157 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1e46cf2181dd4aa58c57261a69b36c62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.480742 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_63e6d842edad46dc9de10477cb7a04f1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.488209 2118 log.cc:1199] T a6445b29591548be9f9577db1ba0bfa5 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/a6445b29591548be9f9577db1ba0bfa5
I20250901 01:52:34.488575 2118 ts_tablet_manager.cc:1939] T a6445b29591548be9f9577db1ba0bfa5 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:34.488859 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 201) took 5628 ms. Trace:
I20250901 01:52:34.489722 2780 raft_consensus.cc:491] T 435b1cca5a7d4d49959a310cb250b2a0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:34.489871 2780 raft_consensus.cc:513] T 435b1cca5a7d4d49959a310cb250b2a0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:34.489722 2118 rpcz_store.cc:276] 0901 01:52:28.860109 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.860177 (+ 68us) service_pool.cc:225] Handling call
0901 01:52:34.470896 (+5610719us) ts_tablet_manager.cc:1145] Deleting tablet a6445b29591548be9f9577db1ba0bfa5
0901 01:52:34.470905 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:34.481645 (+ 10740us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:34.487020 (+ 5375us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:34.488844 (+ 1824us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5610513}
I20250901 01:52:34.490374 2780 leader_election.cc:290] T 435b1cca5a7d4d49959a310cb250b2a0 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:34.490427 2118 tablet_replica.cc:331] T 1495de70f0844f82a0eea99c059f076b P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:34.490586 992 leader_election.cc:304] T 81ce448b148447ddbc41875d863e0518 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:34.490689 859 leader_election.cc:304] T d0c3c363dda444519a17dca63ad9caa7 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:34.490798 2780 raft_consensus.cc:2802] T 81ce448b148447ddbc41875d863e0518 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:34.490832 2118 raft_consensus.cc:2241] T 1495de70f0844f82a0eea99c059f076b P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:34.490890 2780 raft_consensus.cc:695] T 81ce448b148447ddbc41875d863e0518 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:34.490975 2118 raft_consensus.cc:2270] T 1495de70f0844f82a0eea99c059f076b P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:34.491093 2780 consensus_queue.cc:237] T 81ce448b148447ddbc41875d863e0518 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:34.491429 2608 raft_consensus.cc:2802] T d0c3c363dda444519a17dca63ad9caa7 P 00c282f779c1477d920f24dc0dc16628 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250901 01:52:34.491521 2608 raft_consensus.cc:491] T d0c3c363dda444519a17dca63ad9caa7 P 00c282f779c1477d920f24dc0dc16628 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:34.491604 2608 raft_consensus.cc:3058] T d0c3c363dda444519a17dca63ad9caa7 P 00c282f779c1477d920f24dc0dc16628 [term 2 FOLLOWER]: Advancing to term 3
W20250901 01:52:34.492101 992 leader_election.cc:336] T 81ce448b148447ddbc41875d863e0518 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:34.492887 2118 ts_tablet_manager.cc:1905] T 1495de70f0844f82a0eea99c059f076b P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:34.493981 992 consensus_peers.cc:489] T d0b7c84ec9394a20ad77add3808e555a P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:34.494177 2608 raft_consensus.cc:513] T d0c3c363dda444519a17dca63ad9caa7 P 00c282f779c1477d920f24dc0dc16628 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
W20250901 01:52:34.495056 992 leader_election.cc:336] T 435b1cca5a7d4d49959a310cb250b2a0 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:34.495309 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9a3dff4886474a5d9421c42986d360c6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.496302 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_140ad29a8a5e422f81f6defd5a0ce895"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.497205 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f2098daf03414192adc0a0027acab70b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.498035 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ed3f9254bced4f81b7bd7e47d9a9852f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.490439 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_70416fd48e9b43b58d4d7836249645f5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:34.499581 858 leader_election.cc:336] T d0c3c363dda444519a17dca63ad9caa7 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:34.499986 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b45ed0090bb94284a208903b07bb2bdb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.500255 2118 ts_tablet_manager.cc:1918] T 1495de70f0844f82a0eea99c059f076b P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:34.500362 2118 log.cc:1199] T 1495de70f0844f82a0eea99c059f076b P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/1495de70f0844f82a0eea99c059f076b
I20250901 01:52:34.500521 1483 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_e0902dfbdd984f85b3fdac8d283731f7" } modify_external_catalogs: true
I20250901 01:52:34.500690 1483 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_e0902dfbdd984f85b3fdac8d283731f7" } modify_external_catalogs: true
I20250901 01:52:34.501180 1151 tablet_bootstrap.cc:492] T 508406601c994e42a5c5b1f665e74957 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:34.501293 1151 ts_tablet_manager.cc:1397] T 508406601c994e42a5c5b1f665e74957 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.002s
I20250901 01:52:34.501310 2118 ts_tablet_manager.cc:1939] T 1495de70f0844f82a0eea99c059f076b P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:34.501636 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 202) took 5636 ms. Trace:
I20250901 01:52:34.501701 2118 rpcz_store.cc:276] 0901 01:52:28.864739 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.864792 (+ 53us) service_pool.cc:225] Handling call
0901 01:52:34.490384 (+5625592us) ts_tablet_manager.cc:1145] Deleting tablet 1495de70f0844f82a0eea99c059f076b
0901 01:52:34.490394 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:34.496678 (+ 6284us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:34.500176 (+ 3498us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:34.501620 (+ 1444us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5625441}
I20250901 01:52:34.502032 1151 raft_consensus.cc:357] T 508406601c994e42a5c5b1f665e74957 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:34.502184 1151 raft_consensus.cc:383] T 508406601c994e42a5c5b1f665e74957 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:34.502236 1151 raft_consensus.cc:738] T 508406601c994e42a5c5b1f665e74957 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:34.502274 2118 tablet_replica.cc:331] T 8b9d79a25452410891c09fd3f7c4be6d P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:34.502444 2118 raft_consensus.cc:2241] T 8b9d79a25452410891c09fd3f7c4be6d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:34.502439 1151 consensus_queue.cc:260] T 508406601c994e42a5c5b1f665e74957 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:34.502635 2118 raft_consensus.cc:2270] T 8b9d79a25452410891c09fd3f7c4be6d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:34.502944 1151 ts_tablet_manager.cc:1428] T 508406601c994e42a5c5b1f665e74957 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250901 01:52:34.503207 1151 tablet_bootstrap.cc:492] T 7f46e9ddadaf4ff986d8c991178daa0c P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:34.503201 2118 ts_tablet_manager.cc:1905] T 8b9d79a25452410891c09fd3f7c4be6d P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:34.505911 1151 tablet_bootstrap.cc:654] T 7f46e9ddadaf4ff986d8c991178daa0c P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:34.505893 2608 leader_election.cc:290] T d0c3c363dda444519a17dca63ad9caa7 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 3 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:34.506197 2829 tablet.cc:1722] T 858218c4427c41e980b2b5c969a5af40 P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250901 01:52:34.507349 859 leader_election.cc:343] T ae21b9ae2e3b472eb67fb5f96388a50f P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:34.507476 859 leader_election.cc:304] T ae21b9ae2e3b472eb67fb5f96388a50f P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
W20250901 01:52:34.507705 992 leader_election.cc:343] T 435b1cca5a7d4d49959a310cb250b2a0 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 435b1cca5a7d4d49959a310cb250b2a0
I20250901 01:52:34.507807 992 leader_election.cc:304] T 435b1cca5a7d4d49959a310cb250b2a0 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:34.508232 2856 raft_consensus.cc:2747] T 435b1cca5a7d4d49959a310cb250b2a0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:34.509285 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0c3c363dda444519a17dca63ad9caa7" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58"
I20250901 01:52:34.509337 2608 raft_consensus.cc:2747] T ae21b9ae2e3b472eb67fb5f96388a50f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:34.509440 1052 raft_consensus.cc:3058] T d0c3c363dda444519a17dca63ad9caa7 P fd27ab6665304aaea53adbd102dd4f58 [term 2 FOLLOWER]: Advancing to term 3
I20250901 01:52:34.510293 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_812d8a20f3ad4b2fa16ada199f68a5dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.511864 1052 raft_consensus.cc:2466] T d0c3c363dda444519a17dca63ad9caa7 P fd27ab6665304aaea53adbd102dd4f58 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c282f779c1477d920f24dc0dc16628 in term 3.
W20250901 01:52:34.512544 858 consensus_peers.cc:489] T 3d90c6ef06d1436d8e2bb1333e7d9985 P 00c282f779c1477d920f24dc0dc16628 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250901 01:52:34.512812 859 leader_election.cc:304] T d0c3c363dda444519a17dca63ad9caa7 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:34.513160 2778 tablet.cc:1722] T 858218c4427c41e980b2b5c969a5af40 P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:34.515636 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5e4d30928cc4497a88eea1e2448eee98"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.517079 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a5d50dd2ed7a415699f8823dbdf99957"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.517297 2871 raft_consensus.cc:2802] T d0c3c363dda444519a17dca63ad9caa7 P 00c282f779c1477d920f24dc0dc16628 [term 3 FOLLOWER]: Leader election won for term 3
I20250901 01:52:34.517406 2871 raft_consensus.cc:695] T d0c3c363dda444519a17dca63ad9caa7 P 00c282f779c1477d920f24dc0dc16628 [term 3 LEADER]: Becoming Leader. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Running, Role: LEADER
I20250901 01:52:34.517638 2871 consensus_queue.cc:237] T d0c3c363dda444519a17dca63ad9caa7 P 00c282f779c1477d920f24dc0dc16628 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:34.518718 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ffb125099eea401a8b891a8a0635e19c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.520668 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4059d9adc6134ffb9aad4da163441b2e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.521586 1151 tablet_bootstrap.cc:492] T 7f46e9ddadaf4ff986d8c991178daa0c P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:34.521730 1151 ts_tablet_manager.cc:1397] T 7f46e9ddadaf4ff986d8c991178daa0c P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.006s
I20250901 01:52:34.522513 1151 raft_consensus.cc:357] T 7f46e9ddadaf4ff986d8c991178daa0c P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:34.522678 1151 raft_consensus.cc:383] T 7f46e9ddadaf4ff986d8c991178daa0c P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:34.522742 1151 raft_consensus.cc:738] T 7f46e9ddadaf4ff986d8c991178daa0c P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:34.523478 1151 consensus_queue.cc:260] T 7f46e9ddadaf4ff986d8c991178daa0c P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:34.524788 1151 ts_tablet_manager.cc:1428] T 7f46e9ddadaf4ff986d8c991178daa0c P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250901 01:52:34.525072 1151 tablet_bootstrap.cc:492] T ae21b9ae2e3b472eb67fb5f96388a50f P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:34.526374 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a79e7296fb8c4ce4bd217ae38bddd3e9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.527288 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c6fd61b92f414e61a7f8aef593d03cd2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.528384 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3ead165564db444e99dd5e5572754e9b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.529402 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_60b26dd468e94626833b77bd7b017e25"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.530097 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d1d8d37c6c994bccb39120c3d1135337"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.530766 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e4d9e748fbcd449591ee99e6c99f2a54"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.531454 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a66d501319c745bf98d23f2fb858f458"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.531730 2118 ts_tablet_manager.cc:1918] T 8b9d79a25452410891c09fd3f7c4be6d P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:34.531857 2118 log.cc:1199] T 8b9d79a25452410891c09fd3f7c4be6d P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/8b9d79a25452410891c09fd3f7c4be6d
I20250901 01:52:34.532182 2118 ts_tablet_manager.cc:1939] T 8b9d79a25452410891c09fd3f7c4be6d P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:34.532171 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7102852e191740ad909403a693345fb7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.532630 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 203) took 5665 ms. Trace:
I20250901 01:52:34.532733 2118 rpcz_store.cc:276] 0901 01:52:28.867085 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.867143 (+ 58us) service_pool.cc:225] Handling call
0901 01:52:34.502231 (+5635088us) ts_tablet_manager.cc:1145] Deleting tablet 8b9d79a25452410891c09fd3f7c4be6d
0901 01:52:34.502241 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:34.525092 (+ 22851us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:34.531520 (+ 6428us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:34.532614 (+ 1094us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5622119}
I20250901 01:52:34.532904 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3dbd3712c45d4cbf96d54e979c6c8094"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.533327 2118 tablet_replica.cc:331] T 7304937048ca48a7bd153f3cdcfaeff0 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:34.533545 2118 raft_consensus.cc:2241] T 7304937048ca48a7bd153f3cdcfaeff0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:34.533711 2118 raft_consensus.cc:2270] T 7304937048ca48a7bd153f3cdcfaeff0 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:34.534193 2118 ts_tablet_manager.cc:1905] T 7304937048ca48a7bd153f3cdcfaeff0 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:34.537122 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5127faa0c27244a5b206917d45d347bc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.537858 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_941a7daa809f4894993e004dd5af4f8e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.538600 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_37fadacb7ddc45ec9f67afaff546498b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.539366 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b2bdd99312e5449b81861786d55988d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.540122 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_310abcd2a21f46e692ad6449fde69982"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.540871 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9dbe3e61f8274851bd3b71538927cc3d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.540952 2118 ts_tablet_manager.cc:1918] T 7304937048ca48a7bd153f3cdcfaeff0 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:34.541039 2118 log.cc:1199] T 7304937048ca48a7bd153f3cdcfaeff0 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/7304937048ca48a7bd153f3cdcfaeff0
I20250901 01:52:34.541289 2118 ts_tablet_manager.cc:1939] T 7304937048ca48a7bd153f3cdcfaeff0 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:34.541621 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 204) took 5563 ms. Trace:
I20250901 01:52:34.541623 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_bdf8a9f4da7346e082ecb14b1a29c17f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.541735 2118 rpcz_store.cc:276] 0901 01:52:28.978450 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:28.978611 (+ 161us) service_pool.cc:225] Handling call
0901 01:52:34.533281 (+5554670us) ts_tablet_manager.cc:1145] Deleting tablet 7304937048ca48a7bd153f3cdcfaeff0
0901 01:52:34.533291 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:34.537685 (+ 4394us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:34.540877 (+ 3192us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:34.541605 (+ 728us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5554420}
I20250901 01:52:34.542359 2118 tablet_replica.cc:331] T 186ec810d02e4748aad15228b5866ac1 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:34.542585 2118 raft_consensus.cc:2241] T 186ec810d02e4748aad15228b5866ac1 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
W20250901 01:52:34.542577 1472 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.2:40561, user_credentials={real_user=slave}} blocked reactor thread for 68420us
I20250901 01:52:34.542791 2118 raft_consensus.cc:2270] T 186ec810d02e4748aad15228b5866ac1 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250901 01:52:34.542898 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet f8a5886ddb644a86b76df44aab309057 because the tablet was not found. No further retry: Not found: Tablet not found: f8a5886ddb644a86b76df44aab309057
W20250901 01:52:34.543095 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 1286c0ab85704837a99b6e5a62184d77 because the tablet was not found. No further retry: Not found: Tablet not found: 1286c0ab85704837a99b6e5a62184d77
W20250901 01:52:34.543263 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 822382a1781b4304bce29355ce83730e because the tablet was not found. No further retry: Not found: Tablet not found: 822382a1781b4304bce29355ce83730e
W20250901 01:52:34.543470 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet d77aadd7291647a0bae7c421f244c252 because the tablet was not found. No further retry: Not found: Tablet not found: d77aadd7291647a0bae7c421f244c252
I20250901 01:52:34.543500 2118 ts_tablet_manager.cc:1905] T 186ec810d02e4748aad15228b5866ac1 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:34.543742 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet a6445b29591548be9f9577db1ba0bfa5 (table default.Table_e225c82a2d6045f6b16915cdb72f5734 [id=6f29cdc9974049368af2ae10beadb772]) successfully deleted
I20250901 01:52:34.543916 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 1495de70f0844f82a0eea99c059f076b (table default.Table_7570c3012f8a4ae7bcfc7346eadac92a [id=70a5ce71d16b47d4bc6dae97be4548e7]) successfully deleted
I20250901 01:52:34.544129 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 8b9d79a25452410891c09fd3f7c4be6d (table default.Table_442e11b9211c4b209c68f653d7bfc313 [id=af349c3f534e4e8e99fdae0344187c44]) successfully deleted
I20250901 01:52:34.544279 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 7304937048ca48a7bd153f3cdcfaeff0 (table default.Table_1e59568bdcd14662ae9d5e98293d74bd [id=8051adf4041f4a2686a0b8e3850ef29c]) successfully deleted
I20250901 01:52:34.544849 1151 tablet_bootstrap.cc:654] T ae21b9ae2e3b472eb67fb5f96388a50f P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:34.551769 2118 ts_tablet_manager.cc:1918] T 186ec810d02e4748aad15228b5866ac1 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:34.551957 2118 log.cc:1199] T 186ec810d02e4748aad15228b5866ac1 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/186ec810d02e4748aad15228b5866ac1
I20250901 01:52:34.552363 2118 ts_tablet_manager.cc:1939] T 186ec810d02e4748aad15228b5866ac1 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:34.552930 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 205) took 5525 ms. Trace:
I20250901 01:52:34.553071 2118 rpcz_store.cc:276] 0901 01:52:29.027255 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:29.027315 (+ 60us) service_pool.cc:225] Handling call
0901 01:52:34.542314 (+5514999us) ts_tablet_manager.cc:1145] Deleting tablet 186ec810d02e4748aad15228b5866ac1
0901 01:52:34.542323 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:34.548551 (+ 6228us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:34.551684 (+ 3133us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:34.552912 (+ 1228us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5514834}
I20250901 01:52:34.553658 2118 tablet_replica.cc:331] T 3a8231590ea643b0a1f83eec4d9a8191 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:34.553900 2118 raft_consensus.cc:2241] T 3a8231590ea643b0a1f83eec4d9a8191 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:34.554116 2118 raft_consensus.cc:2270] T 3a8231590ea643b0a1f83eec4d9a8191 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:34.554702 2118 ts_tablet_manager.cc:1905] T 3a8231590ea643b0a1f83eec4d9a8191 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:34.559536 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_43ab7f84ce4f4f3fa5c1191a26a37a4a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.561426 2118 ts_tablet_manager.cc:1918] T 3a8231590ea643b0a1f83eec4d9a8191 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 01:52:34.561586 2118 log.cc:1199] T 3a8231590ea643b0a1f83eec4d9a8191 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/3a8231590ea643b0a1f83eec4d9a8191
I20250901 01:52:34.561972 2118 ts_tablet_manager.cc:1939] T 3a8231590ea643b0a1f83eec4d9a8191 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:34.562323 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 207) took 5401 ms. Trace:
I20250901 01:52:34.562443 2118 rpcz_store.cc:276] 0901 01:52:29.161088 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:29.164450 (+ 3362us) service_pool.cc:225] Handling call
0901 01:52:34.553614 (+5389164us) ts_tablet_manager.cc:1145] Deleting tablet 3a8231590ea643b0a1f83eec4d9a8191
0901 01:52:34.553623 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:34.558473 (+ 4850us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:34.561351 (+ 2878us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:34.562307 (+ 956us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5388997}
I20250901 01:52:34.562971 2118 tablet_replica.cc:331] T 5f52698fb56d472d9e91fb10cb10c69a P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:34.563201 2118 raft_consensus.cc:2241] T 5f52698fb56d472d9e91fb10cb10c69a P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:34.563395 2118 raft_consensus.cc:2270] T 5f52698fb56d472d9e91fb10cb10c69a P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:34.564641 2118 ts_tablet_manager.cc:1905] T 5f52698fb56d472d9e91fb10cb10c69a P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:34.570982 2118 ts_tablet_manager.cc:1918] T 5f52698fb56d472d9e91fb10cb10c69a P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:34.571136 2118 log.cc:1199] T 5f52698fb56d472d9e91fb10cb10c69a P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/5f52698fb56d472d9e91fb10cb10c69a
I20250901 01:52:34.572600 2778 tablet.cc:1722] T 615b2306fca84003b264da9d5f95ddab P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250901 01:52:34.575713 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 307) took 5084 ms. Trace:
W20250901 01:52:34.577659 992 consensus_peers.cc:489] T 7b16ca1daf394a6da65a4af8da82ec33 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250901 01:52:34.578336 2856 consensus_queue.cc:1035] T d0b7c84ec9394a20ad77add3808e555a P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:34.578691 2829 tablet.cc:1722] T 615b2306fca84003b264da9d5f95ddab P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
W20250901 01:52:34.586877 858 consensus_peers.cc:489] T 5f5396045979450193d461ba1334a5c9 P 00c282f779c1477d920f24dc0dc16628 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:34.589068 2118 ts_tablet_manager.cc:1939] T 5f52698fb56d472d9e91fb10cb10c69a P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:34.598645 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 209) took 5415 ms. Trace:
I20250901 01:52:34.598989 2118 rpcz_store.cc:276] 0901 01:52:29.182874 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:29.182925 (+ 51us) service_pool.cc:225] Handling call
0901 01:52:34.562922 (+5379997us) ts_tablet_manager.cc:1145] Deleting tablet 5f52698fb56d472d9e91fb10cb10c69a
0901 01:52:34.562931 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:34.568068 (+ 5137us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:34.570907 (+ 2839us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:34.598628 (+ 27721us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5379819}
I20250901 01:52:34.599419 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e3de8b290ab742fa88bb32e4227b072f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.600060 2118 tablet_replica.cc:331] T e6c10426f60f43e8bde69d9b1b9c8eb6 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:34.600620 2118 raft_consensus.cc:2241] T e6c10426f60f43e8bde69d9b1b9c8eb6 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:34.601089 2118 raft_consensus.cc:2270] T e6c10426f60f43e8bde69d9b1b9c8eb6 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:34.601085 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 186ec810d02e4748aad15228b5866ac1 (table default.Table_9d0d7a93ff15424092012d4d681b8ae0 [id=aad143eea5c2439299aad87723399c58]) successfully deleted
I20250901 01:52:34.601931 2118 ts_tablet_manager.cc:1905] T e6c10426f60f43e8bde69d9b1b9c8eb6 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:34.606644 2613 raft_consensus.cc:491] T 4b4aa063ffe948e69e47f03102581899 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:34.606787 2613 raft_consensus.cc:513] T 4b4aa063ffe948e69e47f03102581899 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:34.608580 992 leader_election.cc:336] T 4b4aa063ffe948e69e47f03102581899 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:34.609941 921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4b4aa063ffe948e69e47f03102581899" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:34.609853 2613 leader_election.cc:290] T 4b4aa063ffe948e69e47f03102581899 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:34.610283 921 raft_consensus.cc:2466] T 4b4aa063ffe948e69e47f03102581899 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:34.610610 992 leader_election.cc:304] T 4b4aa063ffe948e69e47f03102581899 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:34.610929 2775 raft_consensus.cc:2802] T 4b4aa063ffe948e69e47f03102581899 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:34.611114 2775 raft_consensus.cc:491] T 4b4aa063ffe948e69e47f03102581899 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:34.611260 2775 raft_consensus.cc:3058] T 4b4aa063ffe948e69e47f03102581899 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:34.612195 2613 raft_consensus.cc:491] T f23df6f253a943fa8e146bb8c811ef51 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:34.612361 2613 raft_consensus.cc:513] T f23df6f253a943fa8e146bb8c811ef51 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:34.613122 2829 tablet.cc:1722] T 14b466b2a5a249aa841f1b8b06301edc P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250901 01:52:34.575786 2113 rpcz_store.cc:276] 0901 01:52:29.491390 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:29.492197 (+ 807us) service_pool.cc:225] Handling call
0901 01:52:34.384515 (+4892318us) ts_tablet_manager.cc:1145] Deleting tablet 4fd0e4f9d86c4362af96ce9964b72328
0901 01:52:34.384524 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:34.413675 (+ 29151us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:34.435706 (+ 22031us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:34.575691 (+139985us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":621,"tablet-delete.queue_time_us":4877459}
I20250901 01:52:34.614114 2778 tablet.cc:1722] T 14b466b2a5a249aa841f1b8b06301edc P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
W20250901 01:52:34.614359 992 leader_election.cc:336] T f23df6f253a943fa8e146bb8c811ef51 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:34.614846 2613 leader_election.cc:290] T f23df6f253a943fa8e146bb8c811ef51 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:34.615109 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 4fd0e4f9d86c4362af96ce9964b72328 (table default.Table_2d3d2227e6a24bd2a86f457e553e698b [id=f48fd53522234290b6c0373fd7b02570]) successfully deleted
W20250901 01:52:34.619530 992 leader_election.cc:343] T f23df6f253a943fa8e146bb8c811ef51 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: f23df6f253a943fa8e146bb8c811ef51
I20250901 01:52:34.619850 992 leader_election.cc:304] T f23df6f253a943fa8e146bb8c811ef51 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:34.620167 2613 raft_consensus.cc:2747] T f23df6f253a943fa8e146bb8c811ef51 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:34.629228 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 305) took 5138 ms. Trace:
I20250901 01:52:34.629887 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 3a8231590ea643b0a1f83eec4d9a8191 (table default.Table_a4ebd8ab87c842f082da5cd04d1bddac [id=fd6d8e92d6e64a89a7d38aa761376955]) successfully deleted
I20250901 01:52:34.630107 2113 rpcz_store.cc:276] 0901 01:52:29.491176 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:29.492878 (+ 1702us) service_pool.cc:225] Handling call
0901 01:52:34.628195 (+5135317us) ts_tablet_manager.cc:1145] Deleting tablet 3a2bc5ad1cd243daa3c92fceca8ba414
0901 01:52:34.629112 (+ 917us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:34.629211 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5117939}
I20250901 01:52:34.630911 2118 ts_tablet_manager.cc:1918] T e6c10426f60f43e8bde69d9b1b9c8eb6 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 01:52:34.632709 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 5f52698fb56d472d9e91fb10cb10c69a (table default.Table_83cf2d86a1284544ba2e7aa9ebb66bb7 [id=264d1a223e7d43c8a7a661db6722cb17]) successfully deleted
I20250901 01:52:34.635211 2613 raft_consensus.cc:491] T 69f61ec283484f9d8b3eebcea27f741b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:34.635324 2613 raft_consensus.cc:513] T 69f61ec283484f9d8b3eebcea27f741b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:34.635856 2613 leader_election.cc:290] T 69f61ec283484f9d8b3eebcea27f741b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:34.637256 992 leader_election.cc:336] T 69f61ec283484f9d8b3eebcea27f741b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:34.641080 2113 tablet_replica.cc:331] T 69f61ec283484f9d8b3eebcea27f741b P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
W20250901 01:52:34.633217 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 3a2bc5ad1cd243daa3c92fceca8ba414 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3a2bc5ad1cd243daa3c92fceca8ba414 already in progress: creating tablet
I20250901 01:52:34.641826 2113 raft_consensus.cc:2241] T 69f61ec283484f9d8b3eebcea27f741b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:34.642256 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f4651f9cf73c431baabc68d2896c6427"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.652582 2113 raft_consensus.cc:2270] T 69f61ec283484f9d8b3eebcea27f741b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:34.644344 1489 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 5f5396045979450193d461ba1334a5c9
W20250901 01:52:34.657338 1489 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 5f5396045979450193d461ba1334a5c9: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:34.644450 1482 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_a4f42e5055e6416caf95b93122822c6f" } new_table_name: "default.Table_9f7d6f3bd6c6495280e3bd7725bc8f6d" modify_external_catalogs: true
I20250901 01:52:34.658057 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c784c0f4dcaa491c80924a992614ebcc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.658839 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b121fbf3e2f542ada6be24d2a2ec1bc9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:34.659171 992 consensus_peers.cc:489] T 482a9a2cb81f448bb2823892e35920d4 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250901 01:52:34.659572 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3892447a03e049cfa95fcbc41b3e55c6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.645373 1484 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_82d91a1ba2eb4265896824028dec8d3a" } modify_external_catalogs: true
I20250901 01:52:34.660100 1484 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_82d91a1ba2eb4265896824028dec8d3a" } modify_external_catalogs: true
I20250901 01:52:34.660182 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_11370eb09c15427b95690feb7e871740"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.661379 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 5f5396045979450193d461ba1334a5c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:34 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:34.645925 1486 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: default.Table_359a26011f684089ac6da2a62e032680 [id=a4d0af97df5949cca940a81a41b8e454] alter complete (version 2)
I20250901 01:52:34.662025 1489 master_service.cc:948] ReplaceTablet: received request to replace tablet eee34044c39d4013baf78d3705489626 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:34.662748 1489 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_0927e3ea788d43bda5afeda483d87176" } modify_external_catalogs: true
I20250901 01:52:34.663164 1489 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_0927e3ea788d43bda5afeda483d87176" } modify_external_catalogs: true
I20250901 01:52:34.664673 1492 master_service.cc:948] ReplaceTablet: received request to replace tablet 88ceba7f48b04d7885cc0164668700a5 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:34.665310 1492 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_aee74198b51d4f089538c4f2bf3ce4dc" } new_table_name: "default.Table_4bd66ab99a454974aa0f7f4290edfda2" modify_external_catalogs: true
W20250901 01:52:34.670078 1491 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f8f923ddcd2740bcb7e521f4f57c4034 has no consensus state
I20250901 01:52:34.670454 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_64b5fcb3346d4d0e8fb1b8b67e70b749"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.671159 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4624eddc8f9f48f5bca05d844e9af163"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.671790 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b8ca275b709542f3b493ae2db210e552"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.672461 1488 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 4b4aa063ffe948e69e47f03102581899
W20250901 01:52:34.672631 1488 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 4b4aa063ffe948e69e47f03102581899: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:34.673127 1483 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet eea6bfabe6cc4f83905c24f0f7a5dfc9
I20250901 01:52:34.676577 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 4b4aa063ffe948e69e47f03102581899 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:34 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:34.639564 921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69f61ec283484f9d8b3eebcea27f741b" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:34.678234 921 raft_consensus.cc:2466] T 69f61ec283484f9d8b3eebcea27f741b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:34.662663 2871 raft_consensus.cc:491] T 508406601c994e42a5c5b1f665e74957 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:34.678515 2871 raft_consensus.cc:513] T 508406601c994e42a5c5b1f665e74957 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:34.679091 2871 leader_election.cc:290] T 508406601c994e42a5c5b1f665e74957 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:34.679593 992 leader_election.cc:304] T 69f61ec283484f9d8b3eebcea27f741b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
W20250901 01:52:34.679772 992 raft_consensus.cc:2684] T 69f61ec283484f9d8b3eebcea27f741b P fd27ab6665304aaea53adbd102dd4f58: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250901 01:52:34.680051 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "508406601c994e42a5c5b1f665e74957" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:34.680205 1052 raft_consensus.cc:2466] T 508406601c994e42a5c5b1f665e74957 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00c282f779c1477d920f24dc0dc16628 in term 0.
I20250901 01:52:34.680668 859 leader_election.cc:304] T 508406601c994e42a5c5b1f665e74957 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:34.682178 2871 raft_consensus.cc:2802] T 508406601c994e42a5c5b1f665e74957 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:34.682262 2871 raft_consensus.cc:491] T 508406601c994e42a5c5b1f665e74957 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:34.682312 2871 raft_consensus.cc:3058] T 508406601c994e42a5c5b1f665e74957 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
W20250901 01:52:34.684628 858 leader_election.cc:336] T 508406601c994e42a5c5b1f665e74957 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:34.686524 2871 raft_consensus.cc:513] T 508406601c994e42a5c5b1f665e74957 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:34.688006 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "508406601c994e42a5c5b1f665e74957" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58"
I20250901 01:52:34.688162 1052 raft_consensus.cc:3058] T 508406601c994e42a5c5b1f665e74957 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:34.690119 2753 consensus_queue.cc:1035] T 5f5396045979450193d461ba1334a5c9 P 00c282f779c1477d920f24dc0dc16628 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:34.691234 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 4b4aa063ffe948e69e47f03102581899 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:34 UTC) from {username='slave'} at 127.0.0.1:34472
W20250901 01:52:34.692724 1473 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet f8f923ddcd2740bcb7e521f4f57c4034 (table default.Table_e09ac1a9050547038400d4b52299d22f [id=b9d81ddd38fe47838877b010be2b40d0], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet f8f923ddcd2740bcb7e521f4f57c4034 has no consensus state
I20250901 01:52:34.692065 2871 leader_election.cc:290] T 508406601c994e42a5c5b1f665e74957 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:34.692938 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 5f5396045979450193d461ba1334a5c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:34 UTC) from {username='slave'} at 127.0.0.1:34472
W20250901 01:52:34.692942 858 leader_election.cc:336] T 508406601c994e42a5c5b1f665e74957 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:34.634460 2118 log.cc:1199] T e6c10426f60f43e8bde69d9b1b9c8eb6 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/e6c10426f60f43e8bde69d9b1b9c8eb6
I20250901 01:52:34.696250 2118 ts_tablet_manager.cc:1939] T e6c10426f60f43e8bde69d9b1b9c8eb6 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:34.697038 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 208) took 5514 ms. Trace:
I20250901 01:52:34.697547 2118 rpcz_store.cc:276] 0901 01:52:29.182733 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:29.184625 (+ 1892us) service_pool.cc:225] Handling call
0901 01:52:34.600008 (+5415383us) ts_tablet_manager.cc:1145] Deleting tablet e6c10426f60f43e8bde69d9b1b9c8eb6
0901 01:52:34.600021 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:34.622334 (+ 22313us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:34.630823 (+ 8489us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:34.697024 (+ 66201us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5415182}
I20250901 01:52:34.698653 2118 tablet_replica.cc:331] T 3a2bc5ad1cd243daa3c92fceca8ba414 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:34.699494 2118 raft_consensus.cc:2241] T 3a2bc5ad1cd243daa3c92fceca8ba414 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:34.699608 2778 tablet.cc:1722] T 615b2306fca84003b264da9d5f95ddab P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250901 01:52:34.700323 2829 tablet.cc:1722] T 615b2306fca84003b264da9d5f95ddab P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250901 01:52:34.702073 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskddMNdm/build/asan/bin/kudu
/tmp/dist-test-taskddMNdm/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:42375
--webserver_interface=127.0.104.61
--webserver_port=46533
--builtin_ntp_servers=127.0.104.20:46333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:43871,127.0.104.61:42375,127.0.104.60:44231
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250901 01:52:34.639269 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_26c51c6f9b4a4250a3423787b364f04a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.705013 2856 consensus_queue.cc:1035] T 93bcac1fd95d44dc927abfca81692921 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:34.707470 2118 raft_consensus.cc:2270] T 3a2bc5ad1cd243daa3c92fceca8ba414 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:34.707983 2118 ts_tablet_manager.cc:1905] T 3a2bc5ad1cd243daa3c92fceca8ba414 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:34.713708 2118 ts_tablet_manager.cc:1918] T 3a2bc5ad1cd243daa3c92fceca8ba414 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:34.714627 2113 ts_tablet_manager.cc:1905] T 69f61ec283484f9d8b3eebcea27f741b P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:34.714690 2118 log.cc:1199] T 3a2bc5ad1cd243daa3c92fceca8ba414 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/3a2bc5ad1cd243daa3c92fceca8ba414
I20250901 01:52:34.714993 2118 ts_tablet_manager.cc:1939] T 3a2bc5ad1cd243daa3c92fceca8ba414 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:34.715260 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 210) took 5459 ms. Trace:
I20250901 01:52:34.715327 2118 rpcz_store.cc:276] 0901 01:52:29.256182 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:29.256242 (+ 60us) service_pool.cc:225] Handling call
0901 01:52:34.698608 (+5442366us) ts_tablet_manager.cc:1145] Deleting tablet 3a2bc5ad1cd243daa3c92fceca8ba414
0901 01:52:34.698617 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:34.710924 (+ 12307us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:34.713632 (+ 2708us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:34.715245 (+ 1613us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5442174}
I20250901 01:52:34.715632 2778 tablet.cc:1722] T 7954393c8c1c4457ab8bb723c03a9c8b P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 7
I20250901 01:52:34.715797 2118 tablet_replica.cc:331] T 4fd0e4f9d86c4362af96ce9964b72328 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:34.715994 2118 raft_consensus.cc:2241] T 4fd0e4f9d86c4362af96ce9964b72328 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:34.716164 2118 raft_consensus.cc:2270] T 4fd0e4f9d86c4362af96ce9964b72328 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:34.716698 2118 ts_tablet_manager.cc:1905] T 4fd0e4f9d86c4362af96ce9964b72328 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:34.716718 2775 raft_consensus.cc:513] T 4b4aa063ffe948e69e47f03102581899 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:34.717255 2775 leader_election.cc:290] T 4b4aa063ffe948e69e47f03102581899 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:34.717895 921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4b4aa063ffe948e69e47f03102581899" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:34.718046 921 raft_consensus.cc:3058] T 4b4aa063ffe948e69e47f03102581899 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:34.718122 2829 tablet.cc:1722] T 7954393c8c1c4457ab8bb723c03a9c8b P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 7
I20250901 01:52:34.718904 1052 raft_consensus.cc:2466] T 508406601c994e42a5c5b1f665e74957 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c282f779c1477d920f24dc0dc16628 in term 1.
I20250901 01:52:34.719148 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet e6c10426f60f43e8bde69d9b1b9c8eb6 (table default.Table_7c21b68ef2e74136a807c1dd90e17f4e [id=d3dbeb4258424133844e20b181ee44e4]) successfully deleted
I20250901 01:52:34.720790 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2d3205eaa5ce4c66a8385064d7dacc93"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.721614 859 leader_election.cc:304] T 508406601c994e42a5c5b1f665e74957 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:34.722095 2871 raft_consensus.cc:2802] T 508406601c994e42a5c5b1f665e74957 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:34.722209 2871 raft_consensus.cc:695] T 508406601c994e42a5c5b1f665e74957 P 00c282f779c1477d920f24dc0dc16628 [term 1 LEADER]: Becoming Leader. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Running, Role: LEADER
I20250901 01:52:34.722188 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1be4a2ec1bad453590d7ce080542e1df"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.722429 2871 consensus_queue.cc:237] T 508406601c994e42a5c5b1f665e74957 P 00c282f779c1477d920f24dc0dc16628 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:34.719417 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 3a2bc5ad1cd243daa3c92fceca8ba414 (table default.Table_122585be3bdc45fb9498d02564fb8567 [id=e7b9f3937d0241b291db9ccbaa523da2]) successfully deleted
I20250901 01:52:34.725345 1484 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet d602c8986c75492ead9936ac04e2622d
E20250901 01:52:34.726507 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 0c88c339e59a4c1da21756cba0645bdd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0927e3ea788d43bda5afeda483d87176'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.726675 1696 catalog_manager.cc:6205] Error selecting replicas for tablet a656adbf1b29431687e0989f0db3459c, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2d3d2227e6a24bd2a86f457e553e698b'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.726785 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 494b6488aa0d4355b4caa46004267465, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7c21b68ef2e74136a807c1dd90e17f4e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.726927 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 01deaae6abb141789f0cd240f3058c73, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7baa92b46d98454a891222be49a6ca48'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.727013 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 0ee3f35392954817b350e5f48653be78, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_442e11b9211c4b209c68f653d7bfc313'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.727118 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae4844183a3c4a04a775e6fd878391a3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_aee74198b51d4f089538c4f2bf3ce4dc'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.727196 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9745582cebfe47b58c21d0191ab4cb9b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77728af9eee44f29a1e2131d5b75b7ca'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.727298 1696 catalog_manager.cc:6205] Error selecting replicas for tablet e2ed32b3292141278af28026528e96c2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b5077f55951d48be92a67407ffe26a08'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.727404 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9f62a9d97f514747b940b488907369f3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d53a456dbca64daf92aeb5fb4517b417'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.727510 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5a7fa68d29644ae8afd2711e39dde59a, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d24e3a18ef4843fdb7b74ce23d47cf9e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.727604 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 415c9d7b552b42c384b444a20491f65c, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_97c44ea002094df8a6af3dccd0099c39'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.727700 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ff2bd017417c41b5b8225b8f00283e63, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0c7355975d6b44dbbe3a77df942b4d43'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.727802 1696 catalog_manager.cc:6205] Error selecting replicas for tablet dcc60bb000f440f1aca9133928b5ce20, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2fbca8463d6546c9b8f5b018fc60251d'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.727909 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 465c02a130c54e09b5a74fb4dd2ef762, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ed6fd284be6f4ae3a008e21e7ce17930'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.727996 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f8f923ddcd2740bcb7e521f4f57c4034, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_e09ac1a9050547038400d4b52299d22f'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.728102 1696 catalog_manager.cc:6205] Error selecting replicas for tablet b1ba0e19dc0b462e80c537bf2d183f5b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_66b111271c15410aab23682f8f6081ac'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.728189 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 8e10a01e77d14cf28c2c5e0205e595b9, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_83cf2d86a1284544ba2e7aa9ebb66bb7'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.728291 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5f0e054f62de4cf6bc6f69ad6664b192, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ef6832d0845044ed9f165ca3541b4c3f'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.728374 1696 catalog_manager.cc:6205] Error selecting replicas for tablet fd2ce82a4ee24d0faa44980a3967e187, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_63d0f2ead9d946d39ce40162ebd769d6'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.728696 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae9a4ad4fdfb4771ba23a91f7007447e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_5e84ae9a9c2a4084a1f9c5ffb4594de2'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.728797 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 086aa948f93644da8e5835b0fac87ddd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b4fbc34da806475c912052d0fab3fc53'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.728894 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 93aa22afc0ad4d7fa24807ea3c5d5c21, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0f4b4094d2974b7591875375eda6bed0'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.728974 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9074543dea9949e7a085717b40913348, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_53e30205c5fd488eb3b48c685a159d04'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.729084 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 3a62034e688348ed9e2ac9de009ed3a2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_e225c82a2d6045f6b16915cdb72f5734'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.729182 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 18498c9cb09141d08c1c5c0854b63ebd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77f4810bc83c4d948531da2a45a3792e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.729311 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f5e3316adeae47cd848833dd84b972bb, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7570c3012f8a4ae7bcfc7346eadac92a'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.729414 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 81e8c6567953484ab000fe2eb4398301, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_a920a3ffbabf4d83a6563c3423cde12a'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.729526 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 6ce9521b925b4f97bb4a810ce77e147e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_9e04f976a93f4d30baccf90fa3622dfe'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.729624 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2f61edb50d7a42198bfd70639c7bf4de, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_a5c783cbdb694af69047b31d99ff8def'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:34.730762 1485 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: default.Table_3d5afc22c4e14d6990aea31564dfde20 [id=557afe29976d40f6bfb6ae4ca348f887] alter complete (version 2)
W20250901 01:52:34.733841 992 leader_election.cc:336] T 4b4aa063ffe948e69e47f03102581899 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:34.734316 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet e1c8fe96a6ba4edba4f09c00f7add75e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:19 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:34.738209 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d6f5af02c635423baadcc693eb434487"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.741317 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 764bfeb2a7d14c40815b5ec1b07bdd62 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:23 UTC) from {username='slave'} at 127.0.0.1:34472
W20250901 01:52:34.741353 992 consensus_peers.cc:489] T 7954393c8c1c4457ab8bb723c03a9c8b P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250901 01:52:34.742408 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_0db8e6394253469db57907c799e58007"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.744901 921 raft_consensus.cc:2466] T 4b4aa063ffe948e69e47f03102581899 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:34.746153 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e1198c0a658f444284a46eca3034ffa6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.747189 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_0120f1237f0749fa8d83c9ce87399e64"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.748001 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_19d0def8717d4340840d5f3203de37ed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.749583 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b1062c6f55e3448c9bc2a93d03b83151"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.750502 992 leader_election.cc:304] T 4b4aa063ffe948e69e47f03102581899 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:34.750756 2856 raft_consensus.cc:2802] T 4b4aa063ffe948e69e47f03102581899 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:34.751246 2856 raft_consensus.cc:695] T 4b4aa063ffe948e69e47f03102581899 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:34.751564 2856 consensus_queue.cc:237] T 4b4aa063ffe948e69e47f03102581899 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:34.752272 992 consensus_peers.cc:489] T 93bcac1fd95d44dc927abfca81692921 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:34.758965 1489 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 0c88c339e59a4c1da21756cba0645bdd
W20250901 01:52:34.773013 992 consensus_peers.cc:489] T fae2779fac164b9f8052c3666db803b8 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250901 01:52:34.773624 1492 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ae4844183a3c4a04a775e6fd878391a3 has no consensus state
I20250901 01:52:34.777241 1151 tablet_bootstrap.cc:492] T ae21b9ae2e3b472eb67fb5f96388a50f P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:34.780009 1151 ts_tablet_manager.cc:1397] T ae21b9ae2e3b472eb67fb5f96388a50f P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.255s user 0.010s sys 0.023s
I20250901 01:52:34.779889 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_6a4ffdeef2014208bf49bf29a51c799f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.781028 1151 raft_consensus.cc:357] T ae21b9ae2e3b472eb67fb5f96388a50f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:34.781297 1151 raft_consensus.cc:383] T ae21b9ae2e3b472eb67fb5f96388a50f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:34.781402 1151 raft_consensus.cc:738] T ae21b9ae2e3b472eb67fb5f96388a50f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:34.781700 1151 consensus_queue.cc:260] T ae21b9ae2e3b472eb67fb5f96388a50f P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:34.783000 858 consensus_peers.cc:489] T 0e4009963cfe42d6ba164fc828897f09 P 00c282f779c1477d920f24dc0dc16628 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:34.785897 1151 ts_tablet_manager.cc:1428] T ae21b9ae2e3b472eb67fb5f96388a50f P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s
E20250901 01:52:34.788493 1696 catalog_manager.cc:6205] Error selecting replicas for tablet a656adbf1b29431687e0989f0db3459c, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2d3d2227e6a24bd2a86f457e553e698b'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.789022 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 494b6488aa0d4355b4caa46004267465, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7c21b68ef2e74136a807c1dd90e17f4e'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:34.792140 2608 raft_consensus.cc:491] T 44af4764d4f64e44aec84ccb84be6f5f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:34.792279 2608 raft_consensus.cc:513] T 44af4764d4f64e44aec84ccb84be6f5f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:34.792786 2608 leader_election.cc:290] T 44af4764d4f64e44aec84ccb84be6f5f P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:34.789250 2878 raft_consensus.cc:491] T b2b172757e50404aae78e818921a7d9d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:34.793990 2878 raft_consensus.cc:513] T b2b172757e50404aae78e818921a7d9d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:34.794471 858 leader_election.cc:336] T 44af4764d4f64e44aec84ccb84be6f5f P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:34.794528 2878 leader_election.cc:290] T b2b172757e50404aae78e818921a7d9d P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:34.795665 1151 tablet_bootstrap.cc:492] T 44af4764d4f64e44aec84ccb84be6f5f P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:34.790486 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a8c344f8fce44e07acdf5fadad62d34f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.800943 1151 tablet_bootstrap.cc:654] T 44af4764d4f64e44aec84ccb84be6f5f P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:34.800871 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9d94df14784047c8a2cb23ec6b72471e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.801646 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f07346443c8546e58a528c6e807b13fc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.790990 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_43dd42efe91742cf8d16a214f887a1b5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.797048 1485 master_service.cc:948] ReplaceTablet: received request to replace tablet 2df2240631844defa433270a3829d65b from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:34.796995 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_48c98f1559ef4608a8a0560991665bab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.797518 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4a237149441542fdb1329836386f0f35"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.797603 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a8d7d51b2e5548a39c858f42ba2fc342"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.806432 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "44af4764d4f64e44aec84ccb84be6f5f" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:34.807487 922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2b172757e50404aae78e818921a7d9d" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:34.807685 922 raft_consensus.cc:2466] T b2b172757e50404aae78e818921a7d9d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:34.797823 1482 master_service.cc:948] ReplaceTablet: received request to replace tablet 5dd0c6b806574caf9126dad94833fcf7 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:34.798142 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ee2dd52fabe74177a516cb22a5c396b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.799701 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b37b743e95a44ef4a7dddc8ba3514d77"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:34.815775 859 leader_election.cc:343] T 44af4764d4f64e44aec84ccb84be6f5f P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:34.815898 859 leader_election.cc:304] T 44af4764d4f64e44aec84ccb84be6f5f P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:34.816488 2357 raft_consensus.cc:2747] T 44af4764d4f64e44aec84ccb84be6f5f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:34.817240 992 leader_election.cc:336] T b2b172757e50404aae78e818921a7d9d P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:34.817638 992 leader_election.cc:304] T b2b172757e50404aae78e818921a7d9d P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:34.818351 2780 raft_consensus.cc:2802] T b2b172757e50404aae78e818921a7d9d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:34.818475 2780 raft_consensus.cc:491] T b2b172757e50404aae78e818921a7d9d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:34.818589 2780 raft_consensus.cc:3058] T b2b172757e50404aae78e818921a7d9d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
E20250901 01:52:34.818101 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 01deaae6abb141789f0cd240f3058c73, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7baa92b46d98454a891222be49a6ca48'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.819326 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 0ee3f35392954817b350e5f48653be78, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_442e11b9211c4b209c68f653d7bfc313'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.819487 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae4844183a3c4a04a775e6fd878391a3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_4bd66ab99a454974aa0f7f4290edfda2'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.819640 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9745582cebfe47b58c21d0191ab4cb9b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77728af9eee44f29a1e2131d5b75b7ca'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.819800 1696 catalog_manager.cc:6205] Error selecting replicas for tablet e2ed32b3292141278af28026528e96c2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b5077f55951d48be92a67407ffe26a08'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.820922 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9f62a9d97f514747b940b488907369f3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d53a456dbca64daf92aeb5fb4517b417'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.821161 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5a7fa68d29644ae8afd2711e39dde59a, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d24e3a18ef4843fdb7b74ce23d47cf9e'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:34.821378 2780 raft_consensus.cc:513] T b2b172757e50404aae78e818921a7d9d P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:34.822036 2780 leader_election.cc:290] T b2b172757e50404aae78e818921a7d9d P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
E20250901 01:52:34.823154 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 415c9d7b552b42c384b444a20491f65c, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_97c44ea002094df8a6af3dccd0099c39'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.823370 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ff2bd017417c41b5b8225b8f00283e63, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0c7355975d6b44dbbe3a77df942b4d43'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.824147 1696 catalog_manager.cc:6205] Error selecting replicas for tablet dcc60bb000f440f1aca9133928b5ce20, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2fbca8463d6546c9b8f5b018fc60251d'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.824326 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 465c02a130c54e09b5a74fb4dd2ef762, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ed6fd284be6f4ae3a008e21e7ce17930'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:34.825249 2113 ts_tablet_manager.cc:1918] T 69f61ec283484f9d8b3eebcea27f741b P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
E20250901 01:52:34.825477 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f8f923ddcd2740bcb7e521f4f57c4034, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_e09ac1a9050547038400d4b52299d22f'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:34.820668 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b8c48fe1878b40c999f7e1d80b2ce059"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:34.827222 1696 catalog_manager.cc:6205] Error selecting replicas for tablet b1ba0e19dc0b462e80c537bf2d183f5b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_66b111271c15410aab23682f8f6081ac'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.827694 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 8e10a01e77d14cf28c2c5e0205e595b9, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_83cf2d86a1284544ba2e7aa9ebb66bb7'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.827904 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5f0e054f62de4cf6bc6f69ad6664b192, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ef6832d0845044ed9f165ca3541b4c3f'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:34.829682 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d1842f665ae64bf28a73b08bcc516a89"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.834280 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_35efe985b9df4fb39aead3f2b72ed8dd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.835177 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5346c08d101e493b91d748c90cacebc5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.835369 922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2b172757e50404aae78e818921a7d9d" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:34.835548 922 raft_consensus.cc:3058] T b2b172757e50404aae78e818921a7d9d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:34.835906 1491 master_service.cc:948] ReplaceTablet: received request to replace tablet abf6e3be60f74872b2308dc95bf9cd37 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:34.830173 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a55acaba91a7405687589e45d5d0d2c9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.838267 922 raft_consensus.cc:2466] T b2b172757e50404aae78e818921a7d9d P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:34.830580 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_09ce19d87239476fb7467a7f8e4df8a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.830845 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_bc6046fca439437196d9dbab55807c60"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.831255 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_82fdf23d53b54788a5fbcaa5f3583f51"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.831808 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2e8b19d723f54e288e5fd62d49e32a1e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.832322 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_55f85e1ead2e4844950fc8b9a0e45e8c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.832806 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8ef0a05b1db54d89ba7cbb6e5f106771"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:34.846267 1696 catalog_manager.cc:6205] Error selecting replicas for tablet fd2ce82a4ee24d0faa44980a3967e187, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_63d0f2ead9d946d39ce40162ebd769d6'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:34.846496 2113 log.cc:1199] T 69f61ec283484f9d8b3eebcea27f741b P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/69f61ec283484f9d8b3eebcea27f741b
E20250901 01:52:34.846773 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae9a4ad4fdfb4771ba23a91f7007447e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_5e84ae9a9c2a4084a1f9c5ffb4594de2'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:34.847241 2113 ts_tablet_manager.cc:1939] T 69f61ec283484f9d8b3eebcea27f741b P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
E20250901 01:52:34.847427 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 086aa948f93644da8e5835b0fac87ddd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b4fbc34da806475c912052d0fab3fc53'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:34.847448 2118 ts_tablet_manager.cc:1918] T 4fd0e4f9d86c4362af96ce9964b72328 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 01:52:34.847554 2118 log.cc:1199] T 4fd0e4f9d86c4362af96ce9964b72328 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/4fd0e4f9d86c4362af96ce9964b72328
E20250901 01:52:34.847707 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 082db24bb3fd4089950504145a20a075, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7f5b278599994063862981b6b041ca8c'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.847841 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 93aa22afc0ad4d7fa24807ea3c5d5c21, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0f4b4094d2974b7591875375eda6bed0'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.847985 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9074543dea9949e7a085717b40913348, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_53e30205c5fd488eb3b48c685a159d04'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.848101 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 3a62034e688348ed9e2ac9de009ed3a2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_e225c82a2d6045f6b16915cdb72f5734'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.848238 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 18498c9cb09141d08c1c5c0854b63ebd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77f4810bc83c4d948531da2a45a3792e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.848376 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f5e3316adeae47cd848833dd84b972bb, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7570c3012f8a4ae7bcfc7346eadac92a'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.848567 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 81e8c6567953484ab000fe2eb4398301, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_a920a3ffbabf4d83a6563c3423cde12a'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.848691 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 6ce9521b925b4f97bb4a810ce77e147e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_9e04f976a93f4d30baccf90fa3622dfe'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:34.848822 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2f61edb50d7a42198bfd70639c7bf4de, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_a5c783cbdb694af69047b31d99ff8def'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:34.851125 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 315) took 5265 ms. Trace:
I20250901 01:52:34.851222 2113 rpcz_store.cc:276] 0901 01:52:29.586132 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:29.586423 (+ 291us) service_pool.cc:225] Handling call
0901 01:52:34.639800 (+5053377us) ts_tablet_manager.cc:1145] Deleting tablet 69f61ec283484f9d8b3eebcea27f741b
0901 01:52:34.639814 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:34.772997 (+133183us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:34.825143 (+ 52146us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:34.851101 (+ 25958us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5053157}
I20250901 01:52:34.851634 2113 tablet_replica.cc:331] T fecf7d538b4c4c619182d5eccb44dd8b P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:34.851817 2113 raft_consensus.cc:2241] T fecf7d538b4c4c619182d5eccb44dd8b P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:34.852773 2118 ts_tablet_manager.cc:1939] T 4fd0e4f9d86c4362af96ce9964b72328 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:34.854331 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 211) took 5593 ms. Trace:
I20250901 01:52:34.854465 2118 rpcz_store.cc:276] 0901 01:52:29.260445 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:29.260540 (+ 95us) service_pool.cc:225] Handling call
0901 01:52:34.715699 (+5455159us) ts_tablet_manager.cc:1145] Deleting tablet 4fd0e4f9d86c4362af96ce9964b72328
0901 01:52:34.715708 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:34.844473 (+128765us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:34.847366 (+ 2893us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:34.854312 (+ 6946us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":896,"tablet-delete.queue_time_us":5454847}
I20250901 01:52:34.857170 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 212) took 5595 ms. Trace:
I20250901 01:52:34.857306 2118 rpcz_store.cc:276] 0901 01:52:29.261513 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:29.261564 (+ 51us) service_pool.cc:225] Handling call
0901 01:52:34.854936 (+5593372us) ts_tablet_manager.cc:1145] Deleting tablet ad1a6be9fcc045d3bb4e71a2f6bd5ccd
0901 01:52:34.857058 (+ 2122us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:34.857152 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5593199}
I20250901 01:52:34.858660 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 213) took 5596 ms. Trace:
I20250901 01:52:34.858790 2118 rpcz_store.cc:276] 0901 01:52:29.262648 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:29.262693 (+ 45us) service_pool.cc:225] Handling call
0901 01:52:34.857726 (+5595033us) ts_tablet_manager.cc:1145] Deleting tablet 121803e79ab64ec0b948b448646b9d0a
0901 01:52:34.858572 (+ 846us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:34.858639 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5594894}
I20250901 01:52:34.859253 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 216) took 5596 ms. Trace:
I20250901 01:52:34.859373 2118 rpcz_store.cc:276] 0901 01:52:29.263195 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:29.263294 (+ 99us) service_pool.cc:225] Handling call
0901 01:52:34.859176 (+5595882us) ts_tablet_manager.cc:1145] Deleting tablet b7b802f976f3471faaaecc964ec87e48
0901 01:52:34.859184 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:34.859238 (+ 54us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5595729}
I20250901 01:52:34.859864 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 217) took 5596 ms. Trace:
I20250901 01:52:34.859993 2118 rpcz_store.cc:276] 0901 01:52:29.263285 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:29.263478 (+ 193us) service_pool.cc:225] Handling call
0901 01:52:34.859769 (+5596291us) ts_tablet_manager.cc:1145] Deleting tablet 2240302d9178445d8700e35f5ea274f7
0901 01:52:34.859780 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:34.859849 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5596117}
I20250901 01:52:34.860666 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 215) took 5597 ms. Trace:
I20250901 01:52:34.860402 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 4fd0e4f9d86c4362af96ce9964b72328 (table default.Table_2d3d2227e6a24bd2a86f457e553e698b [id=f48fd53522234290b6c0373fd7b02570]) successfully deleted
I20250901 01:52:34.860797 2118 rpcz_store.cc:276] 0901 01:52:29.263104 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:29.263571 (+ 467us) service_pool.cc:225] Handling call
0901 01:52:34.860587 (+5597016us) ts_tablet_manager.cc:1145] Deleting tablet 0cb3262973934c8689b851adad60afe0
0901 01:52:34.860596 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:34.860648 (+ 52us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5596908}
I20250901 01:52:34.861199 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 218) took 5597 ms. Trace:
W20250901 01:52:34.861200 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet ad1a6be9fcc045d3bb4e71a2f6bd5ccd because the tablet was not found. No further retry: Not found: Tablet not found: ad1a6be9fcc045d3bb4e71a2f6bd5ccd
I20250901 01:52:34.861286 2118 rpcz_store.cc:276] 0901 01:52:29.263396 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:29.263682 (+ 286us) service_pool.cc:225] Handling call
0901 01:52:34.861123 (+5597441us) ts_tablet_manager.cc:1145] Deleting tablet af2c4bcbb00741f789edee109252d8cb
0901 01:52:34.861133 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:34.861185 (+ 52us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5597289}
W20250901 01:52:34.861550 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 121803e79ab64ec0b948b448646b9d0a because the tablet was not found. No further retry: Not found: Tablet not found: 121803e79ab64ec0b948b448646b9d0a
I20250901 01:52:34.861694 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 214) took 5598 ms. Trace:
W20250901 01:52:34.861833 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet b7b802f976f3471faaaecc964ec87e48 because the tablet was not found. No further retry: Not found: Tablet not found: b7b802f976f3471faaaecc964ec87e48
I20250901 01:52:34.861780 2118 rpcz_store.cc:276] 0901 01:52:29.262993 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:29.263696 (+ 703us) service_pool.cc:225] Handling call
0901 01:52:34.861621 (+5597925us) ts_tablet_manager.cc:1145] Deleting tablet b6daeff0070a4db08e90020752d21c29
0901 01:52:34.861630 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:34.861680 (+ 50us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":38,"tablet-delete.queue_time_us":5597781}
W20250901 01:52:34.862166 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 0cb3262973934c8689b851adad60afe0 because the tablet was not found. No further retry: Not found: Tablet not found: 0cb3262973934c8689b851adad60afe0
I20250901 01:52:34.862176 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 219) took 5598 ms. Trace:
I20250901 01:52:34.862243 2118 rpcz_store.cc:276] 0901 01:52:29.263517 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:29.264246 (+ 729us) service_pool.cc:225] Handling call
0901 01:52:34.862103 (+5597857us) ts_tablet_manager.cc:1145] Deleting tablet 7f64ebd080944190be2e64d02d70a8a7
0901 01:52:34.862112 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:34.862162 (+ 50us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5597659}
W20250901 01:52:34.862473 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet af2c4bcbb00741f789edee109252d8cb because the tablet was not found. No further retry: Not found: Tablet not found: af2c4bcbb00741f789edee109252d8cb
I20250901 01:52:34.862628 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 220) took 5597 ms. Trace:
I20250901 01:52:34.862701 2118 rpcz_store.cc:276] 0901 01:52:29.264698 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:29.264784 (+ 86us) service_pool.cc:225] Handling call
0901 01:52:34.862558 (+5597774us) ts_tablet_manager.cc:1145] Deleting tablet 25ce0fe8c9544f828694184d19d13df9
0901 01:52:34.862567 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:34.862614 (+ 47us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5597601}
W20250901 01:52:34.862920 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet b6daeff0070a4db08e90020752d21c29 because the tablet was not found. No further retry: Not found: Tablet not found: b6daeff0070a4db08e90020752d21c29
I20250901 01:52:34.863067 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 222) took 5598 ms. Trace:
I20250901 01:52:34.863148 2118 rpcz_store.cc:276] 0901 01:52:29.264916 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:29.264973 (+ 57us) service_pool.cc:225] Handling call
0901 01:52:34.862998 (+5598025us) ts_tablet_manager.cc:1145] Deleting tablet c58edaa76ebf47dfb1d9d5fab6de7dfa
0901 01:52:34.863006 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:34.863054 (+ 48us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5597847}
W20250901 01:52:34.863210 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 7f64ebd080944190be2e64d02d70a8a7 because the tablet was not found. No further retry: Not found: Tablet not found: 7f64ebd080944190be2e64d02d70a8a7
W20250901 01:52:34.863463 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 25ce0fe8c9544f828694184d19d13df9 because the tablet was not found. No further retry: Not found: Tablet not found: 25ce0fe8c9544f828694184d19d13df9
I20250901 01:52:34.863521 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 221) took 5598 ms. Trace:
I20250901 01:52:34.863617 2118 rpcz_store.cc:276] 0901 01:52:29.264818 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:29.265071 (+ 253us) service_pool.cc:225] Handling call
0901 01:52:34.863450 (+5598379us) ts_tablet_manager.cc:1145] Deleting tablet eaa521c52a2f4408b59770bd2a05907c
0901 01:52:34.863459 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:34.863507 (+ 48us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5598250}
W20250901 01:52:34.863845 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet c58edaa76ebf47dfb1d9d5fab6de7dfa because the tablet was not found. No further retry: Not found: Tablet not found: c58edaa76ebf47dfb1d9d5fab6de7dfa
I20250901 01:52:34.864005 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 223) took 5599 ms. Trace:
I20250901 01:52:34.864097 2118 rpcz_store.cc:276] 0901 01:52:29.265022 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:29.265744 (+ 722us) service_pool.cc:225] Handling call
0901 01:52:34.863931 (+5598187us) ts_tablet_manager.cc:1145] Deleting tablet e249474dcbf642189560c91ec3a99fdf
0901 01:52:34.863939 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:34.863991 (+ 52us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5597990}
W20250901 01:52:34.864313 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet eaa521c52a2f4408b59770bd2a05907c because the tablet was not found. No further retry: Not found: Tablet not found: eaa521c52a2f4408b59770bd2a05907c
W20250901 01:52:34.864642 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet e249474dcbf642189560c91ec3a99fdf because the tablet was not found. No further retry: Not found: Tablet not found: e249474dcbf642189560c91ec3a99fdf
I20250901 01:52:34.867520 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 224) took 5569 ms. Trace:
I20250901 01:52:34.867632 2118 rpcz_store.cc:276] 0901 01:52:29.298140 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:29.298189 (+ 49us) service_pool.cc:225] Handling call
0901 01:52:34.867444 (+5569255us) ts_tablet_manager.cc:1145] Deleting tablet e4fdbc8d4bff4173a20db2c182acfd9e
0901 01:52:34.867453 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:34.867505 (+ 52us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5569028}
I20250901 01:52:34.868062 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 225) took 5562 ms. Trace:
I20250901 01:52:34.868141 2118 rpcz_store.cc:276] 0901 01:52:29.305680 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:29.305732 (+ 52us) service_pool.cc:225] Handling call
0901 01:52:34.867987 (+5562255us) ts_tablet_manager.cc:1145] Deleting tablet e5e250e6c84d4248b857bf1478d47703
0901 01:52:34.867996 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:34.868045 (+ 49us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5562018}
I20250901 01:52:34.868552 2118 tablet_replica.cc:331] T 69f61ec283484f9d8b3eebcea27f741b P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:34.868768 2118 raft_consensus.cc:2241] T 69f61ec283484f9d8b3eebcea27f741b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:34.869298 2118 raft_consensus.cc:2270] T 69f61ec283484f9d8b3eebcea27f741b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:34.869882 2118 ts_tablet_manager.cc:1905] T 69f61ec283484f9d8b3eebcea27f741b P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:34.879072 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet e4fdbc8d4bff4173a20db2c182acfd9e because the tablet was not found. No further retry: Not found: Tablet not found: e4fdbc8d4bff4173a20db2c182acfd9e
W20250901 01:52:34.880188 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet e5e250e6c84d4248b857bf1478d47703 because the tablet was not found. No further retry: Not found: Tablet not found: e5e250e6c84d4248b857bf1478d47703
I20250901 01:52:34.885478 2118 ts_tablet_manager.cc:1918] T 69f61ec283484f9d8b3eebcea27f741b P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:34.885622 2118 log.cc:1199] T 69f61ec283484f9d8b3eebcea27f741b P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/69f61ec283484f9d8b3eebcea27f741b
W20250901 01:52:34.841856 992 consensus_peers.cc:489] T 92f0ad59fbbb435ea872c669a52f854a P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250901 01:52:34.896474 992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:40731, user_credentials={real_user=slave}} blocked reactor thread for 48640us
I20250901 01:52:34.897136 2118 ts_tablet_manager.cc:1939] T 69f61ec283484f9d8b3eebcea27f741b P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:34.897468 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 227) took 5309 ms. Trace:
I20250901 01:52:34.897576 2118 rpcz_store.cc:276] 0901 01:52:29.587883 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:29.587958 (+ 75us) service_pool.cc:225] Handling call
0901 01:52:34.868497 (+5280539us) ts_tablet_manager.cc:1145] Deleting tablet 69f61ec283484f9d8b3eebcea27f741b
0901 01:52:34.868506 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:34.873267 (+ 4761us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:34.885319 (+ 12052us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:34.897451 (+ 12132us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5280277}
I20250901 01:52:34.897926 2118 tablet_replica.cc:331] T fecf7d538b4c4c619182d5eccb44dd8b P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:34.898133 2118 raft_consensus.cc:2241] T fecf7d538b4c4c619182d5eccb44dd8b P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:34.898288 2118 raft_consensus.cc:2270] T fecf7d538b4c4c619182d5eccb44dd8b P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:34.900228 2871 raft_consensus.cc:491] T e9f1f892658b4736859ff507536920a1 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:34.901211 2871 raft_consensus.cc:513] T e9f1f892658b4736859ff507536920a1 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:34.901832 2871 leader_election.cc:290] T e9f1f892658b4736859ff507536920a1 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:34.902644 2118 ts_tablet_manager.cc:1905] T fecf7d538b4c4c619182d5eccb44dd8b P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:34.903663 858 leader_election.cc:336] T e9f1f892658b4736859ff507536920a1 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:34.907387 1113 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.CreateTable -> {remote=127.0.104.60:44231, user_credentials={real_user=slave}} blocked reactor thread for 56809.1us
I20250901 01:52:34.908960 2118 ts_tablet_manager.cc:1918] T fecf7d538b4c4c619182d5eccb44dd8b P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:34.909096 2118 log.cc:1199] T fecf7d538b4c4c619182d5eccb44dd8b P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/fecf7d538b4c4c619182d5eccb44dd8b
I20250901 01:52:34.908787 992 leader_election.cc:304] T b2b172757e50404aae78e818921a7d9d P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:34.909435 2118 ts_tablet_manager.cc:1939] T fecf7d538b4c4c619182d5eccb44dd8b P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:34.909775 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 228) took 5321 ms. Trace:
I20250901 01:52:34.909858 2118 rpcz_store.cc:276] 0901 01:52:29.588018 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:29.588066 (+ 48us) service_pool.cc:225] Handling call
0901 01:52:34.897880 (+5309814us) ts_tablet_manager.cc:1145] Deleting tablet fecf7d538b4c4c619182d5eccb44dd8b
0901 01:52:34.897890 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:34.905892 (+ 8002us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:34.908861 (+ 2969us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:34.909758 (+ 897us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5309585}
I20250901 01:52:34.910456 2118 tablet_replica.cc:331] T b8ca4e66c6a74f9d8d90a69be5d6a58f P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:34.910650 2118 raft_consensus.cc:2241] T b8ca4e66c6a74f9d8d90a69be5d6a58f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:34.910794 2118 raft_consensus.cc:2270] T b8ca4e66c6a74f9d8d90a69be5d6a58f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:34.911269 2118 ts_tablet_manager.cc:1905] T b8ca4e66c6a74f9d8d90a69be5d6a58f P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:34.912006 2780 raft_consensus.cc:2802] T b2b172757e50404aae78e818921a7d9d P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:34.912173 2780 raft_consensus.cc:695] T b2b172757e50404aae78e818921a7d9d P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:34.912506 2780 consensus_queue.cc:237] T b2b172757e50404aae78e818921a7d9d P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:34.917366 2118 ts_tablet_manager.cc:1918] T b8ca4e66c6a74f9d8d90a69be5d6a58f P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:34.917529 2118 log.cc:1199] T b8ca4e66c6a74f9d8d90a69be5d6a58f P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/b8ca4e66c6a74f9d8d90a69be5d6a58f
I20250901 01:52:34.917915 2118 ts_tablet_manager.cc:1939] T b8ca4e66c6a74f9d8d90a69be5d6a58f P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:34.918238 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 229) took 4649 ms. Trace:
I20250901 01:52:34.918345 2118 rpcz_store.cc:276] 0901 01:52:30.268563 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.268617 (+ 54us) service_pool.cc:225] Handling call
0901 01:52:34.910411 (+4641794us) ts_tablet_manager.cc:1145] Deleting tablet b8ca4e66c6a74f9d8d90a69be5d6a58f
0901 01:52:34.910421 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:34.914480 (+ 4059us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:34.917290 (+ 2810us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:34.918211 (+ 921us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4641595}
I20250901 01:52:34.919261 2118 tablet_replica.cc:331] T 629db80a8ab444e582628578155638fd P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:34.919618 2118 raft_consensus.cc:2241] T 629db80a8ab444e582628578155638fd P 00c282f779c1477d920f24dc0dc16628 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:34.920142 2118 raft_consensus.cc:2270] T 629db80a8ab444e582628578155638fd P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:34.921120 2118 ts_tablet_manager.cc:1905] T 629db80a8ab444e582628578155638fd P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:34.923892 2885 tablet.cc:1722] T a8483e7212d94daba7e5401256c33aef P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250901 01:52:34.918879 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet b8ca4e66c6a74f9d8d90a69be5d6a58f (table default.Table_259a716264db43c49671b6fef6665f0e [id=aebce875b7dd4718a3611eec4ca56917]) successfully deleted
W20250901 01:52:34.926333 992 leader_election.cc:336] T b2b172757e50404aae78e818921a7d9d P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:34.928905 2118 ts_tablet_manager.cc:1918] T 629db80a8ab444e582628578155638fd P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 01:52:34.929092 2118 log.cc:1199] T 629db80a8ab444e582628578155638fd P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/629db80a8ab444e582628578155638fd
I20250901 01:52:34.929468 2118 ts_tablet_manager.cc:1939] T 629db80a8ab444e582628578155638fd P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:34.929780 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 230) took 4659 ms. Trace:
I20250901 01:52:34.929903 2118 rpcz_store.cc:276] 0901 01:52:30.270729 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.270778 (+ 49us) service_pool.cc:225] Handling call
0901 01:52:34.919170 (+4648392us) ts_tablet_manager.cc:1145] Deleting tablet 629db80a8ab444e582628578155638fd
0901 01:52:34.919185 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:34.925961 (+ 6776us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:34.928804 (+ 2843us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:34.929765 (+ 961us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4648155}
I20250901 01:52:34.930413 2113 pending_rounds.cc:70] T fecf7d538b4c4c619182d5eccb44dd8b P fd27ab6665304aaea53adbd102dd4f58: Trying to abort 1 pending ops.
I20250901 01:52:34.930902 2113 pending_rounds.cc:77] T fecf7d538b4c4c619182d5eccb44dd8b P fd27ab6665304aaea53adbd102dd4f58: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7195408607030890496 op_type: NO_OP noop_request { }
I20250901 01:52:34.930963 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 629db80a8ab444e582628578155638fd (table default.Table_f0e56dba1e8a4ed49a760e204c1ff96e [id=c3865e058daa4cea9a89f5046fca712a]) successfully deleted
I20250901 01:52:34.931005 2113 raft_consensus.cc:2887] T fecf7d538b4c4c619182d5eccb44dd8b P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250901 01:52:34.931092 2113 raft_consensus.cc:2270] T fecf7d538b4c4c619182d5eccb44dd8b P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:34.931424 2118 tablet_replica.cc:331] T 7b16ca1daf394a6da65a4af8da82ec33 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:34.931638 2118 raft_consensus.cc:2241] T 7b16ca1daf394a6da65a4af8da82ec33 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:34.931833 2118 raft_consensus.cc:2270] T 7b16ca1daf394a6da65a4af8da82ec33 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:34.932394 2118 ts_tablet_manager.cc:1905] T 7b16ca1daf394a6da65a4af8da82ec33 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:34.933213 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_df1849ff98434f7cabf03996e44f3a7f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.934543 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_858c19e7323146f4beca97ad8200234a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.935232 2780 raft_consensus.cc:491] T 2973b59bb1df41d18d9a5b6a29ae336f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:34.935357 2780 raft_consensus.cc:513] T 2973b59bb1df41d18d9a5b6a29ae336f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:34.935882 2780 leader_election.cc:290] T 2973b59bb1df41d18d9a5b6a29ae336f P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:34.936520 1489 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_7570c3012f8a4ae7bcfc7346eadac92a" } new_table_name: "default.Table_fc962c8bc653498e85cc0bb41639e9ad" modify_external_catalogs: true
I20250901 01:52:34.937963 1483 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_946298d2a13646aeb4c189678909d1be" } modify_external_catalogs: true
I20250901 01:52:34.938123 1483 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_946298d2a13646aeb4c189678909d1be" } modify_external_catalogs: true
I20250901 01:52:34.938789 1486 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_7790af96855e45c0a316ba3a4a87c9ed" } new_table_name: "default.Table_3c2306df46ec4c14b64b5fbd9d21eb98" modify_external_catalogs: true
I20250901 01:52:34.939795 2113 ts_tablet_manager.cc:1905] T fecf7d538b4c4c619182d5eccb44dd8b P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:34.940361 1484 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_df46d9604f6a4a19b4d7c3424f48d90a" } new_table_name: "default.Table_213d020a80574a0cbdd476d5465feb15" modify_external_catalogs: true
I20250901 01:52:34.941633 1485 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_dd92add170f94aaf966f77d81e2c3883" } modify_external_catalogs: true
I20250901 01:52:34.941789 1485 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_dd92add170f94aaf966f77d81e2c3883" } modify_external_catalogs: true
I20250901 01:52:34.943078 1488 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_63ef1dcd92db46d298446117a62cfba2" } modify_external_catalogs: true
I20250901 01:52:34.943226 1488 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_63ef1dcd92db46d298446117a62cfba2" } modify_external_catalogs: true
I20250901 01:52:34.944387 2118 ts_tablet_manager.cc:1918] T 7b16ca1daf394a6da65a4af8da82ec33 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:34.944380 1491 master_service.cc:948] ReplaceTablet: received request to replace tablet b2b172757e50404aae78e818921a7d9d from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:34.944612 2118 log.cc:1199] T 7b16ca1daf394a6da65a4af8da82ec33 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/7b16ca1daf394a6da65a4af8da82ec33
I20250901 01:52:34.945039 2118 ts_tablet_manager.cc:1939] T 7b16ca1daf394a6da65a4af8da82ec33 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:34.945371 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 231) took 4602 ms. Trace:
I20250901 01:52:34.945606 1151 tablet_bootstrap.cc:492] T 44af4764d4f64e44aec84ccb84be6f5f P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:34.945725 1151 ts_tablet_manager.cc:1397] T 44af4764d4f64e44aec84ccb84be6f5f P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.150s user 0.006s sys 0.024s
I20250901 01:52:34.946434 1151 raft_consensus.cc:357] T 44af4764d4f64e44aec84ccb84be6f5f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:34.946573 1151 raft_consensus.cc:383] T 44af4764d4f64e44aec84ccb84be6f5f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:34.946617 1151 raft_consensus.cc:738] T 44af4764d4f64e44aec84ccb84be6f5f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:34.946770 1151 consensus_queue.cc:260] T 44af4764d4f64e44aec84ccb84be6f5f P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:34.947203 1151 ts_tablet_manager.cc:1428] T 44af4764d4f64e44aec84ccb84be6f5f P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250901 01:52:34.947989 1151 tablet_bootstrap.cc:492] T 34ebb441c27747ec860ff2e25f9a0adc P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:34.950493 1151 tablet_bootstrap.cc:654] T 34ebb441c27747ec860ff2e25f9a0adc P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:34.948810 2753 raft_consensus.cc:491] T 88ceba7f48b04d7885cc0164668700a5 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:34.950814 2118 rpcz_store.cc:276] 0901 01:52:30.342955 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.343035 (+ 80us) service_pool.cc:225] Handling call
0901 01:52:34.931378 (+4588343us) ts_tablet_manager.cc:1145] Deleting tablet 7b16ca1daf394a6da65a4af8da82ec33
0901 01:52:34.931389 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:34.941164 (+ 9775us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:34.944285 (+ 3121us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:34.945352 (+ 1067us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4588117}
I20250901 01:52:34.950908 2753 raft_consensus.cc:513] T 88ceba7f48b04d7885cc0164668700a5 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:34.951541 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 7b16ca1daf394a6da65a4af8da82ec33 (table default.Table_9d8c1fb24bc24631a2ba2b34a395f24a [id=2d12d6d6275e499ba3928e76bfd3cfc7]) successfully deleted
I20250901 01:52:34.951767 2753 leader_election.cc:290] T 88ceba7f48b04d7885cc0164668700a5 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:34.952708 2118 tablet_replica.cc:331] T 081e2630e284498aa819f85bf8326315 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
W20250901 01:52:34.953277 858 leader_election.cc:336] T 88ceba7f48b04d7885cc0164668700a5 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:34.953601 2118 raft_consensus.cc:2241] T 081e2630e284498aa819f85bf8326315 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:34.953815 2118 raft_consensus.cc:2270] T 081e2630e284498aa819f85bf8326315 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:34.954341 2118 ts_tablet_manager.cc:1905] T 081e2630e284498aa819f85bf8326315 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:34.955593 2113 ts_tablet_manager.cc:1918] T fecf7d538b4c4c619182d5eccb44dd8b P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:34.955752 2113 log.cc:1199] T fecf7d538b4c4c619182d5eccb44dd8b P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/fecf7d538b4c4c619182d5eccb44dd8b
I20250901 01:52:34.956171 2113 ts_tablet_manager.cc:1939] T fecf7d538b4c4c619182d5eccb44dd8b P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:34.957201 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 316) took 5369 ms. Trace:
I20250901 01:52:34.962977 2829 tablet.cc:1722] T a8483e7212d94daba7e5401256c33aef P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250901 01:52:34.963461 2113 rpcz_store.cc:276] 0901 01:52:29.587619 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:29.588291 (+ 672us) service_pool.cc:225] Handling call
0901 01:52:34.851588 (+5263297us) ts_tablet_manager.cc:1145] Deleting tablet fecf7d538b4c4c619182d5eccb44dd8b
0901 01:52:34.851598 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:34.945632 (+ 94034us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:34.955481 (+ 9849us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:34.957180 (+ 1699us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5263089}
I20250901 01:52:34.964296 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 317) took 4739 ms. Trace:
I20250901 01:52:34.964614 2113 rpcz_store.cc:276] 0901 01:52:30.224700 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.228500 (+ 3800us) service_pool.cc:225] Handling call
0901 01:52:34.964163 (+4735663us) ts_tablet_manager.cc:1145] Deleting tablet b8ca4e66c6a74f9d8d90a69be5d6a58f
0901 01:52:34.964173 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:34.964278 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4723631}
W20250901 01:52:34.965144 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet b8ca4e66c6a74f9d8d90a69be5d6a58f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b8ca4e66c6a74f9d8d90a69be5d6a58f already in progress: creating tablet
I20250901 01:52:34.965541 2113 tablet_replica.cc:331] T 629db80a8ab444e582628578155638fd P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:34.965804 2113 raft_consensus.cc:2241] T 629db80a8ab444e582628578155638fd P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:34.966073 2113 raft_consensus.cc:2270] T 629db80a8ab444e582628578155638fd P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:34.966694 2113 ts_tablet_manager.cc:1905] T 629db80a8ab444e582628578155638fd P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:34.969013 1151 tablet_bootstrap.cc:492] T 34ebb441c27747ec860ff2e25f9a0adc P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:34.968984 1488 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet f13b972e25214f168421439fe3943e8a
I20250901 01:52:34.969209 1151 ts_tablet_manager.cc:1397] T 34ebb441c27747ec860ff2e25f9a0adc P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.001s
W20250901 01:52:34.969230 1488 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet f13b972e25214f168421439fe3943e8a: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:34.969861 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4f6da887de8243f9b679f4d0f8607b66"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.970082 1151 raft_consensus.cc:357] T 34ebb441c27747ec860ff2e25f9a0adc P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:34.970214 1151 raft_consensus.cc:383] T 34ebb441c27747ec860ff2e25f9a0adc P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:34.970258 1151 raft_consensus.cc:738] T 34ebb441c27747ec860ff2e25f9a0adc P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:34.971307 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet f13b972e25214f168421439fe3943e8a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:34 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:34.972126 893 tablet_service.cc:1515] Processing DeleteTablet for tablet f13b972e25214f168421439fe3943e8a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:34 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:34.972623 1151 consensus_queue.cc:260] T 34ebb441c27747ec860ff2e25f9a0adc P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:34.973436 1151 ts_tablet_manager.cc:1428] T 34ebb441c27747ec860ff2e25f9a0adc P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250901 01:52:34.973719 2118 ts_tablet_manager.cc:1918] T 081e2630e284498aa819f85bf8326315 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 01:52:34.973827 2118 log.cc:1199] T 081e2630e284498aa819f85bf8326315 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/081e2630e284498aa819f85bf8326315
I20250901 01:52:34.974165 2118 ts_tablet_manager.cc:1939] T 081e2630e284498aa819f85bf8326315 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:34.974872 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 232) took 4631 ms. Trace:
I20250901 01:52:34.974956 2118 rpcz_store.cc:276] 0901 01:52:30.343277 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.343314 (+ 37us) service_pool.cc:225] Handling call
0901 01:52:34.952660 (+4609346us) ts_tablet_manager.cc:1145] Deleting tablet 081e2630e284498aa819f85bf8326315
0901 01:52:34.952671 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:34.964106 (+ 11435us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:34.973602 (+ 9496us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:34.974851 (+ 1249us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4609191}
I20250901 01:52:34.976158 1151 tablet_bootstrap.cc:492] T d4c9aa723aee4dc4b6237cb5c3722b52 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:34.977241 1485 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet ea846f88b3534beebd0001b7f17b5ca9
W20250901 01:52:34.977522 1485 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet ea846f88b3534beebd0001b7f17b5ca9: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:34.978291 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet ea846f88b3534beebd0001b7f17b5ca9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:34 UTC) from {username='slave'} at 127.0.0.1:35946
W20250901 01:52:34.979060 1489 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f5e3316adeae47cd848833dd84b972bb has no consensus state
I20250901 01:52:34.979478 1151 tablet_bootstrap.cc:654] T d4c9aa723aee4dc4b6237cb5c3722b52 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:34.981607 893 tablet_service.cc:1515] Processing DeleteTablet for tablet ea846f88b3534beebd0001b7f17b5ca9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:34 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:34.984308 1489 master_service.cc:948] ReplaceTablet: received request to replace tablet fae2779fac164b9f8052c3666db803b8 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:34.985191 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_97ca32b8dae342dd8fb3b8df4d3cb30d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.986374 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_200d10cf45c44cb6a7166e3971b7ec9e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.986888 2113 ts_tablet_manager.cc:1918] T 629db80a8ab444e582628578155638fd P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 01:52:34.987037 2113 log.cc:1199] T 629db80a8ab444e582628578155638fd P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/629db80a8ab444e582628578155638fd
I20250901 01:52:34.987314 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e5aab953667844d291221e7f3df29ab3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.989555 1490 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 2973b59bb1df41d18d9a5b6a29ae336f (table default.Table_7790af96855e45c0a316ba3a4a87c9ed [id=746845bd84c6444b8a1e5683eaba4a68]). Expected version 1 got 0
I20250901 01:52:34.989847 1490 catalog_manager.cc:5582] T b4278a013ae84082bc7a8497b04ac147 P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: UNKNOWN } } }
I20250901 01:52:34.990196 1490 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet b4278a013ae84082bc7a8497b04ac147 (table default.Table_297737979dae4534a0eef6ca9e87e98d [id=6b36921a6764455aa19284d36c3f8d32]). Expected version 1 got 0
W20250901 01:52:34.991376 1486 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2973b59bb1df41d18d9a5b6a29ae336f consensus state has no leader
I20250901 01:52:34.993641 1482 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 5dd0c6b806574caf9126dad94833fcf7
W20250901 01:52:34.993831 1482 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 5dd0c6b806574caf9126dad94833fcf7: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:34.994175 1482 catalog_manager.cc:6652] ReplaceTablet: tablet 5dd0c6b806574caf9126dad94833fcf7 deleted and replaced by tablet fefbf5ae030446b292d8068e95577534
I20250901 01:52:34.994812 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5373aeeafdd0495892efce9e14e5bf1a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.995211 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 5dd0c6b806574caf9126dad94833fcf7 with delete_type TABLET_DATA_DELETED (replaced by tablet fefbf5ae030446b292d8068e95577534) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:34.997736 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d9f0263bce504531934f2613b35560dd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:34.998540 1055 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9f1f892658b4736859ff507536920a1" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:34.999137 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ef1c585d9d9940c591b1f2482e749c86"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.000355 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5bc2565ced254500a63cf929d19a47af"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.001602 1482 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_2d3d2227e6a24bd2a86f457e553e698b" } modify_external_catalogs: true
I20250901 01:52:35.001762 1482 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_2d3d2227e6a24bd2a86f457e553e698b" } modify_external_catalogs: true
I20250901 01:52:35.002799 1483 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 5ddbc7e6820247a987f4d54757d1191d
I20250901 01:52:35.002817 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 5dd0c6b806574caf9126dad94833fcf7 with delete_type TABLET_DATA_DELETED (replaced by tablet fefbf5ae030446b292d8068e95577534) from {username='slave'} at 127.0.0.1:34472
W20250901 01:52:35.002979 1483 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 5ddbc7e6820247a987f4d54757d1191d: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:35.003767 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 5ddbc7e6820247a987f4d54757d1191d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:34 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:35.004875 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 5ddbc7e6820247a987f4d54757d1191d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:34 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:35.006244 1491 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet b2b172757e50404aae78e818921a7d9d
W20250901 01:52:35.006419 1491 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet b2b172757e50404aae78e818921a7d9d: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:35.006733 1491 catalog_manager.cc:6652] ReplaceTablet: tablet b2b172757e50404aae78e818921a7d9d deleted and replaced by tablet 75883d9fd00e429aa8853903a9fdd7e4
I20250901 01:52:35.007117 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet b2b172757e50404aae78e818921a7d9d with delete_type TABLET_DATA_DELETED (replaced by tablet 75883d9fd00e429aa8853903a9fdd7e4) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:35.008148 893 tablet_service.cc:1515] Processing DeleteTablet for tablet b2b172757e50404aae78e818921a7d9d with delete_type TABLET_DATA_DELETED (replaced by tablet 75883d9fd00e429aa8853903a9fdd7e4) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:35.010468 1491 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_4dacb3b1d2e94e42ac8937eb6eb58375" } new_table_name: "default.Table_9fb44a0a69994184bc2ffe62c55d2c25" modify_external_catalogs: true
I20250901 01:52:35.013710 2113 ts_tablet_manager.cc:1939] T 629db80a8ab444e582628578155638fd P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:35.014644 1482 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet a656adbf1b29431687e0989f0db3459c
I20250901 01:52:35.016392 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 318) took 4791 ms. Trace:
I20250901 01:52:35.017833 2113 rpcz_store.cc:276] 0901 01:52:30.224836 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.228748 (+ 3912us) service_pool.cc:225] Handling call
0901 01:52:34.965478 (+4736730us) ts_tablet_manager.cc:1145] Deleting tablet 629db80a8ab444e582628578155638fd
0901 01:52:34.965488 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:34.976542 (+ 11054us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:34.986787 (+ 10245us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:35.016373 (+ 29586us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4724868}
I20250901 01:52:35.018493 2113 tablet_replica.cc:331] T 7b16ca1daf394a6da65a4af8da82ec33 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:35.018733 2113 raft_consensus.cc:2241] T 7b16ca1daf394a6da65a4af8da82ec33 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:35.019137 2113 raft_consensus.cc:2270] T 7b16ca1daf394a6da65a4af8da82ec33 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:35.019464 2118 tablet_replica.cc:331] T f4fd117c395e41c2ab679c684908c5b7 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:35.019661 2118 raft_consensus.cc:2241] T f4fd117c395e41c2ab679c684908c5b7 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:35.020085 2113 ts_tablet_manager.cc:1905] T 7b16ca1daf394a6da65a4af8da82ec33 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:35.019860 2118 raft_consensus.cc:2270] T f4fd117c395e41c2ab679c684908c5b7 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:35.021308 2118 ts_tablet_manager.cc:1905] T f4fd117c395e41c2ab679c684908c5b7 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:35.020656 1151 tablet_bootstrap.cc:492] T d4c9aa723aee4dc4b6237cb5c3722b52 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:35.022270 1151 ts_tablet_manager.cc:1397] T d4c9aa723aee4dc4b6237cb5c3722b52 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.046s user 0.005s sys 0.002s
I20250901 01:52:35.023046 1151 raft_consensus.cc:357] T d4c9aa723aee4dc4b6237cb5c3722b52 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:35.023310 1151 raft_consensus.cc:383] T d4c9aa723aee4dc4b6237cb5c3722b52 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:35.023428 1151 raft_consensus.cc:738] T d4c9aa723aee4dc4b6237cb5c3722b52 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:35.023732 1151 consensus_queue.cc:260] T d4c9aa723aee4dc4b6237cb5c3722b52 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:35.024226 1151 ts_tablet_manager.cc:1428] T d4c9aa723aee4dc4b6237cb5c3722b52 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250901 01:52:35.024664 1151 tablet_bootstrap.cc:492] T 6ba8241644984394ada95b561e76467e P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:35.028200 1492 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 93bcac1fd95d44dc927abfca81692921 (table default.Table_b8f8ee74648c406eaf589d3ab80934db [id=b8f69313439543be9c11eea88dfa69c5]). Expected version 1 got 0
I20250901 01:52:35.028964 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 629db80a8ab444e582628578155638fd (table default.Table_f0e56dba1e8a4ed49a760e204c1ff96e [id=c3865e058daa4cea9a89f5046fca712a]) successfully deleted
I20250901 01:52:35.031968 1492 catalog_manager.cc:5582] T d0c3c363dda444519a17dca63ad9caa7 P 00c282f779c1477d920f24dc0dc16628 reported cstate change: term changed from 0 to 3, leader changed from <none> to 00c282f779c1477d920f24dc0dc16628 (127.0.104.2). New cstate: current_term: 3 leader_uuid: "00c282f779c1477d920f24dc0dc16628" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: UNKNOWN } } }
I20250901 01:52:35.032469 1492 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet d0c3c363dda444519a17dca63ad9caa7 (table default.Table_2f51253b29ae46f8ae7b33f230aab3ac [id=8c1b263824ec4c15a5ec424364c78895]). Expected version 5 got 0
I20250901 01:52:35.032914 1490 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: default.Table_ee3fa2db164c4209b58ffaf8102700ba [id=78cd71321e9a480ca2d7d88327f157ee] alter complete (version 4)
I20250901 01:52:35.033143 1055 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "88ceba7f48b04d7885cc0164668700a5" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:35.033623 992 consensus_peers.cc:489] T fcb7b5b7d63d4f0b906fea97cb548de2 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20250901 01:52:35.034680 1151 tablet_bootstrap.cc:654] T 6ba8241644984394ada95b561e76467e P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:35.036805 2113 ts_tablet_manager.cc:1918] T 7b16ca1daf394a6da65a4af8da82ec33 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:35.036984 2113 log.cc:1199] T 7b16ca1daf394a6da65a4af8da82ec33 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/7b16ca1daf394a6da65a4af8da82ec33
I20250901 01:52:35.037447 2113 ts_tablet_manager.cc:1939] T 7b16ca1daf394a6da65a4af8da82ec33 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:35.038576 921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2973b59bb1df41d18d9a5b6a29ae336f" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:35.038790 921 raft_consensus.cc:2466] T 2973b59bb1df41d18d9a5b6a29ae336f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:35.039561 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 321) took 4707 ms. Trace:
I20250901 01:52:35.039690 2113 rpcz_store.cc:276] 0901 01:52:30.332567 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.332616 (+ 49us) service_pool.cc:225] Handling call
0901 01:52:35.018438 (+4685822us) ts_tablet_manager.cc:1145] Deleting tablet 7b16ca1daf394a6da65a4af8da82ec33
0901 01:52:35.018450 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.028073 (+ 9623us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:35.036704 (+ 8631us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:35.039542 (+ 2838us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4685550}
I20250901 01:52:35.040323 2118 ts_tablet_manager.cc:1918] T f4fd117c395e41c2ab679c684908c5b7 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 01:52:35.040642 2113 tablet_replica.cc:331] T 081e2630e284498aa819f85bf8326315 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:35.040813 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 7b16ca1daf394a6da65a4af8da82ec33 (table default.Table_9d8c1fb24bc24631a2ba2b34a395f24a [id=2d12d6d6275e499ba3928e76bfd3cfc7]) successfully deleted
I20250901 01:52:35.040877 2113 raft_consensus.cc:2241] T 081e2630e284498aa819f85bf8326315 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:35.041183 2113 raft_consensus.cc:2270] T 081e2630e284498aa819f85bf8326315 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:35.043947 2118 log.cc:1199] T f4fd117c395e41c2ab679c684908c5b7 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/f4fd117c395e41c2ab679c684908c5b7
I20250901 01:52:35.044318 2118 ts_tablet_manager.cc:1939] T f4fd117c395e41c2ab679c684908c5b7 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:35.044631 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 233) took 4660 ms. Trace:
I20250901 01:52:35.044708 2118 rpcz_store.cc:276] 0901 01:52:30.384003 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.384170 (+ 167us) service_pool.cc:225] Handling call
0901 01:52:35.019414 (+4635244us) ts_tablet_manager.cc:1145] Deleting tablet f4fd117c395e41c2ab679c684908c5b7
0901 01:52:35.019424 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.037666 (+ 18242us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:35.040245 (+ 2579us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:35.044617 (+ 4372us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4603833}
I20250901 01:52:35.045023 2113 ts_tablet_manager.cc:1905] T 081e2630e284498aa819f85bf8326315 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:35.045130 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet f4fd117c395e41c2ab679c684908c5b7 (table default.Table_0c7355975d6b44dbbe3a77df942b4d43 [id=305490ecfddb4d46927f7bb128c7f878]) successfully deleted
I20250901 01:52:35.046310 2118 tablet_replica.cc:331] T f80f13db501c42c7950499df8fcb6fc8 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:35.046662 2118 raft_consensus.cc:2241] T f80f13db501c42c7950499df8fcb6fc8 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:35.046816 1490 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: default.Table_d3fd1940d13e4afba8b3bb674bd4e4ee [id=543f2074598d4c29972e564be1123fca] alter complete (version 7)
I20250901 01:52:35.047211 2118 raft_consensus.cc:2270] T f80f13db501c42c7950499df8fcb6fc8 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:35.047693 2118 ts_tablet_manager.cc:1905] T f80f13db501c42c7950499df8fcb6fc8 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:35.049945 859 leader_election.cc:343] T e9f1f892658b4736859ff507536920a1 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:35.050067 859 leader_election.cc:304] T e9f1f892658b4736859ff507536920a1 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:35.050545 2753 raft_consensus.cc:2747] T e9f1f892658b4736859ff507536920a1 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:35.053563 2118 ts_tablet_manager.cc:1918] T f80f13db501c42c7950499df8fcb6fc8 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 01:52:35.053673 2118 log.cc:1199] T f80f13db501c42c7950499df8fcb6fc8 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/f80f13db501c42c7950499df8fcb6fc8
I20250901 01:52:35.054034 2118 ts_tablet_manager.cc:1939] T f80f13db501c42c7950499df8fcb6fc8 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:35.054327 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 235) took 4662 ms. Trace:
I20250901 01:52:35.054409 2118 rpcz_store.cc:276] 0901 01:52:30.392348 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.392395 (+ 47us) service_pool.cc:225] Handling call
0901 01:52:35.046264 (+4653869us) ts_tablet_manager.cc:1145] Deleting tablet f80f13db501c42c7950499df8fcb6fc8
0901 01:52:35.046275 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.050672 (+ 4397us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:35.053482 (+ 2810us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:35.054310 (+ 828us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4630622}
I20250901 01:52:35.054704 2753 consensus_queue.cc:1035] T 0e4009963cfe42d6ba164fc828897f09 P 00c282f779c1477d920f24dc0dc16628 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:35.055084 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 237) took 4547 ms. Trace:
I20250901 01:52:35.055164 2118 rpcz_store.cc:276] 0901 01:52:30.507467 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.508455 (+ 988us) service_pool.cc:225] Handling call
0901 01:52:35.054984 (+4546529us) ts_tablet_manager.cc:1145] Deleting tablet 55bd2daa07294358999956cb553d973f
0901 01:52:35.054996 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.055067 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4540633}
W20250901 01:52:35.055636 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 55bd2daa07294358999956cb553d973f because the tablet was not found. No further retry: Not found: Tablet not found: 55bd2daa07294358999956cb553d973f
I20250901 01:52:35.056044 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 238) took 4541 ms. Trace:
I20250901 01:52:35.056176 2118 rpcz_store.cc:276] 0901 01:52:30.514932 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.514979 (+ 47us) service_pool.cc:225] Handling call
0901 01:52:35.055958 (+4540979us) ts_tablet_manager.cc:1145] Deleting tablet e8e7d2a72f704f52a8f8d46911a8a9e4
0901 01:52:35.055968 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.056028 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4540796}
W20250901 01:52:35.056712 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet e8e7d2a72f704f52a8f8d46911a8a9e4 because the tablet was not found. No further retry: Not found: Tablet not found: e8e7d2a72f704f52a8f8d46911a8a9e4
I20250901 01:52:35.056941 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 239) took 4536 ms. Trace:
I20250901 01:52:35.057051 2118 rpcz_store.cc:276] 0901 01:52:30.520149 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.520594 (+ 445us) service_pool.cc:225] Handling call
0901 01:52:35.056573 (+4535979us) ts_tablet_manager.cc:1145] Deleting tablet becbaae4fd7e44faa679e9479465a7c4
0901 01:52:35.056581 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.056924 (+ 343us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4535824}
W20250901 01:52:35.057456 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet becbaae4fd7e44faa679e9479465a7c4 because the tablet was not found. No further retry: Not found: Tablet not found: becbaae4fd7e44faa679e9479465a7c4
I20250901 01:52:35.057478 2113 ts_tablet_manager.cc:1918] T 081e2630e284498aa819f85bf8326315 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 01:52:35.057587 2113 log.cc:1199] T 081e2630e284498aa819f85bf8326315 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/081e2630e284498aa819f85bf8326315
I20250901 01:52:35.057821 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 241) took 4536 ms. Trace:
I20250901 01:52:35.057907 2113 ts_tablet_manager.cc:1939] T 081e2630e284498aa819f85bf8326315 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:35.057924 2118 rpcz_store.cc:276] 0901 01:52:30.521595 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.523300 (+ 1705us) service_pool.cc:225] Handling call
0901 01:52:35.057725 (+4534425us) ts_tablet_manager.cc:1145] Deleting tablet 879fa66e55544118aafe504587393c66
0901 01:52:35.057733 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.057807 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4534253}
W20250901 01:52:35.058315 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 879fa66e55544118aafe504587393c66 because the tablet was not found. No further retry: Not found: Tablet not found: 879fa66e55544118aafe504587393c66
I20250901 01:52:35.058888 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 242) took 4535 ms. Trace:
I20250901 01:52:35.058993 2118 rpcz_store.cc:276] 0901 01:52:30.523713 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.523775 (+ 62us) service_pool.cc:225] Handling call
0901 01:52:35.058819 (+4535044us) ts_tablet_manager.cc:1145] Deleting tablet 410d44e5f1974ed69e5dee65e98856a4
0901 01:52:35.058827 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.058874 (+ 47us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4534890}
W20250901 01:52:35.059569 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 410d44e5f1974ed69e5dee65e98856a4 because the tablet was not found. No further retry: Not found: Tablet not found: 410d44e5f1974ed69e5dee65e98856a4
I20250901 01:52:35.059729 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 240) took 4538 ms. Trace:
I20250901 01:52:35.059870 2118 rpcz_store.cc:276] 0901 01:52:30.521191 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.523963 (+ 2772us) service_pool.cc:225] Handling call
0901 01:52:35.059398 (+4535435us) ts_tablet_manager.cc:1145] Deleting tablet 31614db4c53a4f3c96fd190d326ce601
0901 01:52:35.059405 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.059453 (+ 48us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4535300}
W20250901 01:52:35.060334 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 31614db4c53a4f3c96fd190d326ce601 because the tablet was not found. No further retry: Not found: Tablet not found: 31614db4c53a4f3c96fd190d326ce601
I20250901 01:52:35.061064 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 244) took 4525 ms. Trace:
I20250901 01:52:35.061174 2118 rpcz_store.cc:276] 0901 01:52:30.535644 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.536011 (+ 367us) service_pool.cc:225] Handling call
0901 01:52:35.060988 (+4524977us) ts_tablet_manager.cc:1145] Deleting tablet 8ed6616ce54548ba83ba299c4ecf3c38
0901 01:52:35.060995 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.061050 (+ 55us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4508451}
W20250901 01:52:35.061556 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 8ed6616ce54548ba83ba299c4ecf3c38 because the tablet was not found. No further retry: Not found: Tablet not found: 8ed6616ce54548ba83ba299c4ecf3c38
I20250901 01:52:35.061865 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 245) took 4514 ms. Trace:
I20250901 01:52:35.061969 2118 rpcz_store.cc:276] 0901 01:52:30.547450 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.547499 (+ 49us) service_pool.cc:225] Handling call
0901 01:52:35.061797 (+4514298us) ts_tablet_manager.cc:1145] Deleting tablet cf806854a8074b0798b3f9f3df587c63
0901 01:52:35.061805 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.061852 (+ 47us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4508735}
W20250901 01:52:35.062362 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet cf806854a8074b0798b3f9f3df587c63 because the tablet was not found. No further retry: Not found: Tablet not found: cf806854a8074b0798b3f9f3df587c63
I20250901 01:52:35.062657 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 250) took 4500 ms. Trace:
I20250901 01:52:35.062757 2118 rpcz_store.cc:276] 0901 01:52:30.562692 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.562733 (+ 41us) service_pool.cc:225] Handling call
0901 01:52:35.062591 (+4499858us) ts_tablet_manager.cc:1145] Deleting tablet d58f69fad01f403eab74a47d3695ed83
0901 01:52:35.062599 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.062644 (+ 45us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4499699}
W20250901 01:52:35.063141 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet d58f69fad01f403eab74a47d3695ed83 because the tablet was not found. No further retry: Not found: Tablet not found: d58f69fad01f403eab74a47d3695ed83
I20250901 01:52:35.063519 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 251) took 4500 ms. Trace:
I20250901 01:52:35.063630 2118 rpcz_store.cc:276] 0901 01:52:30.563340 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.563383 (+ 43us) service_pool.cc:225] Handling call
0901 01:52:35.063452 (+4500069us) ts_tablet_manager.cc:1145] Deleting tablet a289a66b2dfa44dc8c8547ff83c1716e
0901 01:52:35.063459 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.063504 (+ 45us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4499901}
W20250901 01:52:35.064014 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet a289a66b2dfa44dc8c8547ff83c1716e because the tablet was not found. No further retry: Not found: Tablet not found: a289a66b2dfa44dc8c8547ff83c1716e
I20250901 01:52:35.064297 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 252) took 4500 ms. Trace:
I20250901 01:52:35.064414 2118 rpcz_store.cc:276] 0901 01:52:30.563941 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.563983 (+ 42us) service_pool.cc:225] Handling call
0901 01:52:35.064235 (+4500252us) ts_tablet_manager.cc:1145] Deleting tablet e2aa32450a584395ae07b49f76a7b26d
0901 01:52:35.064242 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.064285 (+ 43us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4500125}
W20250901 01:52:35.065114 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet e2aa32450a584395ae07b49f76a7b26d because the tablet was not found. No further retry: Not found: Tablet not found: e2aa32450a584395ae07b49f76a7b26d
I20250901 01:52:35.065150 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 322) took 4731 ms. Trace:
I20250901 01:52:35.065232 2113 rpcz_store.cc:276] 0901 01:52:30.334150 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.334200 (+ 50us) service_pool.cc:225] Handling call
0901 01:52:35.040594 (+4706394us) ts_tablet_manager.cc:1145] Deleting tablet 081e2630e284498aa819f85bf8326315
0901 01:52:35.040604 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.048382 (+ 7778us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:35.057384 (+ 9002us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:35.065126 (+ 7742us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4706223}
I20250901 01:52:35.065553 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 248) took 4504 ms. Trace:
I20250901 01:52:35.065651 2118 rpcz_store.cc:276] 0901 01:52:30.561343 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.564442 (+ 3099us) service_pool.cc:225] Handling call
0901 01:52:35.065459 (+4501017us) ts_tablet_manager.cc:1145] Deleting tablet 7cc1efcb6bff4aba8015d2d3b7f4c923
0901 01:52:35.065471 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.065536 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4500838}
I20250901 01:52:35.066011 2113 tablet_replica.cc:331] T f4fd117c395e41c2ab679c684908c5b7 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
W20250901 01:52:35.066084 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 7cc1efcb6bff4aba8015d2d3b7f4c923 because the tablet was not found. No further retry: Not found: Tablet not found: 7cc1efcb6bff4aba8015d2d3b7f4c923
I20250901 01:52:35.066634 2113 raft_consensus.cc:2241] T f4fd117c395e41c2ab679c684908c5b7 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:35.067009 2113 raft_consensus.cc:2270] T f4fd117c395e41c2ab679c684908c5b7 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:35.067759 2113 ts_tablet_manager.cc:1905] T f4fd117c395e41c2ab679c684908c5b7 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
E20250901 01:52:35.068895 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 494b6488aa0d4355b4caa46004267465, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7c21b68ef2e74136a807c1dd90e17f4e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.069043 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 01deaae6abb141789f0cd240f3058c73, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7baa92b46d98454a891222be49a6ca48'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.069131 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 0ee3f35392954817b350e5f48653be78, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_442e11b9211c4b209c68f653d7bfc313'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.069239 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae4844183a3c4a04a775e6fd878391a3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_4bd66ab99a454974aa0f7f4290edfda2'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.069367 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9745582cebfe47b58c21d0191ab4cb9b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77728af9eee44f29a1e2131d5b75b7ca'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.069465 1696 catalog_manager.cc:6205] Error selecting replicas for tablet e2ed32b3292141278af28026528e96c2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b5077f55951d48be92a67407ffe26a08'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.069556 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9f62a9d97f514747b940b488907369f3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d53a456dbca64daf92aeb5fb4517b417'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.069660 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5a7fa68d29644ae8afd2711e39dde59a, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d24e3a18ef4843fdb7b74ce23d47cf9e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.069738 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 415c9d7b552b42c384b444a20491f65c, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_97c44ea002094df8a6af3dccd0099c39'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.069850 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ff2bd017417c41b5b8225b8f00283e63, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0c7355975d6b44dbbe3a77df942b4d43'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.069932 1696 catalog_manager.cc:6205] Error selecting replicas for tablet dcc60bb000f440f1aca9133928b5ce20, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2fbca8463d6546c9b8f5b018fc60251d'. Need at least 3 replicas, but only 2 tablet servers are available
W20250901 01:52:35.069983 859 leader_election.cc:343] T 88ceba7f48b04d7885cc0164668700a5 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
E20250901 01:52:35.070036 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 465c02a130c54e09b5a74fb4dd2ef762, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ed6fd284be6f4ae3a008e21e7ce17930'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.070125 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f8f923ddcd2740bcb7e521f4f57c4034, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_e09ac1a9050547038400d4b52299d22f'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.070091 859 leader_election.cc:304] T 88ceba7f48b04d7885cc0164668700a5 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
E20250901 01:52:35.070222 1696 catalog_manager.cc:6205] Error selecting replicas for tablet b1ba0e19dc0b462e80c537bf2d183f5b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_66b111271c15410aab23682f8f6081ac'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.070305 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 8e10a01e77d14cf28c2c5e0205e595b9, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_83cf2d86a1284544ba2e7aa9ebb66bb7'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.070394 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5f0e054f62de4cf6bc6f69ad6664b192, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ef6832d0845044ed9f165ca3541b4c3f'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.070477 1696 catalog_manager.cc:6205] Error selecting replicas for tablet fd2ce82a4ee24d0faa44980a3967e187, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_63d0f2ead9d946d39ce40162ebd769d6'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.070487 2753 raft_consensus.cc:2747] T 88ceba7f48b04d7885cc0164668700a5 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
E20250901 01:52:35.070588 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae9a4ad4fdfb4771ba23a91f7007447e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_5e84ae9a9c2a4084a1f9c5ffb4594de2'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.070670 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 086aa948f93644da8e5835b0fac87ddd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b4fbc34da806475c912052d0fab3fc53'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.070770 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 082db24bb3fd4089950504145a20a075, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7f5b278599994063862981b6b041ca8c'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.070868 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 93aa22afc0ad4d7fa24807ea3c5d5c21, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0f4b4094d2974b7591875375eda6bed0'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.070966 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9074543dea9949e7a085717b40913348, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_53e30205c5fd488eb3b48c685a159d04'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.071058 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 3a62034e688348ed9e2ac9de009ed3a2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_e225c82a2d6045f6b16915cdb72f5734'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.072476 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 18498c9cb09141d08c1c5c0854b63ebd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77f4810bc83c4d948531da2a45a3792e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.072682 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f5e3316adeae47cd848833dd84b972bb, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_fc962c8bc653498e85cc0bb41639e9ad'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.072853 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 81e8c6567953484ab000fe2eb4398301, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_a920a3ffbabf4d83a6563c3423cde12a'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.073124 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 6ce9521b925b4f97bb4a810ce77e147e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_9e04f976a93f4d30baccf90fa3622dfe'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.073282 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2f61edb50d7a42198bfd70639c7bf4de, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_a5c783cbdb694af69047b31d99ff8def'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.077353 1484 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_2f51253b29ae46f8ae7b33f230aab3ac" } modify_external_catalogs: true
I20250901 01:52:35.077556 1484 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_2f51253b29ae46f8ae7b33f230aab3ac" } modify_external_catalogs: true
I20250901 01:52:35.077797 1490 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: default.Table_3cd96297e5ac445d9fcfb30dd37697da [id=cecdd8d6ca6d445c9fd82decd2a95366] alter complete (version 5)
I20250901 01:52:35.079238 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet fecf7d538b4c4c619182d5eccb44dd8b with delete_type TABLET_DATA_DELETED (replaced by tablet 65e5cdc10e9642b7b0df775cb02861fa) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:35.079942 1491 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_3cd96297e5ac445d9fcfb30dd37697da" } new_table_name: "default.Table_4e6fae563c5843f4953527abbe862adc" modify_external_catalogs: true
I20250901 01:52:35.081223 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 7f46e9ddadaf4ff986d8c991178daa0c with delete_type TABLET_DATA_DELETED (replaced by tablet 7d7346ef9a2e4a35874605256ce41c6a) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:35.082829 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_207196a1a0204f859962c73975aa8e38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.083900 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_0af1eee7e2a942b291110c6b0d345b50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.084900 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d601df586e0845f592aea74eb44025a2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.085374 2113 ts_tablet_manager.cc:1918] T f4fd117c395e41c2ab679c684908c5b7 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 01:52:35.085513 2113 log.cc:1199] T f4fd117c395e41c2ab679c684908c5b7 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/f4fd117c395e41c2ab679c684908c5b7
I20250901 01:52:35.086036 2113 ts_tablet_manager.cc:1939] T f4fd117c395e41c2ab679c684908c5b7 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:35.085951 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ec1837e9301943a5941f3f317f4a4aa7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.086824 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a77c39252a4b4786afa6e386a74397cc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.089282 2878 raft_consensus.cc:491] T 34ebb441c27747ec860ff2e25f9a0adc P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:35.089401 2878 raft_consensus.cc:513] T 34ebb441c27747ec860ff2e25f9a0adc P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:35.089529 992 leader_election.cc:304] T 2973b59bb1df41d18d9a5b6a29ae336f P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:35.089946 2878 leader_election.cc:290] T 34ebb441c27747ec860ff2e25f9a0adc P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:35.089884 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b0be0f0ac7974ef9b32dc0fa3b2bbf29"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.090934 1483 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 93bcac1fd95d44dc927abfca81692921 (table default.Table_b8f8ee74648c406eaf589d3ab80934db [id=b8f69313439543be9c11eea88dfa69c5]). Expected version 1 got 0
I20250901 01:52:35.091396 2780 raft_consensus.cc:2802] T 2973b59bb1df41d18d9a5b6a29ae336f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:35.091472 2780 raft_consensus.cc:491] T 2973b59bb1df41d18d9a5b6a29ae336f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:35.091531 2780 raft_consensus.cc:3058] T 2973b59bb1df41d18d9a5b6a29ae336f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:35.091854 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 323) took 4707 ms. Trace:
I20250901 01:52:35.091990 2113 rpcz_store.cc:276] 0901 01:52:30.384664 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.384853 (+ 189us) service_pool.cc:225] Handling call
0901 01:52:35.065961 (+4681108us) ts_tablet_manager.cc:1145] Deleting tablet f4fd117c395e41c2ab679c684908c5b7
0901 01:52:35.065972 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.072214 (+ 6242us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:35.085267 (+ 13053us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:35.091837 (+ 6570us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5113,"tablet-delete.queue_time_us":4680815}
I20250901 01:52:35.092882 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 327) took 4702 ms. Trace:
I20250901 01:52:35.093011 2113 rpcz_store.cc:276] 0901 01:52:30.390205 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.390249 (+ 44us) service_pool.cc:225] Handling call
0901 01:52:35.092790 (+4702541us) ts_tablet_manager.cc:1145] Deleting tablet c763d252598745a088344aa0eb5fa3d3
0901 01:52:35.092801 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.092869 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4702353}
I20250901 01:52:35.093482 2113 tablet_replica.cc:331] T f80f13db501c42c7950499df8fcb6fc8 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:35.093880 2113 raft_consensus.cc:2241] T f80f13db501c42c7950499df8fcb6fc8 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:35.094213 2113 raft_consensus.cc:2270] T f80f13db501c42c7950499df8fcb6fc8 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:35.094321 2780 raft_consensus.cc:513] T 2973b59bb1df41d18d9a5b6a29ae336f P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:35.094851 2113 ts_tablet_manager.cc:1905] T f80f13db501c42c7950499df8fcb6fc8 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:35.095029 1483 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet ae21b9ae2e3b472eb67fb5f96388a50f (table default.Table_046289f393ff432aae0b033bdec1fb17 [id=db57b18a9ce24e068b362f5fbe940531]). Expected version 1 got 0
I20250901 01:52:35.096413 2753 raft_consensus.cc:491] T 34ebb441c27747ec860ff2e25f9a0adc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:35.096578 2753 raft_consensus.cc:513] T 34ebb441c27747ec860ff2e25f9a0adc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:35.092907 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet f4fd117c395e41c2ab679c684908c5b7 (table default.Table_0c7355975d6b44dbbe3a77df942b4d43 [id=305490ecfddb4d46927f7bb128c7f878]) successfully deleted
I20250901 01:52:35.097188 2753 leader_election.cc:290] T 34ebb441c27747ec860ff2e25f9a0adc P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:35.097328 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet c763d252598745a088344aa0eb5fa3d3 because the tablet was not found. No further retry: Not found: Tablet not found: c763d252598745a088344aa0eb5fa3d3
I20250901 01:52:35.091048 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e3406a49689045d3825b080cfe955ad2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:35.099176 858 leader_election.cc:336] T 34ebb441c27747ec860ff2e25f9a0adc P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:35.094879 2780 leader_election.cc:290] T 2973b59bb1df41d18d9a5b6a29ae336f P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:35.102119 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_00273d8a792646bb9ccaa6052f7a05a5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.103821 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_049d8204c40f451298882b3b8af7f0e8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.105598 2113 ts_tablet_manager.cc:1918] T f80f13db501c42c7950499df8fcb6fc8 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 01:52:35.106142 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_96106b2ece32450ab6a39c2dd59c28b5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.107149 2113 log.cc:1199] T f80f13db501c42c7950499df8fcb6fc8 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/f80f13db501c42c7950499df8fcb6fc8
I20250901 01:52:35.107576 1483 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: default.Table_9f7d6f3bd6c6495280e3bd7725bc8f6d [id=6a01b6c6977546be9c6128ca5f200a92] alter complete (version 2)
I20250901 01:52:35.107717 2780 consensus_queue.cc:1035] T f195b040a6ae4b7b83a8308ab0a12b37 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:35.108623 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 247) took 4554 ms. Trace:
I20250901 01:52:35.108743 2118 rpcz_store.cc:276] 0901 01:52:30.553842 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.564888 (+ 11046us) service_pool.cc:225] Handling call
0901 01:52:35.108472 (+4543584us) ts_tablet_manager.cc:1145] Deleting tablet 72ba689f9230411995dc96e2b45601df
0901 01:52:35.108493 (+ 21us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.108597 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4501660}
I20250901 01:52:35.108945 2113 ts_tablet_manager.cc:1939] T f80f13db501c42c7950499df8fcb6fc8 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:35.109133 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 246) took 4556 ms. Trace:
I20250901 01:52:35.109208 2118 rpcz_store.cc:276] 0901 01:52:30.552918 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.565054 (+ 12136us) service_pool.cc:225] Handling call
0901 01:52:35.109051 (+4543997us) ts_tablet_manager.cc:1145] Deleting tablet 2599f98abfdc4871b573edfc2ad97d22
0901 01:52:35.109061 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.109117 (+ 56us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4543863}
W20250901 01:52:35.109318 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 72ba689f9230411995dc96e2b45601df because the tablet was not found. No further retry: Not found: Tablet not found: 72ba689f9230411995dc96e2b45601df
I20250901 01:52:35.109477 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 253) took 4544 ms. Trace:
W20250901 01:52:35.109800 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 2599f98abfdc4871b573edfc2ad97d22 because the tablet was not found. No further retry: Not found: Tablet not found: 2599f98abfdc4871b573edfc2ad97d22
I20250901 01:52:35.110219 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 328) took 4717 ms. Trace:
E20250901 01:52:35.111779 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 494b6488aa0d4355b4caa46004267465, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7c21b68ef2e74136a807c1dd90e17f4e'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.109546 2118 rpcz_store.cc:276] 0901 01:52:30.564626 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.565282 (+ 656us) service_pool.cc:225] Handling call
0901 01:52:35.109409 (+4544127us) ts_tablet_manager.cc:1145] Deleting tablet f9b293bf6a17453b9ea9892a6970d0b0
0901 01:52:35.109417 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.109463 (+ 46us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4543981}
E20250901 01:52:35.111927 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 01deaae6abb141789f0cd240f3058c73, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7baa92b46d98454a891222be49a6ca48'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.112037 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 0ee3f35392954817b350e5f48653be78, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_442e11b9211c4b209c68f653d7bfc313'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.112176 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae4844183a3c4a04a775e6fd878391a3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_4bd66ab99a454974aa0f7f4290edfda2'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.112018 2113 rpcz_store.cc:276] 0901 01:52:30.392788 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.392835 (+ 47us) service_pool.cc:225] Handling call
0901 01:52:35.093433 (+4700598us) ts_tablet_manager.cc:1145] Deleting tablet f80f13db501c42c7950499df8fcb6fc8
0901 01:52:35.093442 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.098696 (+ 5254us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:35.105492 (+ 6796us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:35.110199 (+ 4707us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4700398}
I20250901 01:52:35.112210 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 254) took 4547 ms. Trace:
E20250901 01:52:35.112273 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9745582cebfe47b58c21d0191ab4cb9b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77728af9eee44f29a1e2131d5b75b7ca'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.112275 2118 rpcz_store.cc:276] 0901 01:52:30.565055 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.565482 (+ 427us) service_pool.cc:225] Handling call
0901 01:52:35.112009 (+4546527us) ts_tablet_manager.cc:1145] Deleting tablet ddbc15685fca4c708329420109ad0c57
0901 01:52:35.112136 (+ 127us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.112194 (+ 58us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4546321}
E20250901 01:52:35.112377 1696 catalog_manager.cc:6205] Error selecting replicas for tablet e2ed32b3292141278af28026528e96c2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b5077f55951d48be92a67407ffe26a08'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.112746 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 329) took 4719 ms. Trace:
I20250901 01:52:35.112866 2113 rpcz_store.cc:276] 0901 01:52:30.393431 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.393476 (+ 45us) service_pool.cc:225] Handling call
0901 01:52:35.112659 (+4719183us) ts_tablet_manager.cc:1145] Deleting tablet 579b9cdf819f4609a532e96bec093fd5
0901 01:52:35.112669 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.112731 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4719003}
W20250901 01:52:35.113286 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 579b9cdf819f4609a532e96bec093fd5 because the tablet was not found. No further retry: Not found: Tablet not found: 579b9cdf819f4609a532e96bec093fd5
I20250901 01:52:35.113336 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 331) took 4616 ms. Trace:
I20250901 01:52:35.113401 2113 rpcz_store.cc:276] 0901 01:52:30.497243 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.498840 (+ 1597us) service_pool.cc:225] Handling call
0901 01:52:35.113261 (+4614421us) ts_tablet_manager.cc:1145] Deleting tablet a59ac57aebe24860aa02f53ce0485910
0901 01:52:35.113270 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.113323 (+ 53us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4614205}
W20250901 01:52:35.113758 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet a59ac57aebe24860aa02f53ce0485910 because the tablet was not found. No further retry: Not found: Tablet not found: a59ac57aebe24860aa02f53ce0485910
W20250901 01:52:35.115437 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet f9b293bf6a17453b9ea9892a6970d0b0 because the tablet was not found. No further retry: Not found: Tablet not found: f9b293bf6a17453b9ea9892a6970d0b0
W20250901 01:52:35.115795 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet ddbc15685fca4c708329420109ad0c57 because the tablet was not found. No further retry: Not found: Tablet not found: ddbc15685fca4c708329420109ad0c57
E20250901 01:52:35.115790 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9f62a9d97f514747b940b488907369f3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d53a456dbca64daf92aeb5fb4517b417'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.116281 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5a7fa68d29644ae8afd2711e39dde59a, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d24e3a18ef4843fdb7b74ce23d47cf9e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.116451 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 415c9d7b552b42c384b444a20491f65c, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_97c44ea002094df8a6af3dccd0099c39'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.116513 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 333) took 4619 ms. Trace:
I20250901 01:52:35.116523 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 255) took 4550 ms. Trace:
I20250901 01:52:35.116591 2118 rpcz_store.cc:276] 0901 01:52:30.565670 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.569165 (+ 3495us) service_pool.cc:225] Handling call
0901 01:52:35.112505 (+4543340us) ts_tablet_manager.cc:1145] Deleting tablet 617cc33d0d23439cb5f1e0c31bf11281
0901 01:52:35.116431 (+ 3926us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.116505 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4543209}
E20250901 01:52:35.116792 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ff2bd017417c41b5b8225b8f00283e63, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0c7355975d6b44dbbe3a77df942b4d43'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.116658 2113 rpcz_store.cc:276] 0901 01:52:30.497429 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.499152 (+ 1723us) service_pool.cc:225] Handling call
0901 01:52:35.115909 (+4616757us) ts_tablet_manager.cc:1145] Deleting tablet 824229183e024feaa975c7f0387c9c1a
0901 01:52:35.116435 (+ 526us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.116496 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6528,"tablet-delete.queue_time_us":4616617}
E20250901 01:52:35.116950 1696 catalog_manager.cc:6205] Error selecting replicas for tablet dcc60bb000f440f1aca9133928b5ce20, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2fbca8463d6546c9b8f5b018fc60251d'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.117215 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 465c02a130c54e09b5a74fb4dd2ef762, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ed6fd284be6f4ae3a008e21e7ce17930'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.117345 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 75883d9fd00e429aa8853903a9fdd7e4, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2e67ad583a0b413e9190cb55137ccf3a'. Need at least 3 replicas, but only 2 tablet servers are available
W20250901 01:52:35.117513 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 824229183e024feaa975c7f0387c9c1a because the tablet was not found. No further retry: Not found: Tablet not found: 824229183e024feaa975c7f0387c9c1a
I20250901 01:52:35.117594 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 336) took 4619 ms. Trace:
I20250901 01:52:35.117661 2113 rpcz_store.cc:276] 0901 01:52:30.497701 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.499308 (+ 1607us) service_pool.cc:225] Handling call
0901 01:52:35.117520 (+4618212us) ts_tablet_manager.cc:1145] Deleting tablet c9a4c80bec1d41458974a0c449aecaf7
0901 01:52:35.117528 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.117580 (+ 52us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4618075}
W20250901 01:52:35.117765 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 617cc33d0d23439cb5f1e0c31bf11281 because the tablet was not found. No further retry: Not found: Tablet not found: 617cc33d0d23439cb5f1e0c31bf11281
E20250901 01:52:35.118028 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f8f923ddcd2740bcb7e521f4f57c4034, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_e09ac1a9050547038400d4b52299d22f'. Need at least 3 replicas, but only 2 tablet servers are available
W20250901 01:52:35.118175 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet c9a4c80bec1d41458974a0c449aecaf7 because the tablet was not found. No further retry: Not found: Tablet not found: c9a4c80bec1d41458974a0c449aecaf7
I20250901 01:52:35.118724 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 335) took 4621 ms. Trace:
I20250901 01:52:35.118840 2113 rpcz_store.cc:276] 0901 01:52:30.497620 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.499621 (+ 2001us) service_pool.cc:225] Handling call
0901 01:52:35.118460 (+4618839us) ts_tablet_manager.cc:1145] Deleting tablet 40d74d7b5a3b4d1dae09eebd9e07a0c4
0901 01:52:35.118469 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.118709 (+ 240us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4618671}
W20250901 01:52:35.119331 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 40d74d7b5a3b4d1dae09eebd9e07a0c4 because the tablet was not found. No further retry: Not found: Tablet not found: 40d74d7b5a3b4d1dae09eebd9e07a0c4
I20250901 01:52:35.119750 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 340) took 4621 ms. Trace:
E20250901 01:52:35.120016 1696 catalog_manager.cc:6205] Error selecting replicas for tablet b1ba0e19dc0b462e80c537bf2d183f5b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_66b111271c15410aab23682f8f6081ac'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.120152 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 8e10a01e77d14cf28c2c5e0205e595b9, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_83cf2d86a1284544ba2e7aa9ebb66bb7'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.120239 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5f0e054f62de4cf6bc6f69ad6664b192, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ef6832d0845044ed9f165ca3541b4c3f'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.119899 2113 rpcz_store.cc:276] 0901 01:52:30.498044 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.499963 (+ 1919us) service_pool.cc:225] Handling call
0901 01:52:35.119645 (+4619682us) ts_tablet_manager.cc:1145] Deleting tablet 12c04e0c1f804179ae23509cdb24d395
0901 01:52:35.119658 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.119731 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4619542}
E20250901 01:52:35.120348 1696 catalog_manager.cc:6205] Error selecting replicas for tablet fd2ce82a4ee24d0faa44980a3967e187, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_63d0f2ead9d946d39ce40162ebd769d6'. Need at least 3 replicas, but only 2 tablet servers are available
W20250901 01:52:35.120769 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 12c04e0c1f804179ae23509cdb24d395 because the tablet was not found. No further retry: Not found: Tablet not found: 12c04e0c1f804179ae23509cdb24d395
I20250901 01:52:35.121130 1484 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet d0c3c363dda444519a17dca63ad9caa7
I20250901 01:52:35.121259 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 342) took 4623 ms. Trace:
I20250901 01:52:35.121354 2113 rpcz_store.cc:276] 0901 01:52:30.498201 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.500187 (+ 1986us) service_pool.cc:225] Handling call
0901 01:52:35.121139 (+4620952us) ts_tablet_manager.cc:1145] Deleting tablet 50e3da53433f4c14b57db8a733fa53bc
0901 01:52:35.121150 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.121239 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4620841}
E20250901 01:52:35.121300 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae9a4ad4fdfb4771ba23a91f7007447e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_5e84ae9a9c2a4084a1f9c5ffb4594de2'. Need at least 3 replicas, but only 2 tablet servers are available
W20250901 01:52:35.121510 1484 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet d0c3c363dda444519a17dca63ad9caa7: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
E20250901 01:52:35.121754 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 086aa948f93644da8e5835b0fac87ddd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b4fbc34da806475c912052d0fab3fc53'. Need at least 3 replicas, but only 2 tablet servers are available
W20250901 01:52:35.121820 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 50e3da53433f4c14b57db8a733fa53bc because the tablet was not found. No further retry: Not found: Tablet not found: 50e3da53433f4c14b57db8a733fa53bc
E20250901 01:52:35.122156 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 082db24bb3fd4089950504145a20a075, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7f5b278599994063862981b6b041ca8c'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.122284 1696 catalog_manager.cc:6205] Error selecting replicas for tablet fefbf5ae030446b292d8068e95577534, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_3dde393d3a564e84b8ef08c0d25265ce'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.122375 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 93aa22afc0ad4d7fa24807ea3c5d5c21, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0f4b4094d2974b7591875375eda6bed0'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.122476 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9074543dea9949e7a085717b40913348, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_53e30205c5fd488eb3b48c685a159d04'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.122562 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 3a62034e688348ed9e2ac9de009ed3a2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_e225c82a2d6045f6b16915cdb72f5734'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.122663 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 18498c9cb09141d08c1c5c0854b63ebd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77f4810bc83c4d948531da2a45a3792e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.122738 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f5e3316adeae47cd848833dd84b972bb, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_fc962c8bc653498e85cc0bb41639e9ad'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.122834 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 81e8c6567953484ab000fe2eb4398301, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_a920a3ffbabf4d83a6563c3423cde12a'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.122917 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 6ce9521b925b4f97bb4a810ce77e147e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_9e04f976a93f4d30baccf90fa3622dfe'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.123015 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2f61edb50d7a42198bfd70639c7bf4de, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_a5c783cbdb694af69047b31d99ff8def'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.124166 2753 raft_consensus.cc:491] T f8d9bbf0c5be462a8ddc539238cbdf97 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:35.124301 2753 raft_consensus.cc:513] T f8d9bbf0c5be462a8ddc539238cbdf97 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:35.124908 2753 leader_election.cc:290] T f8d9bbf0c5be462a8ddc539238cbdf97 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:35.125319 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet d0c3c363dda444519a17dca63ad9caa7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:35 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:35.125805 893 tablet_service.cc:1515] Processing DeleteTablet for tablet d0c3c363dda444519a17dca63ad9caa7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:35 UTC) from {username='slave'} at 127.0.0.1:34472
W20250901 01:52:35.127278 858 leader_election.cc:336] T f8d9bbf0c5be462a8ddc539238cbdf97 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:35.127504 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 344) took 4629 ms. Trace:
I20250901 01:52:35.127600 2113 rpcz_store.cc:276] 0901 01:52:30.498352 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.500372 (+ 2020us) service_pool.cc:225] Handling call
0901 01:52:35.123021 (+4622649us) ts_tablet_manager.cc:1145] Deleting tablet 1286c0ab85704837a99b6e5a62184d77
0901 01:52:35.123035 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.127482 (+ 4447us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4622357}
W20250901 01:52:35.128083 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 1286c0ab85704837a99b6e5a62184d77 because the tablet was not found. No further retry: Not found: Tablet not found: 1286c0ab85704837a99b6e5a62184d77
I20250901 01:52:35.128645 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 330) took 4631 ms. Trace:
I20250901 01:52:35.128785 2113 rpcz_store.cc:276] 0901 01:52:30.497135 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.498762 (+ 1627us) service_pool.cc:225] Handling call
0901 01:52:35.128548 (+4629786us) ts_tablet_manager.cc:1145] Deleting tablet 822382a1781b4304bce29355ce83730e
0901 01:52:35.128558 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.128629 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4613275}
W20250901 01:52:35.129364 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 822382a1781b4304bce29355ce83730e because the tablet was not found. No further retry: Not found: Tablet not found: 822382a1781b4304bce29355ce83730e
I20250901 01:52:35.129753 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 332) took 4632 ms. Trace:
I20250901 01:52:35.129884 2113 rpcz_store.cc:276] 0901 01:52:30.497344 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.499088 (+ 1744us) service_pool.cc:225] Handling call
0901 01:52:35.129663 (+4630575us) ts_tablet_manager.cc:1145] Deleting tablet abe27e749fa0485195e0064a49d5431e
0901 01:52:35.129674 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.129735 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4614331}
W20250901 01:52:35.130352 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet abe27e749fa0485195e0064a49d5431e because the tablet was not found. No further retry: Not found: Tablet not found: abe27e749fa0485195e0064a49d5431e
I20250901 01:52:35.130760 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "34ebb441c27747ec860ff2e25f9a0adc" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:35.130963 1052 raft_consensus.cc:2466] T 34ebb441c27747ec860ff2e25f9a0adc P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00c282f779c1477d920f24dc0dc16628 in term 0.
I20250901 01:52:35.131263 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 337) took 4633 ms. Trace:
I20250901 01:52:35.132146 2113 rpcz_store.cc:276] 0901 01:52:30.497785 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.499818 (+ 2033us) service_pool.cc:225] Handling call
0901 01:52:35.130624 (+4630806us) ts_tablet_manager.cc:1145] Deleting tablet 121803e79ab64ec0b948b448646b9d0a
0901 01:52:35.130636 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.131247 (+ 611us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4615251}
W20250901 01:52:35.132683 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 121803e79ab64ec0b948b448646b9d0a because the tablet was not found. No further retry: Not found: Tablet not found: 121803e79ab64ec0b948b448646b9d0a
I20250901 01:52:35.138301 1484 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 0e4009963cfe42d6ba164fc828897f09 (table default.Table_1adc70cccd2f4cb3a79df96861cb4181 [id=24fffc7ebf5a47bdb37211910301acf7]). Expected version 1 got 0
I20250901 01:52:35.139389 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f8d9bbf0c5be462a8ddc539238cbdf97" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
E20250901 01:52:35.139564 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 494b6488aa0d4355b4caa46004267465, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7c21b68ef2e74136a807c1dd90e17f4e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.139758 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 01deaae6abb141789f0cd240f3058c73, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7baa92b46d98454a891222be49a6ca48'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.139889 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 0ee3f35392954817b350e5f48653be78, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_442e11b9211c4b209c68f653d7bfc313'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.139887 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 338) took 4641 ms. Trace:
E20250901 01:52:35.140034 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae4844183a3c4a04a775e6fd878391a3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_4bd66ab99a454974aa0f7f4290edfda2'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.140156 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9745582cebfe47b58c21d0191ab4cb9b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77728af9eee44f29a1e2131d5b75b7ca'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.140043 2113 rpcz_store.cc:276] 0901 01:52:30.497866 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.499871 (+ 2005us) service_pool.cc:225] Handling call
0901 01:52:35.132938 (+4633067us) ts_tablet_manager.cc:1145] Deleting tablet 82d163681fda47f38d79863bd28f576d
0901 01:52:35.139340 (+ 6402us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.139812 (+ 472us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4617526}
E20250901 01:52:35.140295 1696 catalog_manager.cc:6205] Error selecting replicas for tablet e2ed32b3292141278af28026528e96c2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b5077f55951d48be92a67407ffe26a08'. Need at least 3 replicas, but only 2 tablet servers are available
W20250901 01:52:35.140690 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 82d163681fda47f38d79863bd28f576d because the tablet was not found. No further retry: Not found: Tablet not found: 82d163681fda47f38d79863bd28f576d
E20250901 01:52:35.144903 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9f62a9d97f514747b940b488907369f3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d53a456dbca64daf92aeb5fb4517b417'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.145181 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5a7fa68d29644ae8afd2711e39dde59a, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d24e3a18ef4843fdb7b74ce23d47cf9e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.145309 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 415c9d7b552b42c384b444a20491f65c, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_97c44ea002094df8a6af3dccd0099c39'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.145479 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ff2bd017417c41b5b8225b8f00283e63, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0c7355975d6b44dbbe3a77df942b4d43'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.145604 1696 catalog_manager.cc:6205] Error selecting replicas for tablet dcc60bb000f440f1aca9133928b5ce20, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2fbca8463d6546c9b8f5b018fc60251d'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.145752 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 465c02a130c54e09b5a74fb4dd2ef762, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ed6fd284be6f4ae3a008e21e7ce17930'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.145838 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 75883d9fd00e429aa8853903a9fdd7e4, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2e67ad583a0b413e9190cb55137ccf3a'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.145944 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f8f923ddcd2740bcb7e521f4f57c4034, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_e09ac1a9050547038400d4b52299d22f'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.146037 1696 catalog_manager.cc:6205] Error selecting replicas for tablet b1ba0e19dc0b462e80c537bf2d183f5b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_66b111271c15410aab23682f8f6081ac'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.146129 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 8e10a01e77d14cf28c2c5e0205e595b9, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_83cf2d86a1284544ba2e7aa9ebb66bb7'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.146209 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5f0e054f62de4cf6bc6f69ad6664b192, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ef6832d0845044ed9f165ca3541b4c3f'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.146298 1696 catalog_manager.cc:6205] Error selecting replicas for tablet fd2ce82a4ee24d0faa44980a3967e187, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_63d0f2ead9d946d39ce40162ebd769d6'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.146390 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae9a4ad4fdfb4771ba23a91f7007447e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_5e84ae9a9c2a4084a1f9c5ffb4594de2'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.146421 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 339) took 4648 ms. Trace:
E20250901 01:52:35.146495 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 086aa948f93644da8e5835b0fac87ddd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b4fbc34da806475c912052d0fab3fc53'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.146584 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 082db24bb3fd4089950504145a20a075, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7f5b278599994063862981b6b041ca8c'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.146497 2113 rpcz_store.cc:276] 0901 01:52:30.497961 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.499919 (+ 1958us) service_pool.cc:225] Handling call
0901 01:52:35.140971 (+4641052us) ts_tablet_manager.cc:1145] Deleting tablet 4a87e68850c4459eb2c8eafac89c6aa9
0901 01:52:35.146328 (+ 5357us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.146401 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4625514}
E20250901 01:52:35.146698 1696 catalog_manager.cc:6205] Error selecting replicas for tablet fefbf5ae030446b292d8068e95577534, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_3dde393d3a564e84b8ef08c0d25265ce'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.146785 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 93aa22afc0ad4d7fa24807ea3c5d5c21, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0f4b4094d2974b7591875375eda6bed0'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.146840 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 341) took 4648 ms. Trace:
E20250901 01:52:35.146893 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9074543dea9949e7a085717b40913348, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_53e30205c5fd488eb3b48c685a159d04'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.146978 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 3a62034e688348ed9e2ac9de009ed3a2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_e225c82a2d6045f6b16915cdb72f5734'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.146919 2113 rpcz_store.cc:276] 0901 01:52:30.498125 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.500128 (+ 2003us) service_pool.cc:225] Handling call
0901 01:52:35.146750 (+4646622us) ts_tablet_manager.cc:1145] Deleting tablet 670d90a37b934579b353bb4e4820127b
0901 01:52:35.146761 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.146824 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4631253}
E20250901 01:52:35.147073 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 18498c9cb09141d08c1c5c0854b63ebd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77f4810bc83c4d948531da2a45a3792e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.147158 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f5e3316adeae47cd848833dd84b972bb, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_fc962c8bc653498e85cc0bb41639e9ad'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.147183 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 343) took 4648 ms. Trace:
E20250901 01:52:35.147246 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 81e8c6567953484ab000fe2eb4398301, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_a920a3ffbabf4d83a6563c3423cde12a'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.147244 2113 rpcz_store.cc:276] 0901 01:52:30.498278 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.500318 (+ 2040us) service_pool.cc:225] Handling call
0901 01:52:35.147099 (+4646781us) ts_tablet_manager.cc:1145] Deleting tablet f8a5886ddb644a86b76df44aab309057
0901 01:52:35.147108 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.147171 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4631557}
E20250901 01:52:35.147320 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 6ce9521b925b4f97bb4a810ce77e147e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_9e04f976a93f4d30baccf90fa3622dfe'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.147405 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2f61edb50d7a42198bfd70639c7bf4de, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_a5c783cbdb694af69047b31d99ff8def'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.147507 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 345) took 4649 ms. Trace:
I20250901 01:52:35.147571 2113 rpcz_store.cc:276] 0901 01:52:30.498431 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.500546 (+ 2115us) service_pool.cc:225] Handling call
0901 01:52:35.147414 (+4646868us) ts_tablet_manager.cc:1145] Deleting tablet 9473f0ef978c4bfaa41fb7b0729e8863
0901 01:52:35.147422 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.147493 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4631831}
I20250901 01:52:35.147842 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 348) took 4643 ms. Trace:
I20250901 01:52:35.147909 2113 rpcz_store.cc:276] 0901 01:52:30.504269 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.504315 (+ 46us) service_pool.cc:225] Handling call
0901 01:52:35.147753 (+4643438us) ts_tablet_manager.cc:1145] Deleting tablet 55bd2daa07294358999956cb553d973f
0901 01:52:35.147762 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.147830 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4632129}
I20250901 01:52:35.148156 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 351) took 4601 ms. Trace:
I20250901 01:52:35.148216 2113 rpcz_store.cc:276] 0901 01:52:30.546862 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.547041 (+ 179us) service_pool.cc:225] Handling call
0901 01:52:35.148089 (+4601048us) ts_tablet_manager.cc:1145] Deleting tablet cf806854a8074b0798b3f9f3df587c63
0901 01:52:35.148096 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.148143 (+ 47us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4600832}
I20250901 01:52:35.148468 2113 tablet_replica.cc:331] T f195b040a6ae4b7b83a8308ab0a12b37 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
W20250901 01:52:35.148790 992 consensus_peers.cc:489] T 61ddcf86c9f34726939cc116434f0822 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250901 01:52:35.152710 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 4a87e68850c4459eb2c8eafac89c6aa9 because the tablet was not found. No further retry: Not found: Tablet not found: 4a87e68850c4459eb2c8eafac89c6aa9
W20250901 01:52:35.152942 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 670d90a37b934579b353bb4e4820127b because the tablet was not found. No further retry: Not found: Tablet not found: 670d90a37b934579b353bb4e4820127b
W20250901 01:52:35.153146 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet f8a5886ddb644a86b76df44aab309057 because the tablet was not found. No further retry: Not found: Tablet not found: f8a5886ddb644a86b76df44aab309057
W20250901 01:52:35.153312 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 9473f0ef978c4bfaa41fb7b0729e8863 because the tablet was not found. No further retry: Not found: Tablet not found: 9473f0ef978c4bfaa41fb7b0729e8863
W20250901 01:52:35.153483 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 55bd2daa07294358999956cb553d973f because the tablet was not found. No further retry: Not found: Tablet not found: 55bd2daa07294358999956cb553d973f
W20250901 01:52:35.153682 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet cf806854a8074b0798b3f9f3df587c63 because the tablet was not found. No further retry: Not found: Tablet not found: cf806854a8074b0798b3f9f3df587c63
I20250901 01:52:35.155681 2113 raft_consensus.cc:2241] T f195b040a6ae4b7b83a8308ab0a12b37 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:35.155929 2113 pending_rounds.cc:70] T f195b040a6ae4b7b83a8308ab0a12b37 P fd27ab6665304aaea53adbd102dd4f58: Trying to abort 1 pending ops.
I20250901 01:52:35.155982 2113 pending_rounds.cc:77] T f195b040a6ae4b7b83a8308ab0a12b37 P fd27ab6665304aaea53adbd102dd4f58: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7195408606088716288 op_type: NO_OP noop_request { }
I20250901 01:52:35.156070 2113 raft_consensus.cc:2887] T f195b040a6ae4b7b83a8308ab0a12b37 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250901 01:52:35.157575 2113 raft_consensus.cc:2270] T f195b040a6ae4b7b83a8308ab0a12b37 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:35.158425 2113 ts_tablet_manager.cc:1905] T f195b040a6ae4b7b83a8308ab0a12b37 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:35.159873 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f1359f99b17b49fa864380a63dbafbeb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.160612 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_fb99a0b55326479a89c164dde45f96cd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.161950 2118 tablet_replica.cc:331] T 88ceba7f48b04d7885cc0164668700a5 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:35.162154 2118 raft_consensus.cc:2241] T 88ceba7f48b04d7885cc0164668700a5 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:35.162302 2118 raft_consensus.cc:2270] T 88ceba7f48b04d7885cc0164668700a5 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:35.162822 2118 ts_tablet_manager.cc:1905] T 88ceba7f48b04d7885cc0164668700a5 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:35.164642 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e92753de71df4af6bad0a6ad9da118ac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.166067 2113 ts_tablet_manager.cc:1918] T f195b040a6ae4b7b83a8308ab0a12b37 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:35.166185 2113 log.cc:1199] T f195b040a6ae4b7b83a8308ab0a12b37 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/f195b040a6ae4b7b83a8308ab0a12b37
I20250901 01:52:35.166414 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e92233f08a9749479678dcfdbdf1d9dd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.166543 2113 ts_tablet_manager.cc:1939] T f195b040a6ae4b7b83a8308ab0a12b37 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:35.166556 921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "34ebb441c27747ec860ff2e25f9a0adc" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:35.166723 921 raft_consensus.cc:2466] T 34ebb441c27747ec860ff2e25f9a0adc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:35.167317 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 358) took 4539 ms. Trace:
I20250901 01:52:35.167414 2113 rpcz_store.cc:276] 0901 01:52:30.627404 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.627457 (+ 53us) service_pool.cc:225] Handling call
0901 01:52:35.148388 (+4520931us) ts_tablet_manager.cc:1145] Deleting tablet f195b040a6ae4b7b83a8308ab0a12b37
0901 01:52:35.148430 (+ 42us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.162993 (+ 14563us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:35.165969 (+ 2976us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:35.167297 (+ 1328us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4520753}
I20250901 01:52:35.167855 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet f195b040a6ae4b7b83a8308ab0a12b37 (table default.Table_8b1086c9a4c44239851c586ea7267e3c [id=523de440fbab4bf687c0f6168e48513c]) successfully deleted
I20250901 01:52:35.167881 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_931f01848c0b481996b9cc72db572ee8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.169104 921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2973b59bb1df41d18d9a5b6a29ae336f" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:35.169250 921 raft_consensus.cc:3058] T 2973b59bb1df41d18d9a5b6a29ae336f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:35.169430 1483 master_service.cc:948] ReplaceTablet: received request to replace tablet 5f5396045979450193d461ba1334a5c9 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:35.171991 921 raft_consensus.cc:2466] T 2973b59bb1df41d18d9a5b6a29ae336f P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:35.172580 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_120d3e748acd4173822aa8bc13f87e34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.172940 2113 tablet_replica.cc:331] T abf6e3be60f74872b2308dc95bf9cd37 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:35.173131 2113 raft_consensus.cc:2241] T abf6e3be60f74872b2308dc95bf9cd37 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:35.173477 2113 raft_consensus.cc:2270] T abf6e3be60f74872b2308dc95bf9cd37 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:35.174050 2113 ts_tablet_manager.cc:1905] T abf6e3be60f74872b2308dc95bf9cd37 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:35.174672 1151 tablet_bootstrap.cc:492] T 6ba8241644984394ada95b561e76467e P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:35.174849 1151 ts_tablet_manager.cc:1397] T 6ba8241644984394ada95b561e76467e P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.150s user 0.015s sys 0.033s
I20250901 01:52:35.175606 1151 raft_consensus.cc:357] T 6ba8241644984394ada95b561e76467e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:35.175874 1151 raft_consensus.cc:383] T 6ba8241644984394ada95b561e76467e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:35.175977 1151 raft_consensus.cc:738] T 6ba8241644984394ada95b561e76467e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:35.176285 1151 consensus_queue.cc:260] T 6ba8241644984394ada95b561e76467e P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:35.177959 1151 ts_tablet_manager.cc:1428] T 6ba8241644984394ada95b561e76467e P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250901 01:52:35.180759 1151 tablet_bootstrap.cc:492] T 50472a7aa2f04e0fbd65ac6c86c34b3e P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:35.182284 1151 tablet_bootstrap.cc:654] T 50472a7aa2f04e0fbd65ac6c86c34b3e P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:35.184747 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_23566eae0a174555b8c996262510fb3a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.192692 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_bd7a0af9e67a43368980e8081210a2ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.197144 2113 ts_tablet_manager.cc:1918] T abf6e3be60f74872b2308dc95bf9cd37 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:35.197280 2113 log.cc:1199] T abf6e3be60f74872b2308dc95bf9cd37 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/abf6e3be60f74872b2308dc95bf9cd37
I20250901 01:52:35.197652 2113 ts_tablet_manager.cc:1939] T abf6e3be60f74872b2308dc95bf9cd37 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:35.198534 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 360) took 4506 ms. Trace:
I20250901 01:52:35.198642 2113 rpcz_store.cc:276] 0901 01:52:30.691760 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.691824 (+ 64us) service_pool.cc:225] Handling call
0901 01:52:35.172889 (+4481065us) ts_tablet_manager.cc:1145] Deleting tablet abf6e3be60f74872b2308dc95bf9cd37
0901 01:52:35.172900 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.186302 (+ 13402us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:35.197010 (+ 10708us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:35.198510 (+ 1500us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4480867}
I20250901 01:52:35.199146 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet abf6e3be60f74872b2308dc95bf9cd37 (table default.Table_fea398de1f1948809aebf95c387f4b17 [id=6975ac6754db4664a168e25ad61db164]) successfully deleted
I20250901 01:52:35.203356 2113 tablet_replica.cc:331] T 3618a3eb4f174c43805ce7dc0e5315cb P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:35.203568 2113 raft_consensus.cc:2241] T 3618a3eb4f174c43805ce7dc0e5315cb P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:35.203471 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c9c3556590dc483cbd300a7d50a94acc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.203909 2113 raft_consensus.cc:2270] T 3618a3eb4f174c43805ce7dc0e5315cb P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:35.204509 2113 ts_tablet_manager.cc:1905] T 3618a3eb4f174c43805ce7dc0e5315cb P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:35.206789 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5dbc66e6fde64acaa5c125f90e3f2b19"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.208173 859 leader_election.cc:304] T 34ebb441c27747ec860ff2e25f9a0adc P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:35.208667 2753 raft_consensus.cc:2802] T 34ebb441c27747ec860ff2e25f9a0adc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:35.208758 2753 raft_consensus.cc:491] T 34ebb441c27747ec860ff2e25f9a0adc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:35.208810 2753 raft_consensus.cc:3058] T 34ebb441c27747ec860ff2e25f9a0adc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:35.211294 2753 raft_consensus.cc:513] T 34ebb441c27747ec860ff2e25f9a0adc P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:35.212110 2753 leader_election.cc:290] T 34ebb441c27747ec860ff2e25f9a0adc P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:35.213747 858 leader_election.cc:336] T 34ebb441c27747ec860ff2e25f9a0adc P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:35.214342 1489 master_service.cc:948] ReplaceTablet: received request to replace tablet 6ba88af177874544b764cc32420e7ac9 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:35.219058 2113 ts_tablet_manager.cc:1918] T 3618a3eb4f174c43805ce7dc0e5315cb P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250901 01:52:35.219199 2113 log.cc:1199] T 3618a3eb4f174c43805ce7dc0e5315cb P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/3618a3eb4f174c43805ce7dc0e5315cb
I20250901 01:52:35.219601 2113 ts_tablet_manager.cc:1939] T 3618a3eb4f174c43805ce7dc0e5315cb P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:35.222743 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 361) took 4517 ms. Trace:
I20250901 01:52:35.222868 2113 rpcz_store.cc:276] 0901 01:52:30.705177 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.705237 (+ 60us) service_pool.cc:225] Handling call
0901 01:52:35.203301 (+4498064us) ts_tablet_manager.cc:1145] Deleting tablet 3618a3eb4f174c43805ce7dc0e5315cb
0901 01:52:35.203313 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.212575 (+ 9262us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:35.218911 (+ 6336us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:35.222713 (+ 3802us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4497848}
I20250901 01:52:35.223433 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 3618a3eb4f174c43805ce7dc0e5315cb (table default.Table_d24e3a18ef4843fdb7b74ce23d47cf9e [id=31a2e50c32644321825da40176127015]) successfully deleted
I20250901 01:52:35.225580 2113 tablet_replica.cc:331] T 15e4d972e8254cf899c3e0b71b6e2bd9 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:35.225798 2113 raft_consensus.cc:2241] T 15e4d972e8254cf899c3e0b71b6e2bd9 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:35.226153 2113 raft_consensus.cc:2270] T 15e4d972e8254cf899c3e0b71b6e2bd9 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:35.226877 2113 ts_tablet_manager.cc:1905] T 15e4d972e8254cf899c3e0b71b6e2bd9 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:35.231406 859 leader_election.cc:343] T f8d9bbf0c5be462a8ddc539238cbdf97 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:35.231580 859 leader_election.cc:304] T f8d9bbf0c5be462a8ddc539238cbdf97 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:35.232185 2753 raft_consensus.cc:2747] T f8d9bbf0c5be462a8ddc539238cbdf97 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:35.236840 2753 consensus_queue.cc:1035] T d000ba05e12843b4925e622ab55471c0 P 00c282f779c1477d920f24dc0dc16628 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:35.241164 2113 ts_tablet_manager.cc:1918] T 15e4d972e8254cf899c3e0b71b6e2bd9 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:35.241295 2113 log.cc:1199] T 15e4d972e8254cf899c3e0b71b6e2bd9 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/15e4d972e8254cf899c3e0b71b6e2bd9
I20250901 01:52:35.241652 2113 ts_tablet_manager.cc:1939] T 15e4d972e8254cf899c3e0b71b6e2bd9 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
W20250901 01:52:35.242656 992 leader_election.cc:336] T 2973b59bb1df41d18d9a5b6a29ae336f P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:35.245541 2780 raft_consensus.cc:491] T 6ba8241644984394ada95b561e76467e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:35.245715 2780 raft_consensus.cc:513] T 6ba8241644984394ada95b561e76467e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:35.246335 2780 leader_election.cc:290] T 6ba8241644984394ada95b561e76467e P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:35.246296 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_22bb1855c3b843f1a4b3c61851c1c124"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.247869 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4a4f31df66fc474fa151f2452211c4de"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.248857 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c7134ad7a66a4fe69eeb2e0a72c998c0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.252166 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 362) took 4519 ms. Trace:
I20250901 01:52:35.252265 2113 rpcz_store.cc:276] 0901 01:52:30.732424 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.732725 (+ 301us) service_pool.cc:225] Handling call
0901 01:52:35.225506 (+4492781us) ts_tablet_manager.cc:1145] Deleting tablet 15e4d972e8254cf899c3e0b71b6e2bd9
0901 01:52:35.225524 (+ 18us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.238002 (+ 12478us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:35.241054 (+ 3052us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:35.252147 (+ 11093us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4492586}
I20250901 01:52:35.253050 2113 tablet_replica.cc:331] T d1f2d98712ff4ea6aec665512de068bd P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:35.253301 2113 raft_consensus.cc:2241] T d1f2d98712ff4ea6aec665512de068bd P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:35.253562 2113 raft_consensus.cc:2270] T d1f2d98712ff4ea6aec665512de068bd P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:35.254158 2113 ts_tablet_manager.cc:1905] T d1f2d98712ff4ea6aec665512de068bd P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:35.255316 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_07c7c74e39054f3c9bfd9b7392c3ba2f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.259251 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2fde3aec1830442eb270803fa5443e8d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.266057 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d37c75c8846f491b88710adafbe7dee7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.267122 2113 ts_tablet_manager.cc:1918] T d1f2d98712ff4ea6aec665512de068bd P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:35.267244 2113 log.cc:1199] T d1f2d98712ff4ea6aec665512de068bd P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/d1f2d98712ff4ea6aec665512de068bd
I20250901 01:52:35.267590 2113 ts_tablet_manager.cc:1939] T d1f2d98712ff4ea6aec665512de068bd P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:35.267709 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9664847c63db4f40b087fd9615f5ab62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.268386 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 363) took 4531 ms. Trace:
I20250901 01:52:35.268522 2113 rpcz_store.cc:276] 0901 01:52:30.737165 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.737229 (+ 64us) service_pool.cc:225] Handling call
0901 01:52:35.252981 (+4515752us) ts_tablet_manager.cc:1145] Deleting tablet d1f2d98712ff4ea6aec665512de068bd
0901 01:52:35.252998 (+ 17us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.263922 (+ 10924us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:35.267019 (+ 3097us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:35.268364 (+ 1345us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4515548}
I20250901 01:52:35.272130 2113 tablet_replica.cc:331] T 5d794471dbed4477b9d52fd411fa4d10 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:35.272066 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_010d94791cd148ffba915cfea6d36731"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.272343 2113 raft_consensus.cc:2241] T 5d794471dbed4477b9d52fd411fa4d10 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:35.272671 2113 raft_consensus.cc:2270] T 5d794471dbed4477b9d52fd411fa4d10 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:35.273301 2113 ts_tablet_manager.cc:1905] T 5d794471dbed4477b9d52fd411fa4d10 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:35.274062 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_467e76cdd3704edfb96d194abdc7144d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:35.277335 858 consensus_peers.cc:489] T d000ba05e12843b4925e622ab55471c0 P 00c282f779c1477d920f24dc0dc16628 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:35.277740 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_eea753857bec4c2d9c603eb9164b98c5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.280248 1483 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_297737979dae4534a0eef6ca9e87e98d" } modify_external_catalogs: true
I20250901 01:52:35.280465 1483 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_297737979dae4534a0eef6ca9e87e98d" } modify_external_catalogs: true
I20250901 01:52:35.282977 1484 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_a5c783cbdb694af69047b31d99ff8def" } new_table_name: "default.Table_fe79329825c54cdf8c97a14aa8a30384" modify_external_catalogs: true
I20250901 01:52:35.283157 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1283531e7c68450f8896dfbbdfc48e01"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.285285 2113 ts_tablet_manager.cc:1918] T 5d794471dbed4477b9d52fd411fa4d10 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:35.285434 1490 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_74f5d01b84de4566ae19752b8d366240" } new_table_name: "default.Table_e549eeded6454e85b5bb2a0a66f96a61" modify_external_catalogs: true
I20250901 01:52:35.285645 1492 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_d3fd1940d13e4afba8b3bb674bd4e4ee" } modify_external_catalogs: true
I20250901 01:52:35.285964 1492 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_d3fd1940d13e4afba8b3bb674bd4e4ee" } modify_external_catalogs: true
I20250901 01:52:35.287148 2113 log.cc:1199] T 5d794471dbed4477b9d52fd411fa4d10 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/5d794471dbed4477b9d52fd411fa4d10
I20250901 01:52:35.287487 1483 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet b4278a013ae84082bc7a8497b04ac147
I20250901 01:52:35.287616 2113 ts_tablet_manager.cc:1939] T 5d794471dbed4477b9d52fd411fa4d10 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
W20250901 01:52:35.287667 1483 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet b4278a013ae84082bc7a8497b04ac147: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:35.288440 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 364) took 4538 ms. Trace:
I20250901 01:52:35.288516 2113 rpcz_store.cc:276] 0901 01:52:30.749841 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.750653 (+ 812us) service_pool.cc:225] Handling call
0901 01:52:35.272076 (+4521423us) ts_tablet_manager.cc:1145] Deleting tablet 5d794471dbed4477b9d52fd411fa4d10
0901 01:52:35.272088 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.276209 (+ 4121us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:35.285179 (+ 8970us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:35.288420 (+ 3241us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4521235}
I20250901 01:52:35.289093 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet b4278a013ae84082bc7a8497b04ac147 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:35 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:35.289609 893 tablet_service.cc:1515] Processing DeleteTablet for tablet b4278a013ae84082bc7a8497b04ac147 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:35 UTC) from {username='slave'} at 127.0.0.1:34472
W20250901 01:52:35.290975 1484 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2f61edb50d7a42198bfd70639c7bf4de has no consensus state
I20250901 01:52:35.292528 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 366) took 4287 ms. Trace:
I20250901 01:52:35.292577 1489 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_213d020a80574a0cbdd476d5465feb15" } modify_external_catalogs: true
I20250901 01:52:35.292634 2113 rpcz_store.cc:276] 0901 01:52:31.005043 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:31.005686 (+ 643us) service_pool.cc:225] Handling call
0901 01:52:35.292351 (+4286665us) ts_tablet_manager.cc:1145] Deleting tablet f8d9bbf0c5be462a8ddc539238cbdf97
0901 01:52:35.292364 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.292509 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4286442}
I20250901 01:52:35.292739 1489 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_213d020a80574a0cbdd476d5465feb15" } modify_external_catalogs: true
W20250901 01:52:35.293076 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet f8d9bbf0c5be462a8ddc539238cbdf97 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f8d9bbf0c5be462a8ddc539238cbdf97 already in progress: creating tablet
I20250901 01:52:35.293358 2113 tablet_replica.cc:331] T e87d1281f35648b0b115b67c95e244e7 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:35.293533 2113 raft_consensus.cc:2241] T e87d1281f35648b0b115b67c95e244e7 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:35.293706 2113 raft_consensus.cc:2270] T e87d1281f35648b0b115b67c95e244e7 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:35.295926 2113 ts_tablet_manager.cc:1905] T e87d1281f35648b0b115b67c95e244e7 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:35.298226 1055 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "34ebb441c27747ec860ff2e25f9a0adc" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58"
I20250901 01:52:35.298415 1055 raft_consensus.cc:3058] T 34ebb441c27747ec860ff2e25f9a0adc P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
E20250901 01:52:35.299340 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 494b6488aa0d4355b4caa46004267465, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7c21b68ef2e74136a807c1dd90e17f4e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.299468 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 01deaae6abb141789f0cd240f3058c73, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7baa92b46d98454a891222be49a6ca48'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.299541 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 0ee3f35392954817b350e5f48653be78, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_442e11b9211c4b209c68f653d7bfc313'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.299628 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae4844183a3c4a04a775e6fd878391a3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_4bd66ab99a454974aa0f7f4290edfda2'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.299697 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9745582cebfe47b58c21d0191ab4cb9b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77728af9eee44f29a1e2131d5b75b7ca'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.299789 1696 catalog_manager.cc:6205] Error selecting replicas for tablet e2ed32b3292141278af28026528e96c2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b5077f55951d48be92a67407ffe26a08'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.299860 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9f62a9d97f514747b940b488907369f3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d53a456dbca64daf92aeb5fb4517b417'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.299935 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5a7fa68d29644ae8afd2711e39dde59a, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d24e3a18ef4843fdb7b74ce23d47cf9e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.299998 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 415c9d7b552b42c384b444a20491f65c, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_97c44ea002094df8a6af3dccd0099c39'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.300072 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ff2bd017417c41b5b8225b8f00283e63, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0c7355975d6b44dbbe3a77df942b4d43'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.300135 1696 catalog_manager.cc:6205] Error selecting replicas for tablet dcc60bb000f440f1aca9133928b5ce20, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2fbca8463d6546c9b8f5b018fc60251d'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.300208 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 465c02a130c54e09b5a74fb4dd2ef762, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ed6fd284be6f4ae3a008e21e7ce17930'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.300273 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 75883d9fd00e429aa8853903a9fdd7e4, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2e67ad583a0b413e9190cb55137ccf3a'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.300902 1055 raft_consensus.cc:2466] T 34ebb441c27747ec860ff2e25f9a0adc P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c282f779c1477d920f24dc0dc16628 in term 1.
E20250901 01:52:35.300382 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f8f923ddcd2740bcb7e521f4f57c4034, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_e09ac1a9050547038400d4b52299d22f'. Need at least 3 replicas, but only 2 tablet servers are available
W20250901 01:52:35.302641 1490 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 26e3ad68393547e5b6e3dd6b9ec1dbbe consensus state has no leader
I20250901 01:52:35.305202 2780 raft_consensus.cc:491] T 088ef4861dc14f81ae470c3aaee5b4e1 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:35.305320 2780 raft_consensus.cc:513] T 088ef4861dc14f81ae470c3aaee5b4e1 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:35.305893 2780 leader_election.cc:290] T 088ef4861dc14f81ae470c3aaee5b4e1 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:35.307348 1490 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_d8e85e65a1914147a2c1d1246646fcd2" } modify_external_catalogs: true
I20250901 01:52:35.307507 1490 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_d8e85e65a1914147a2c1d1246646fcd2" } modify_external_catalogs: true
I20250901 01:52:35.308957 1484 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 0e4009963cfe42d6ba164fc828897f09 (table default.Table_1adc70cccd2f4cb3a79df96861cb4181 [id=24fffc7ebf5a47bdb37211910301acf7]). Expected version 1 got 0
I20250901 01:52:35.315564 2113 ts_tablet_manager.cc:1918] T e87d1281f35648b0b115b67c95e244e7 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 01:52:35.315683 2113 log.cc:1199] T e87d1281f35648b0b115b67c95e244e7 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/e87d1281f35648b0b115b67c95e244e7
I20250901 01:52:35.316026 2113 ts_tablet_manager.cc:1939] T e87d1281f35648b0b115b67c95e244e7 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:35.316845 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 367) took 4311 ms. Trace:
I20250901 01:52:35.316924 2113 rpcz_store.cc:276] 0901 01:52:31.005148 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:31.005934 (+ 786us) service_pool.cc:225] Handling call
0901 01:52:35.293308 (+4287374us) ts_tablet_manager.cc:1145] Deleting tablet e87d1281f35648b0b115b67c95e244e7
0901 01:52:35.293318 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.301739 (+ 8421us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:35.315451 (+ 13712us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:35.316825 (+ 1374us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4287243}
I20250901 01:52:35.317324 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet e87d1281f35648b0b115b67c95e244e7 (table default.Table_f840789a58984799bd8057128f6ea439 [id=4453959e273c473fadd737f56f7db3dd]) successfully deleted
I20250901 01:52:35.318454 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 375) took 4135 ms. Trace:
I20250901 01:52:35.319787 2113 rpcz_store.cc:276] 0901 01:52:31.182960 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:31.183014 (+ 54us) service_pool.cc:225] Handling call
0901 01:52:35.318331 (+4135317us) ts_tablet_manager.cc:1145] Deleting tablet 2df2240631844defa433270a3829d65b
0901 01:52:35.318341 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.318437 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4135106}
W20250901 01:52:35.320236 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 2df2240631844defa433270a3829d65b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2df2240631844defa433270a3829d65b already in progress: creating tablet
E20250901 01:52:35.320545 1696 catalog_manager.cc:6205] Error selecting replicas for tablet b1ba0e19dc0b462e80c537bf2d183f5b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_66b111271c15410aab23682f8f6081ac'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.320595 2113 tablet_replica.cc:331] T 616ff34e8d314e0aa0e11ce60be11789 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
E20250901 01:52:35.320739 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 8e10a01e77d14cf28c2c5e0205e595b9, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_83cf2d86a1284544ba2e7aa9ebb66bb7'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.320878 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5f0e054f62de4cf6bc6f69ad6664b192, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ef6832d0845044ed9f165ca3541b4c3f'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.320874 2113 raft_consensus.cc:2241] T 616ff34e8d314e0aa0e11ce60be11789 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
E20250901 01:52:35.321026 1696 catalog_manager.cc:6205] Error selecting replicas for tablet fd2ce82a4ee24d0faa44980a3967e187, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_63d0f2ead9d946d39ce40162ebd769d6'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.321154 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae9a4ad4fdfb4771ba23a91f7007447e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_5e84ae9a9c2a4084a1f9c5ffb4594de2'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.321249 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 086aa948f93644da8e5835b0fac87ddd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b4fbc34da806475c912052d0fab3fc53'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.321327 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 082db24bb3fd4089950504145a20a075, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7f5b278599994063862981b6b041ca8c'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.321409 1696 catalog_manager.cc:6205] Error selecting replicas for tablet fefbf5ae030446b292d8068e95577534, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_3dde393d3a564e84b8ef08c0d25265ce'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.321651 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 93aa22afc0ad4d7fa24807ea3c5d5c21, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0f4b4094d2974b7591875375eda6bed0'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.321766 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9074543dea9949e7a085717b40913348, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_53e30205c5fd488eb3b48c685a159d04'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.321857 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 3a62034e688348ed9e2ac9de009ed3a2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_e225c82a2d6045f6b16915cdb72f5734'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.321949 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 18498c9cb09141d08c1c5c0854b63ebd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77f4810bc83c4d948531da2a45a3792e'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.321664 2113 raft_consensus.cc:2270] T 616ff34e8d314e0aa0e11ce60be11789 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
E20250901 01:52:35.322036 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f5e3316adeae47cd848833dd84b972bb, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_fc962c8bc653498e85cc0bb41639e9ad'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.322130 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 81e8c6567953484ab000fe2eb4398301, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_a920a3ffbabf4d83a6563c3423cde12a'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.322213 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 6ce9521b925b4f97bb4a810ce77e147e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_9e04f976a93f4d30baccf90fa3622dfe'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.322310 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2f61edb50d7a42198bfd70639c7bf4de, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_fe79329825c54cdf8c97a14aa8a30384'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.322851 2113 ts_tablet_manager.cc:1905] T 616ff34e8d314e0aa0e11ce60be11789 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:35.327972 1492 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 7954393c8c1c4457ab8bb723c03a9c8b
W20250901 01:52:35.331524 1492 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 7954393c8c1c4457ab8bb723c03a9c8b: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:35.334574 2113 ts_tablet_manager.cc:1918] T 616ff34e8d314e0aa0e11ce60be11789 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250901 01:52:35.334692 2113 log.cc:1199] T 616ff34e8d314e0aa0e11ce60be11789 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/616ff34e8d314e0aa0e11ce60be11789
I20250901 01:52:35.335108 2113 ts_tablet_manager.cc:1939] T 616ff34e8d314e0aa0e11ce60be11789 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:35.335995 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 376) took 4127 ms. Trace:
I20250901 01:52:35.335426 1490 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 315c15cf0afc44d4a79cb29294281d60
I20250901 01:52:35.336117 2113 rpcz_store.cc:276] 0901 01:52:31.208282 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:31.208332 (+ 50us) service_pool.cc:225] Handling call
0901 01:52:35.320553 (+4112221us) ts_tablet_manager.cc:1145] Deleting tablet 616ff34e8d314e0aa0e11ce60be11789
0901 01:52:35.320562 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.331568 (+ 11006us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:35.334479 (+ 2911us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:35.335970 (+ 1491us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4111994}
W20250901 01:52:35.336295 1490 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 315c15cf0afc44d4a79cb29294281d60: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:35.336738 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 616ff34e8d314e0aa0e11ce60be11789 (table default.Table_0f4b4094d2974b7591875375eda6bed0 [id=5ee13d46e06a4fa59b1070547573e523]) successfully deleted
I20250901 01:52:35.337224 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 315c15cf0afc44d4a79cb29294281d60 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:35 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:35.337664 1483 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_cca52da43da6439eb88ea9715e4a0d50" } new_table_name: "default.Table_8cd45b1e166e4b0c80647bc3c11cb7ee" modify_external_catalogs: true
I20250901 01:52:35.338083 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 315c15cf0afc44d4a79cb29294281d60 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:35 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:35.338387 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 7954393c8c1c4457ab8bb723c03a9c8b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:35 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:35.338575 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 7954393c8c1c4457ab8bb723c03a9c8b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:35 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:35.339929 1490 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_9e04f976a93f4d30baccf90fa3622dfe" } new_table_name: "default.Table_bd74ae506bbe40489259a7be70c3d10b" modify_external_catalogs: true
I20250901 01:52:35.341293 1492 master_service.cc:948] ReplaceTablet: received request to replace tablet f195b040a6ae4b7b83a8308ab0a12b37 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:35.346964 2113 tablet_replica.cc:331] T d000ba05e12843b4925e622ab55471c0 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:35.350523 2878 raft_consensus.cc:491] T ae21b9ae2e3b472eb67fb5f96388a50f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:35.350658 2878 raft_consensus.cc:513] T ae21b9ae2e3b472eb67fb5f96388a50f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:35.351313 2878 leader_election.cc:290] T ae21b9ae2e3b472eb67fb5f96388a50f P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:35.351923 1490 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6ce9521b925b4f97bb4a810ce77e147e has no consensus state
I20250901 01:52:35.353713 1489 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 3eba54e5ddd14c518f4cbd539268ae37
W20250901 01:52:35.354748 1489 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 3eba54e5ddd14c518f4cbd539268ae37: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:35.355500 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 3eba54e5ddd14c518f4cbd539268ae37 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:35 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:35.355639 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 3eba54e5ddd14c518f4cbd539268ae37 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:35 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:35.359721 2113 raft_consensus.cc:2241] T d000ba05e12843b4925e622ab55471c0 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:35.360215 2113 pending_rounds.cc:70] T d000ba05e12843b4925e622ab55471c0 P fd27ab6665304aaea53adbd102dd4f58: Trying to abort 1 pending ops.
I20250901 01:52:35.360301 2113 pending_rounds.cc:77] T d000ba05e12843b4925e622ab55471c0 P fd27ab6665304aaea53adbd102dd4f58: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7195408605864669184 op_type: NO_OP noop_request { }
I20250901 01:52:35.360430 2113 raft_consensus.cc:2887] T d000ba05e12843b4925e622ab55471c0 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250901 01:52:35.360532 2113 raft_consensus.cc:2270] T d000ba05e12843b4925e622ab55471c0 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
E20250901 01:52:35.362300 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 494b6488aa0d4355b4caa46004267465, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7c21b68ef2e74136a807c1dd90e17f4e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.362440 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 01deaae6abb141789f0cd240f3058c73, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7baa92b46d98454a891222be49a6ca48'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.362516 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 0ee3f35392954817b350e5f48653be78, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_442e11b9211c4b209c68f653d7bfc313'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.362601 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae4844183a3c4a04a775e6fd878391a3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_4bd66ab99a454974aa0f7f4290edfda2'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.362677 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9745582cebfe47b58c21d0191ab4cb9b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77728af9eee44f29a1e2131d5b75b7ca'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.362772 1696 catalog_manager.cc:6205] Error selecting replicas for tablet e2ed32b3292141278af28026528e96c2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b5077f55951d48be92a67407ffe26a08'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.362862 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9f62a9d97f514747b940b488907369f3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d53a456dbca64daf92aeb5fb4517b417'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.362948 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5a7fa68d29644ae8afd2711e39dde59a, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d24e3a18ef4843fdb7b74ce23d47cf9e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.363019 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 415c9d7b552b42c384b444a20491f65c, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_97c44ea002094df8a6af3dccd0099c39'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.363101 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ff2bd017417c41b5b8225b8f00283e63, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0c7355975d6b44dbbe3a77df942b4d43'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.363170 1696 catalog_manager.cc:6205] Error selecting replicas for tablet dcc60bb000f440f1aca9133928b5ce20, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2fbca8463d6546c9b8f5b018fc60251d'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.363250 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 465c02a130c54e09b5a74fb4dd2ef762, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ed6fd284be6f4ae3a008e21e7ce17930'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.363319 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 75883d9fd00e429aa8853903a9fdd7e4, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2e67ad583a0b413e9190cb55137ccf3a'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.363402 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f8f923ddcd2740bcb7e521f4f57c4034, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_e09ac1a9050547038400d4b52299d22f'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.363472 1696 catalog_manager.cc:6205] Error selecting replicas for tablet b1ba0e19dc0b462e80c537bf2d183f5b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_66b111271c15410aab23682f8f6081ac'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.363554 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 8e10a01e77d14cf28c2c5e0205e595b9, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_83cf2d86a1284544ba2e7aa9ebb66bb7'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.363631 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5f0e054f62de4cf6bc6f69ad6664b192, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ef6832d0845044ed9f165ca3541b4c3f'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.364950 2113 ts_tablet_manager.cc:1905] T d000ba05e12843b4925e622ab55471c0 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:35.365593 2871 raft_consensus.cc:491] T d4c9aa723aee4dc4b6237cb5c3722b52 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:35.365721 2871 raft_consensus.cc:513] T d4c9aa723aee4dc4b6237cb5c3722b52 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:35.366426 2871 leader_election.cc:290] T d4c9aa723aee4dc4b6237cb5c3722b52 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
E20250901 01:52:35.367259 1696 catalog_manager.cc:6205] Error selecting replicas for tablet fd2ce82a4ee24d0faa44980a3967e187, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_63d0f2ead9d946d39ce40162ebd769d6'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.367391 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae9a4ad4fdfb4771ba23a91f7007447e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_5e84ae9a9c2a4084a1f9c5ffb4594de2'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.367483 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 086aa948f93644da8e5835b0fac87ddd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b4fbc34da806475c912052d0fab3fc53'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.367555 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 082db24bb3fd4089950504145a20a075, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7f5b278599994063862981b6b041ca8c'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.367645 1696 catalog_manager.cc:6205] Error selecting replicas for tablet fefbf5ae030446b292d8068e95577534, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_3dde393d3a564e84b8ef08c0d25265ce'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.367717 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 93aa22afc0ad4d7fa24807ea3c5d5c21, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0f4b4094d2974b7591875375eda6bed0'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.367847 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9074543dea9949e7a085717b40913348, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_53e30205c5fd488eb3b48c685a159d04'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.367918 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 3a62034e688348ed9e2ac9de009ed3a2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_e225c82a2d6045f6b16915cdb72f5734'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.368005 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 18498c9cb09141d08c1c5c0854b63ebd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77f4810bc83c4d948531da2a45a3792e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.368079 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f5e3316adeae47cd848833dd84b972bb, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_fc962c8bc653498e85cc0bb41639e9ad'. Need at least 3 replicas, but only 2 tablet servers are available
W20250901 01:52:35.368050 858 leader_election.cc:336] T d4c9aa723aee4dc4b6237cb5c3722b52 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
E20250901 01:52:35.368161 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 81e8c6567953484ab000fe2eb4398301, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_a920a3ffbabf4d83a6563c3423cde12a'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.368230 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 6ce9521b925b4f97bb4a810ce77e147e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_bd74ae506bbe40489259a7be70c3d10b'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.368315 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2f61edb50d7a42198bfd70639c7bf4de, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_fe79329825c54cdf8c97a14aa8a30384'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.369131 2885 tablet.cc:1722] T 3eba54e5ddd14c518f4cbd539268ae37 P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:35.369761 992 leader_election.cc:304] T 34ebb441c27747ec860ff2e25f9a0adc P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:35.369992 2780 raft_consensus.cc:2762] T 34ebb441c27747ec860ff2e25f9a0adc P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20250901 01:52:35.370572 992 leader_election.cc:304] T 2973b59bb1df41d18d9a5b6a29ae336f P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:35.370869 2780 raft_consensus.cc:2802] T 2973b59bb1df41d18d9a5b6a29ae336f P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:35.370965 2780 raft_consensus.cc:695] T 2973b59bb1df41d18d9a5b6a29ae336f P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:35.371179 2780 consensus_queue.cc:237] T 2973b59bb1df41d18d9a5b6a29ae336f P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:35.379225 2113 ts_tablet_manager.cc:1918] T d000ba05e12843b4925e622ab55471c0 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:35.379734 2113 log.cc:1199] T d000ba05e12843b4925e622ab55471c0 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/d000ba05e12843b4925e622ab55471c0
I20250901 01:52:35.380491 2885 tablet.cc:1722] T 615b2306fca84003b264da9d5f95ddab P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6
I20250901 01:52:35.381121 2113 ts_tablet_manager.cc:1939] T d000ba05e12843b4925e622ab55471c0 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:35.386608 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 378) took 4158 ms. Trace:
I20250901 01:52:35.386768 2113 rpcz_store.cc:276] 0901 01:52:31.228148 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:31.228195 (+ 47us) service_pool.cc:225] Handling call
0901 01:52:35.346901 (+4118706us) ts_tablet_manager.cc:1145] Deleting tablet d000ba05e12843b4925e622ab55471c0
0901 01:52:35.346914 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.360104 (+ 13190us) spinlock_profiling.cc:243] Waited 12.5 ms on lock 0x616000221b54. stack: 00007f762a8d87d6 00007f762a8d8590 00007f76298cb3ad 0000000000972d94 0000000000972d5d 000000000097272a 00007f763635f600 00007f7637672b13 00007f763a1552b7 00007f763a187344 00007f763a168da0 00007f763a168b2d 0000000000a774dc 00007f762a91eb5f 00007f762a921e2b 00007f762a921ca8
0901 01:52:35.368116 (+ 8012us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:35.379098 (+ 10982us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:35.386586 (+ 7488us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":28801920,"tablet-delete.queue_time_us":4118523}
I20250901 01:52:35.387768 2113 tablet_replica.cc:331] T fcb7b5b7d63d4f0b906fea97cb548de2 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:35.388281 2113 raft_consensus.cc:2241] T fcb7b5b7d63d4f0b906fea97cb548de2 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:35.388870 2113 raft_consensus.cc:2270] T fcb7b5b7d63d4f0b906fea97cb548de2 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
E20250901 01:52:35.390174 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 494b6488aa0d4355b4caa46004267465, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7c21b68ef2e74136a807c1dd90e17f4e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.390326 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 01deaae6abb141789f0cd240f3058c73, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7baa92b46d98454a891222be49a6ca48'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.390415 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 0ee3f35392954817b350e5f48653be78, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_442e11b9211c4b209c68f653d7bfc313'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.390512 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae4844183a3c4a04a775e6fd878391a3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_4bd66ab99a454974aa0f7f4290edfda2'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.390602 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9745582cebfe47b58c21d0191ab4cb9b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77728af9eee44f29a1e2131d5b75b7ca'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.390703 1696 catalog_manager.cc:6205] Error selecting replicas for tablet e2ed32b3292141278af28026528e96c2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b5077f55951d48be92a67407ffe26a08'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.390794 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9f62a9d97f514747b940b488907369f3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d53a456dbca64daf92aeb5fb4517b417'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.390904 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5a7fa68d29644ae8afd2711e39dde59a, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d24e3a18ef4843fdb7b74ce23d47cf9e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.390991 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 415c9d7b552b42c384b444a20491f65c, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_97c44ea002094df8a6af3dccd0099c39'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.391090 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ff2bd017417c41b5b8225b8f00283e63, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0c7355975d6b44dbbe3a77df942b4d43'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.391172 1696 catalog_manager.cc:6205] Error selecting replicas for tablet dcc60bb000f440f1aca9133928b5ce20, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2fbca8463d6546c9b8f5b018fc60251d'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.391263 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 465c02a130c54e09b5a74fb4dd2ef762, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ed6fd284be6f4ae3a008e21e7ce17930'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.391342 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 75883d9fd00e429aa8853903a9fdd7e4, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2e67ad583a0b413e9190cb55137ccf3a'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.391433 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f8f923ddcd2740bcb7e521f4f57c4034, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_e09ac1a9050547038400d4b52299d22f'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.391515 1696 catalog_manager.cc:6205] Error selecting replicas for tablet b1ba0e19dc0b462e80c537bf2d183f5b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_66b111271c15410aab23682f8f6081ac'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.391609 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 8e10a01e77d14cf28c2c5e0205e595b9, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_83cf2d86a1284544ba2e7aa9ebb66bb7'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.391688 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5f0e054f62de4cf6bc6f69ad6664b192, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ef6832d0845044ed9f165ca3541b4c3f'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.391786 1696 catalog_manager.cc:6205] Error selecting replicas for tablet fd2ce82a4ee24d0faa44980a3967e187, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_63d0f2ead9d946d39ce40162ebd769d6'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.391861 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae9a4ad4fdfb4771ba23a91f7007447e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_5e84ae9a9c2a4084a1f9c5ffb4594de2'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.391952 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 086aa948f93644da8e5835b0fac87ddd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b4fbc34da806475c912052d0fab3fc53'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.392030 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 082db24bb3fd4089950504145a20a075, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7f5b278599994063862981b6b041ca8c'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.392120 1696 catalog_manager.cc:6205] Error selecting replicas for tablet fefbf5ae030446b292d8068e95577534, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_3dde393d3a564e84b8ef08c0d25265ce'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.392199 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 93aa22afc0ad4d7fa24807ea3c5d5c21, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0f4b4094d2974b7591875375eda6bed0'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.392287 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9074543dea9949e7a085717b40913348, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_53e30205c5fd488eb3b48c685a159d04'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.392365 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 3a62034e688348ed9e2ac9de009ed3a2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_e225c82a2d6045f6b16915cdb72f5734'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.392607 2113 ts_tablet_manager.cc:1905] T fcb7b5b7d63d4f0b906fea97cb548de2 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:35.399411 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet d000ba05e12843b4925e622ab55471c0 with delete_type TABLET_DATA_DELETED (replaced by tablet eea6bfabe6cc4f83905c24f0f7a5dfc9) from {username='slave'} at 127.0.0.1:35946
E20250901 01:52:35.403530 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 18498c9cb09141d08c1c5c0854b63ebd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77f4810bc83c4d948531da2a45a3792e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.403664 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f5e3316adeae47cd848833dd84b972bb, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_fc962c8bc653498e85cc0bb41639e9ad'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.403753 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 81e8c6567953484ab000fe2eb4398301, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_a920a3ffbabf4d83a6563c3423cde12a'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.403836 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 6ce9521b925b4f97bb4a810ce77e147e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_bd74ae506bbe40489259a7be70c3d10b'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.403923 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2f61edb50d7a42198bfd70639c7bf4de, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_fe79329825c54cdf8c97a14aa8a30384'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.405258 2613 consensus_queue.cc:1035] T 61ddcf86c9f34726939cc116434f0822 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:35.408788 2113 ts_tablet_manager.cc:1918] T fcb7b5b7d63d4f0b906fea97cb548de2 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 01:52:35.411183 2118 ts_tablet_manager.cc:1918] T 88ceba7f48b04d7885cc0164668700a5 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:35.411305 2118 log.cc:1199] T 88ceba7f48b04d7885cc0164668700a5 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/88ceba7f48b04d7885cc0164668700a5
I20250901 01:52:35.410425 1151 tablet_bootstrap.cc:492] T 50472a7aa2f04e0fbd65ac6c86c34b3e P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:35.411458 2113 log.cc:1199] T fcb7b5b7d63d4f0b906fea97cb548de2 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/fcb7b5b7d63d4f0b906fea97cb548de2
I20250901 01:52:35.411604 2118 ts_tablet_manager.cc:1939] T 88ceba7f48b04d7885cc0164668700a5 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:35.411710 1151 ts_tablet_manager.cc:1397] T 50472a7aa2f04e0fbd65ac6c86c34b3e P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.231s user 0.018s sys 0.050s
I20250901 01:52:35.411810 2113 ts_tablet_manager.cc:1939] T fcb7b5b7d63d4f0b906fea97cb548de2 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:35.411939 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 256) took 4836 ms. Trace:
I20250901 01:52:35.412026 2118 rpcz_store.cc:276] 0901 01:52:30.575876 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.575921 (+ 45us) service_pool.cc:225] Handling call
0901 01:52:35.116800 (+4540879us) ts_tablet_manager.cc:1145] Deleting tablet 88ceba7f48b04d7885cc0164668700a5
0901 01:52:35.160961 (+ 44161us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.295733 (+134772us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:35.411073 (+115340us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:35.411919 (+ 846us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4540718}
I20250901 01:52:35.412478 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 88ceba7f48b04d7885cc0164668700a5 (table default.Table_53e30205c5fd488eb3b48c685a159d04 [id=60837b865ee7415ea42a85a953d837a5]) successfully deleted
I20250901 01:52:35.413560 2613 consensus_queue.cc:1035] T b4278a013ae84082bc7a8497b04ac147 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:35.413611 2856 consensus_queue.cc:1035] T c6f7c525f59b49938b697de98d4b5835 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:35.414057 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c36db80204334011894093ab9e2b0f3f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.414564 1151 raft_consensus.cc:357] T 50472a7aa2f04e0fbd65ac6c86c34b3e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:35.414870 1151 raft_consensus.cc:383] T 50472a7aa2f04e0fbd65ac6c86c34b3e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:35.414978 1151 raft_consensus.cc:738] T 50472a7aa2f04e0fbd65ac6c86c34b3e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:35.414553 2909 tablet.cc:1722] T 89d40f0331114490b51de3ba6d773571 P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250901 01:52:35.415263 1151 consensus_queue.cc:260] T 50472a7aa2f04e0fbd65ac6c86c34b3e P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:35.415894 1151 ts_tablet_manager.cc:1428] T 50472a7aa2f04e0fbd65ac6c86c34b3e P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250901 01:52:35.416244 1489 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_7f5b278599994063862981b6b041ca8c" } new_table_name: "default.Table_4473168010a84ac7a014b99dd10e35a9" modify_external_catalogs: true
I20250901 01:52:35.416606 1151 tablet_bootstrap.cc:492] T e9f1f892658b4736859ff507536920a1 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:35.422238 1483 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 0e4009963cfe42d6ba164fc828897f09 (table default.Table_1adc70cccd2f4cb3a79df96861cb4181 [id=24fffc7ebf5a47bdb37211910301acf7]). Expected version 1 got 0
I20250901 01:52:35.423476 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_88153f026c9a4dbbb0dd8eb55d53739b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.423606 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_129d555c750c4a6486b330b73bfc06f4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.423877 1490 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_657603f2bb3c43bc85028218860da978" } modify_external_catalogs: true
I20250901 01:52:35.424021 1490 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_657603f2bb3c43bc85028218860da978" } modify_external_catalogs: true
I20250901 01:52:35.423476 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2b0e8f86e9314d7483b762c07e73a117"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.424990 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2c02c7c57b1f4425b44eced396281773"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:35.425938 1489 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 082db24bb3fd4089950504145a20a075 has no consensus state
I20250901 01:52:35.427079 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4ac21d1679a24ab4beee4b07a916265d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.427197 2780 raft_consensus.cc:491] T 2dacb608897d47d28512df87dc320927 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:35.427315 2780 raft_consensus.cc:513] T 2dacb608897d47d28512df87dc320927 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:35.427897 2118 tablet_replica.cc:331] T d4c9aa723aee4dc4b6237cb5c3722b52 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:35.427886 2780 leader_election.cc:290] T 2dacb608897d47d28512df87dc320927 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:35.428122 2118 raft_consensus.cc:2241] T d4c9aa723aee4dc4b6237cb5c3722b52 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:35.428304 2118 raft_consensus.cc:2270] T d4c9aa723aee4dc4b6237cb5c3722b52 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:35.428906 2118 ts_tablet_manager.cc:1905] T d4c9aa723aee4dc4b6237cb5c3722b52 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:35.431458 1490 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 6789505328b1409ea288e8b4eaf94935
W20250901 01:52:35.431617 1490 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 6789505328b1409ea288e8b4eaf94935: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:35.432512 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 6789505328b1409ea288e8b4eaf94935 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:35 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:35.432678 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3484d4cdf95a4845bea427be7bbe4475"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.433120 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 6789505328b1409ea288e8b4eaf94935 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:35 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:35.434060 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1fd7016e9ce542b0a3192cfc0665fa4e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.434391 1483 master_service.cc:948] ReplaceTablet: received request to replace tablet 3239e58531d445fa80165cf678bc277c from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:35.433569 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7609ab701e36441f946d7947cac47de8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.433557 1492 catalog_manager.cc:5582] T 2973b59bb1df41d18d9a5b6a29ae336f P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } }
I20250901 01:52:35.435815 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 2973b59bb1df41d18d9a5b6a29ae336f (table default.Table_3c2306df46ec4c14b64b5fbd9d21eb98 [id=746845bd84c6444b8a1e5683eaba4a68]). Expected version 2 got 0
I20250901 01:52:35.437002 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_41331ace606e4043975c3916d7e26862"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.438201 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1d731a909a084bb287ef90dd681558a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.439196 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8762f54e60a84246ba9a6f58d1e63175"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.440663 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_07d9d03b2ff64682be807572798654a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.440917 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_844afea8f190450a95e60151c375045c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.441735 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e461b0abfdce4eee980e29dbc7131916"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.442854 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d17128c619d149daaeaa3dd2b953d8c4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.446302 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 377) took 4218 ms. Trace:
I20250901 01:52:35.446396 2113 rpcz_store.cc:276] 0901 01:52:31.228033 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:31.228429 (+ 396us) service_pool.cc:225] Handling call
0901 01:52:35.387723 (+4159294us) ts_tablet_manager.cc:1145] Deleting tablet fcb7b5b7d63d4f0b906fea97cb548de2
0901 01:52:35.387732 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.398918 (+ 11186us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:35.408683 (+ 9765us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:35.446278 (+ 37595us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1383,"tablet-delete.queue_time_us":4159167}
I20250901 01:52:35.446836 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet fcb7b5b7d63d4f0b906fea97cb548de2 (table default.Table_2ceaff93c8f44f00aab53e48d8466dfd [id=4d9ea39016ca4bf4859f4f80ea4e9ed6]) successfully deleted
I20250901 01:52:35.447211 1151 tablet_bootstrap.cc:654] T e9f1f892658b4736859ff507536920a1 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:35.447765 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a9b8f9266c234c9fa4f5168336036d93"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.449043 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5d8e5e48b45148f199be1f5b38326d39"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.449458 1492 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: default.Table_4e6fae563c5843f4953527abbe862adc [id=cecdd8d6ca6d445c9fd82decd2a95366] alter complete (version 6)
I20250901 01:52:35.449810 2113 tablet_replica.cc:331] T becbaae4fd7e44faa679e9479465a7c4 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:35.449998 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ae85f4cdde08414bae442a6691391238"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.451148 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_daf06bfe8eb84e82a320c55f5727a568"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.452687 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_6dfab02329d848848a79cc1b9d3d721e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.453572 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_17f75317b93a4a3bad2b9c067b9de812"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.454138 1489 master_service.cc:948] ReplaceTablet: received request to replace tablet 5d794471dbed4477b9d52fd411fa4d10 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:35.458278 1489 master_service.cc:948] ReplaceTablet: received request to replace tablet ed1b41340ce742a88c3e4fd85cd25ca6 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:35.458755 2113 raft_consensus.cc:2241] T becbaae4fd7e44faa679e9479465a7c4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:35.459288 2113 raft_consensus.cc:2270] T becbaae4fd7e44faa679e9479465a7c4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:35.459415 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_382d6ddecbdc497cb0124a19216105e4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:35.460304 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 494b6488aa0d4355b4caa46004267465, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7c21b68ef2e74136a807c1dd90e17f4e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.460546 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 01deaae6abb141789f0cd240f3058c73, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7baa92b46d98454a891222be49a6ca48'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.461362 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 0ee3f35392954817b350e5f48653be78, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_442e11b9211c4b209c68f653d7bfc313'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.461578 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae4844183a3c4a04a775e6fd878391a3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_4bd66ab99a454974aa0f7f4290edfda2'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.462112 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_cfbecd291ea94076b92b5613fc2278a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.462771 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_dc9f22a3f62446e8a551aa47b1c3a4eb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.463069 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_730ed3248328426ba2117e6c47bf1bc4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.463467 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c9f645685b8142c9b77d1a110567eb4a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:35.465018 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9745582cebfe47b58c21d0191ab4cb9b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77728af9eee44f29a1e2131d5b75b7ca'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.465888 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_400df66f85084a2a8f5e91b193990b99"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.466235 1489 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet ed1b41340ce742a88c3e4fd85cd25ca6
E20250901 01:52:35.466598 1696 catalog_manager.cc:6205] Error selecting replicas for tablet e2ed32b3292141278af28026528e96c2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b5077f55951d48be92a67407ffe26a08'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.466794 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9f62a9d97f514747b940b488907369f3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d53a456dbca64daf92aeb5fb4517b417'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.466883 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a46b0554f9d8458fafe6278e58a53e07"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:35.466460 1489 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet ed1b41340ce742a88c3e4fd85cd25ca6: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:35.467337 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3fd5919ec01149378f6da6f2586e1bd8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.470427 1489 catalog_manager.cc:6652] ReplaceTablet: tablet ed1b41340ce742a88c3e4fd85cd25ca6 deleted and replaced by tablet 2821bf7750ba4e38b25ccba2e2bc0eab
I20250901 01:52:35.467808 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5757868942fc4da1bcfaf25def405958"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.468099 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e10ffaf8501b4103af88bf0cf567fe92"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:35.471227 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5a7fa68d29644ae8afd2711e39dde59a, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d24e3a18ef4843fdb7b74ce23d47cf9e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.471352 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 415c9d7b552b42c384b444a20491f65c, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_97c44ea002094df8a6af3dccd0099c39'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.471487 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ff2bd017417c41b5b8225b8f00283e63, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0c7355975d6b44dbbe3a77df942b4d43'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.471607 1696 catalog_manager.cc:6205] Error selecting replicas for tablet dcc60bb000f440f1aca9133928b5ce20, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2fbca8463d6546c9b8f5b018fc60251d'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.471745 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 465c02a130c54e09b5a74fb4dd2ef762, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ed6fd284be6f4ae3a008e21e7ce17930'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.471868 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 75883d9fd00e429aa8853903a9fdd7e4, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2e67ad583a0b413e9190cb55137ccf3a'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.471997 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f8f923ddcd2740bcb7e521f4f57c4034, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_e09ac1a9050547038400d4b52299d22f'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.472107 1696 catalog_manager.cc:6205] Error selecting replicas for tablet b1ba0e19dc0b462e80c537bf2d183f5b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_66b111271c15410aab23682f8f6081ac'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.472234 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 8e10a01e77d14cf28c2c5e0205e595b9, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_83cf2d86a1284544ba2e7aa9ebb66bb7'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.472352 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5f0e054f62de4cf6bc6f69ad6664b192, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ef6832d0845044ed9f165ca3541b4c3f'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.472908 1696 catalog_manager.cc:6205] Error selecting replicas for tablet fd2ce82a4ee24d0faa44980a3967e187, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_63d0f2ead9d946d39ce40162ebd769d6'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.473083 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae9a4ad4fdfb4771ba23a91f7007447e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_5e84ae9a9c2a4084a1f9c5ffb4594de2'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.474284 1151 tablet_bootstrap.cc:492] T e9f1f892658b4736859ff507536920a1 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:35.474427 1151 ts_tablet_manager.cc:1397] T e9f1f892658b4736859ff507536920a1 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.058s user 0.002s sys 0.002s
I20250901 01:52:35.475134 1151 raft_consensus.cc:357] T e9f1f892658b4736859ff507536920a1 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:35.475291 1151 raft_consensus.cc:383] T e9f1f892658b4736859ff507536920a1 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:35.475342 1151 raft_consensus.cc:738] T e9f1f892658b4736859ff507536920a1 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:35.475440 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet ed1b41340ce742a88c3e4fd85cd25ca6 with delete_type TABLET_DATA_DELETED (replaced by tablet 2821bf7750ba4e38b25ccba2e2bc0eab) from {username='slave'} at 127.0.0.1:35946
E20250901 01:52:35.475767 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 086aa948f93644da8e5835b0fac87ddd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b4fbc34da806475c912052d0fab3fc53'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.475812 893 tablet_service.cc:1515] Processing DeleteTablet for tablet ed1b41340ce742a88c3e4fd85cd25ca6 with delete_type TABLET_DATA_DELETED (replaced by tablet 2821bf7750ba4e38b25ccba2e2bc0eab) from {username='slave'} at 127.0.0.1:34472
E20250901 01:52:35.475981 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 082db24bb3fd4089950504145a20a075, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_4473168010a84ac7a014b99dd10e35a9'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.476151 1696 catalog_manager.cc:6205] Error selecting replicas for tablet fefbf5ae030446b292d8068e95577534, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_3dde393d3a564e84b8ef08c0d25265ce'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.476306 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 93aa22afc0ad4d7fa24807ea3c5d5c21, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0f4b4094d2974b7591875375eda6bed0'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.476236 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8ae6efbf1511484fabd0e07aaefb06e3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.477097 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_6ae9080586f74b3bb108d7bee99548ec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:35.477453 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9074543dea9949e7a085717b40913348, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_53e30205c5fd488eb3b48c685a159d04'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.477573 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 3a62034e688348ed9e2ac9de009ed3a2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_e225c82a2d6045f6b16915cdb72f5734'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.477694 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 18498c9cb09141d08c1c5c0854b63ebd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77f4810bc83c4d948531da2a45a3792e'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.467168 2113 ts_tablet_manager.cc:1905] T becbaae4fd7e44faa679e9479465a7c4 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:35.478789 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ab1ae533e8264b50bdb0b3301b122b9d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.475641 1151 consensus_queue.cc:260] T e9f1f892658b4736859ff507536920a1 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:35.479564 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c2433542825e4689a9f16aed7bd7ae1e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.480059 1151 ts_tablet_manager.cc:1428] T e9f1f892658b4736859ff507536920a1 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250901 01:52:35.480341 1151 tablet_bootstrap.cc:492] T 45f6e8ba99b74898a2922b36768c6c36 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:35.480556 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_038439638af245d985630fa5b2937e78"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.481449 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7e0fb4d49bf34e458a3bca592ff58df8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:35.481921 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f5e3316adeae47cd848833dd84b972bb, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_fc962c8bc653498e85cc0bb41639e9ad'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.482081 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 81e8c6567953484ab000fe2eb4398301, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_a920a3ffbabf4d83a6563c3423cde12a'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.482184 1151 tablet_bootstrap.cc:654] T 45f6e8ba99b74898a2922b36768c6c36 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
E20250901 01:52:35.482200 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 6ce9521b925b4f97bb4a810ce77e147e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_bd74ae506bbe40489259a7be70c3d10b'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.482313 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2f61edb50d7a42198bfd70639c7bf4de, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_fe79329825c54cdf8c97a14aa8a30384'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.482896 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d4820310457e4c99a11998b5b225b1cd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.483500 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_33408fbbf33d4f28a037d136af61db0d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.483726 1490 master_service.cc:948] ReplaceTablet: received request to replace tablet c6f7c525f59b49938b697de98d4b5835 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:35.483919 1484 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: default.Table_9fb44a0a69994184bc2ffe62c55d2c25 [id=8ee0aa1ba24042dd8ded004644913b9c] alter complete (version 2)
I20250901 01:52:35.484208 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_db75ba8ad75f4f5a932ae8b8fe8266c7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.483549 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a32b12c4be064075bb6cd1f3764c3803"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.486104 2118 ts_tablet_manager.cc:1918] T d4c9aa723aee4dc4b6237cb5c3722b52 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:35.486213 2118 log.cc:1199] T d4c9aa723aee4dc4b6237cb5c3722b52 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/d4c9aa723aee4dc4b6237cb5c3722b52
I20250901 01:52:35.486547 2118 ts_tablet_manager.cc:1939] T d4c9aa723aee4dc4b6237cb5c3722b52 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:35.487006 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 257) took 4910 ms. Trace:
I20250901 01:52:35.487080 2118 rpcz_store.cc:276] 0901 01:52:30.576203 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.576242 (+ 39us) service_pool.cc:225] Handling call
0901 01:52:35.427841 (+4851599us) ts_tablet_manager.cc:1145] Deleting tablet d4c9aa723aee4dc4b6237cb5c3722b52
0901 01:52:35.427853 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.432007 (+ 4154us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:35.485989 (+ 53982us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:35.486988 (+ 999us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4851464}
I20250901 01:52:35.488833 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9d6c4f54ad59422a8cec1aecd84f132d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.489555 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_83e96b23dcb04ae0a52614347d5124ee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.488883 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_fd0e356d694047d087c0f5ba3c491077"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.490664 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_58c13cbb272f4f90b160da3ef5ae08e8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.491672 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_0bb1d7719b9e4b8fbcf582de1f20e51f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.491817 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_773a80bcdf1741b494ca7bcd532966c7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.492663 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ff5ec41e02ca41c9906913e63ca3d1cc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.493579 2118 tablet_replica.cc:331] T eee34044c39d4013baf78d3705489626 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:35.493959 2118 raft_consensus.cc:2241] T eee34044c39d4013baf78d3705489626 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:35.494217 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet d4c9aa723aee4dc4b6237cb5c3722b52 (table default.Table_e09ac1a9050547038400d4b52299d22f [id=b9d81ddd38fe47838877b010be2b40d0]) successfully deleted
I20250901 01:52:35.496142 1490 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet c6f7c525f59b49938b697de98d4b5835
W20250901 01:52:35.496340 1490 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet c6f7c525f59b49938b697de98d4b5835: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:35.496789 1490 catalog_manager.cc:6652] ReplaceTablet: tablet c6f7c525f59b49938b697de98d4b5835 deleted and replaced by tablet 77911bcb98cf4d8890118eca9136f86e
I20250901 01:52:35.497424 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet c6f7c525f59b49938b697de98d4b5835 with delete_type TABLET_DATA_DELETED (replaced by tablet 77911bcb98cf4d8890118eca9136f86e) from {username='slave'} at 127.0.0.1:35946
E20250901 01:52:35.498849 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 494b6488aa0d4355b4caa46004267465, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7c21b68ef2e74136a807c1dd90e17f4e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.498996 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 01deaae6abb141789f0cd240f3058c73, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7baa92b46d98454a891222be49a6ca48'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.499087 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 0ee3f35392954817b350e5f48653be78, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_442e11b9211c4b209c68f653d7bfc313'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.499199 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae4844183a3c4a04a775e6fd878391a3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_4bd66ab99a454974aa0f7f4290edfda2'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.499282 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9745582cebfe47b58c21d0191ab4cb9b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77728af9eee44f29a1e2131d5b75b7ca'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.499377 1696 catalog_manager.cc:6205] Error selecting replicas for tablet e2ed32b3292141278af28026528e96c2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b5077f55951d48be92a67407ffe26a08'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.499446 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9f62a9d97f514747b940b488907369f3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d53a456dbca64daf92aeb5fb4517b417'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.499534 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5a7fa68d29644ae8afd2711e39dde59a, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d24e3a18ef4843fdb7b74ce23d47cf9e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.499608 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 415c9d7b552b42c384b444a20491f65c, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_97c44ea002094df8a6af3dccd0099c39'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.499708 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ff2bd017417c41b5b8225b8f00283e63, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0c7355975d6b44dbbe3a77df942b4d43'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.499802 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2821bf7750ba4e38b25ccba2e2bc0eab, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ce2f3b515c8f4c1bb557c5ed84b48dba'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.499902 1696 catalog_manager.cc:6205] Error selecting replicas for tablet dcc60bb000f440f1aca9133928b5ce20, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2fbca8463d6546c9b8f5b018fc60251d'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.499984 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 465c02a130c54e09b5a74fb4dd2ef762, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ed6fd284be6f4ae3a008e21e7ce17930'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.500072 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 75883d9fd00e429aa8853903a9fdd7e4, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2e67ad583a0b413e9190cb55137ccf3a'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.500140 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f8f923ddcd2740bcb7e521f4f57c4034, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_e09ac1a9050547038400d4b52299d22f'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.500221 1696 catalog_manager.cc:6205] Error selecting replicas for tablet b1ba0e19dc0b462e80c537bf2d183f5b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_66b111271c15410aab23682f8f6081ac'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.500291 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 8e10a01e77d14cf28c2c5e0205e595b9, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_83cf2d86a1284544ba2e7aa9ebb66bb7'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.500670 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ef22ccf2c5254958ab2e7a87b302bff4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:35.501067 992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:40561, user_credentials={real_user=slave}} blocked reactor thread for 87441.8us
I20250901 01:52:35.497454 893 tablet_service.cc:1515] Processing DeleteTablet for tablet c6f7c525f59b49938b697de98d4b5835 with delete_type TABLET_DATA_DELETED (replaced by tablet 77911bcb98cf4d8890118eca9136f86e) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:35.501654 2118 raft_consensus.cc:2270] T eee34044c39d4013baf78d3705489626 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:35.502282 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f4b8756426004265b2c2db22f1bccc36"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.502574 921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ba8241644984394ada95b561e76467e" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:35.502744 921 raft_consensus.cc:2466] T 6ba8241644984394ada95b561e76467e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:35.503141 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_297cd88c495f48cfa7e5c318cbfd7f9d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:35.503691 992 consensus_peers.cc:489] T c6f7c525f59b49938b697de98d4b5835 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250901 01:52:35.503862 992 leader_election.cc:336] T 2973b59bb1df41d18d9a5b6a29ae336f P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:35.504247 992 consensus_peers.cc:489] T b4278a013ae84082bc7a8497b04ac147 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:35.505618 2118 ts_tablet_manager.cc:1905] T eee34044c39d4013baf78d3705489626 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:35.506263 992 leader_election.cc:336] T 34ebb441c27747ec860ff2e25f9a0adc P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:35.508478 859 leader_election.cc:304] T 34ebb441c27747ec860ff2e25f9a0adc P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:35.508782 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_58ae86a313dd4b40a280bd3feb8503c5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.508893 2753 raft_consensus.cc:2802] T 34ebb441c27747ec860ff2e25f9a0adc P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:35.508996 2753 raft_consensus.cc:695] T 34ebb441c27747ec860ff2e25f9a0adc P 00c282f779c1477d920f24dc0dc16628 [term 1 LEADER]: Becoming Leader. State: Replica: 00c282f779c1477d920f24dc0dc16628, State: Running, Role: LEADER
I20250901 01:52:35.509238 2753 consensus_queue.cc:237] T 34ebb441c27747ec860ff2e25f9a0adc P 00c282f779c1477d920f24dc0dc16628 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:35.509670 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9b21081ecf53483baedce6d659ea6999"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.510872 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e4317c778410437ea92b77f863e29c26"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.512262 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5b944244dd52406aa364ab935425d5b2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.513079 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ffa936457f314704bba08d59457be14f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.513880 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1d5a8bf1889440c1b345471bb5b1f3d1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.515009 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a45e826ebc3549fe98d4dee65f990c79"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.515873 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ff794a8a53d84751ba3d92b686ba32b4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.516554 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_fa7eea0648f441118752c0db5ef65771"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.517328 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e09b1b69acc24a97bf492e44647c101e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.518182 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_12712610a407437a9b52bdfc8e3a1ff1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.519567 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7901a4aee287478090e4cfc7e13be585"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.520190 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8162b510d6b4475e83b443aa3559b57c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.521026 2113 ts_tablet_manager.cc:1918] T becbaae4fd7e44faa679e9479465a7c4 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:35.521143 2113 log.cc:1199] T becbaae4fd7e44faa679e9479465a7c4 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/becbaae4fd7e44faa679e9479465a7c4
I20250901 01:52:35.522396 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5da5a7ae410a41f5b7422b5dcf390807"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.523553 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4345682637654bb5b35935240ee19e50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.524111 2911 raft_consensus.cc:491] T 50472a7aa2f04e0fbd65ac6c86c34b3e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:35.524243 2911 raft_consensus.cc:513] T 50472a7aa2f04e0fbd65ac6c86c34b3e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:35.524734 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2b47caf0c92948818c69be1bd7c999fb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.525502 2911 leader_election.cc:290] T 50472a7aa2f04e0fbd65ac6c86c34b3e P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:35.525743 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_97b23789cd82492c85b4908ae827b16b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.526448 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_071d2bb705ca40d2b81abb010ed523bd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.529470 921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ae21b9ae2e3b472eb67fb5f96388a50f" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:35.529639 921 raft_consensus.cc:2466] T ae21b9ae2e3b472eb67fb5f96388a50f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:35.530845 2118 ts_tablet_manager.cc:1918] T eee34044c39d4013baf78d3705489626 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:35.531230 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c440e082d680451d89df8077796089f5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.532249 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2181383ce7d247ac9e4e1cd6f35a2eb3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.533206 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_14066e36c8de4f65a8aa10e79973523d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.534049 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_69b70924c5e34131b79e20ce2ee2d895"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.534929 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_04e224919b8a49988a47355049161044"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.535766 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8cc6abedaea64f27923ba45b06caaf72"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.537871 1151 tablet_bootstrap.cc:492] T 45f6e8ba99b74898a2922b36768c6c36 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:35.538064 1151 ts_tablet_manager.cc:1397] T 45f6e8ba99b74898a2922b36768c6c36 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.058s user 0.003s sys 0.007s
I20250901 01:52:35.538987 2113 ts_tablet_manager.cc:1939] T becbaae4fd7e44faa679e9479465a7c4 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:35.538945 1151 raft_consensus.cc:357] T 45f6e8ba99b74898a2922b36768c6c36 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:35.539746 1151 raft_consensus.cc:383] T 45f6e8ba99b74898a2922b36768c6c36 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:35.539955 1151 raft_consensus.cc:738] T 45f6e8ba99b74898a2922b36768c6c36 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:35.540238 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 383) took 4272 ms. Trace:
I20250901 01:52:35.540287 1151 consensus_queue.cc:260] T 45f6e8ba99b74898a2922b36768c6c36 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:35.542347 1151 ts_tablet_manager.cc:1428] T 45f6e8ba99b74898a2922b36768c6c36 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250901 01:52:35.544600 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_0c5dd1a03491412f82c58e79de593125"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.545749 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_80dadc0dc55c4ca095a78eac4b04538d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.546587 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_221f5113d0c44300a7826d009eab2138"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.547657 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1ae255e4e67849cc9932250835fdc997"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.547698 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2a7d3a494f3e42dbaa3dfec594a1de37"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.547758 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a49628cf034c4b4ab25aeb07ab7c0940"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.548126 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9d133c2e8c6d491eb5048d8faec69c80"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.548424 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f9e15ca9b0ab48c094f803645d00eb90"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.548842 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8d1914d8f2c34f418dd572e85198c104"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.549191 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_df04aa0b1a714a04b36d2587d982eb6d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:35.500380 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5f0e054f62de4cf6bc6f69ad6664b192, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ef6832d0845044ed9f165ca3541b4c3f'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.549902 1696 catalog_manager.cc:6205] Error selecting replicas for tablet fd2ce82a4ee24d0faa44980a3967e187, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_63d0f2ead9d946d39ce40162ebd769d6'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.550021 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae9a4ad4fdfb4771ba23a91f7007447e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_5e84ae9a9c2a4084a1f9c5ffb4594de2'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.550104 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 086aa948f93644da8e5835b0fac87ddd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b4fbc34da806475c912052d0fab3fc53'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.550206 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 082db24bb3fd4089950504145a20a075, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_4473168010a84ac7a014b99dd10e35a9'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.550287 1696 catalog_manager.cc:6205] Error selecting replicas for tablet fefbf5ae030446b292d8068e95577534, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_3dde393d3a564e84b8ef08c0d25265ce'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.550376 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 93aa22afc0ad4d7fa24807ea3c5d5c21, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0f4b4094d2974b7591875375eda6bed0'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.550450 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9074543dea9949e7a085717b40913348, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_53e30205c5fd488eb3b48c685a159d04'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.550549 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 3a62034e688348ed9e2ac9de009ed3a2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_e225c82a2d6045f6b16915cdb72f5734'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.550626 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 18498c9cb09141d08c1c5c0854b63ebd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77f4810bc83c4d948531da2a45a3792e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.550714 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f5e3316adeae47cd848833dd84b972bb, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_fc962c8bc653498e85cc0bb41639e9ad'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.550786 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 81e8c6567953484ab000fe2eb4398301, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_a920a3ffbabf4d83a6563c3423cde12a'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.550876 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 6ce9521b925b4f97bb4a810ce77e147e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_bd74ae506bbe40489259a7be70c3d10b'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.550949 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2f61edb50d7a42198bfd70639c7bf4de, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_fe79329825c54cdf8c97a14aa8a30384'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.552819 2118 log.cc:1199] T eee34044c39d4013baf78d3705489626 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/eee34044c39d4013baf78d3705489626
I20250901 01:52:35.554451 2118 ts_tablet_manager.cc:1939] T eee34044c39d4013baf78d3705489626 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:35.554818 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3ce26c23abc0474aa2e48d6bd54b3386"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.551896 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_665e8db2c72e43d0befacffd063a312b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.552016 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a7744478ea94422f9f17f6af45964e11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.552181 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_bb7cd7246975422889b0b518a16a6a03"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.553587 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_08cf4854688046619998cddaa90b5f68"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.554061 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_6b30e6145b1141d296a9812933f69417"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.554282 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9cbdc99e9b31448f97a510b18b718447"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.557420 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 258) took 4978 ms. Trace:
I20250901 01:52:35.557504 2118 rpcz_store.cc:276] 0901 01:52:30.578756 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.578801 (+ 45us) service_pool.cc:225] Handling call
0901 01:52:35.493514 (+4914713us) ts_tablet_manager.cc:1145] Deleting tablet eee34044c39d4013baf78d3705489626
0901 01:52:35.493525 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.527752 (+ 34227us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:35.530760 (+ 3008us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:35.557398 (+ 26638us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4914565}
I20250901 01:52:35.557952 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet eee34044c39d4013baf78d3705489626 (table default.Table_99afdac28a4e4ac9ae93b4fbbe704a4d [id=4639e91c14d8427d86c89d3c2b0918ad]) successfully deleted
I20250901 01:52:35.558589 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ba9d1c91ec61420ebb2b94965bcc11a9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.559389 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d8b7faff37784114a33053dbd7f4916e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.560225 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_17f085fec0474914be3dc38cd5dc79a0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.561077 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_dce51c200d5b42e9807643afb68bbb3c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.561167 1151 tablet_bootstrap.cc:492] T 3dae00aa38204ee3ac77325eb334be42 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:35.561887 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_fcb8013d61f14bf7abbf12ab973fb173"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.562881 1151 tablet_bootstrap.cc:654] T 3dae00aa38204ee3ac77325eb334be42 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:35.562880 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_79c8770f4c6f4b7bb3452515715636bf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.540316 2113 rpcz_store.cc:276] 0901 01:52:31.267826 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:31.267899 (+ 73us) service_pool.cc:225] Handling call
0901 01:52:35.449751 (+4181852us) ts_tablet_manager.cc:1145] Deleting tablet becbaae4fd7e44faa679e9479465a7c4
0901 01:52:35.449763 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.511941 (+ 62178us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:35.520892 (+ 8951us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:35.540222 (+ 19330us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4181551}
I20250901 01:52:35.563607 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet becbaae4fd7e44faa679e9479465a7c4 (table default.Table_0e73f80424cd48408f18c2eff1411706 [id=47c13af47c944d0fa79dee115033265d]) successfully deleted
I20250901 01:52:35.563958 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 387) took 4295 ms. Trace:
I20250901 01:52:35.564038 2113 rpcz_store.cc:276] 0901 01:52:31.268224 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:31.268344 (+ 120us) service_pool.cc:225] Handling call
0901 01:52:35.563861 (+4295517us) ts_tablet_manager.cc:1145] Deleting tablet bb0cf5a6a27045a08df6845d7473184d
0901 01:52:35.563871 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.563941 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4295274}
W20250901 01:52:35.565101 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet bb0cf5a6a27045a08df6845d7473184d because the tablet was not found. No further retry: Not found: Tablet not found: bb0cf5a6a27045a08df6845d7473184d
I20250901 01:52:35.565090 1482 catalog_manager.cc:5582] T 34ebb441c27747ec860ff2e25f9a0adc P 00c282f779c1477d920f24dc0dc16628 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00c282f779c1477d920f24dc0dc16628 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "00c282f779c1477d920f24dc0dc16628" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: HEALTHY } } }
I20250901 01:52:35.567094 1490 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet e9f1f892658b4736859ff507536920a1 (table default.Table_fa623571fcac4341935b9cb46ef4818c [id=9aa62cb634314a0a849612c54c1220d9]). Expected version 2 got 0
I20250901 01:52:35.569010 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 259) took 4989 ms. Trace:
I20250901 01:52:35.569799 2118 rpcz_store.cc:276] 0901 01:52:30.579083 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.579120 (+ 37us) service_pool.cc:225] Handling call
0901 01:52:35.568904 (+4989784us) ts_tablet_manager.cc:1145] Deleting tablet fa7fb88ded21432a8757f37ec3db5e71
0901 01:52:35.568915 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.568990 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4989648}
W20250901 01:52:35.570863 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet fa7fb88ded21432a8757f37ec3db5e71 because the tablet was not found. No further retry: Not found: Tablet not found: fa7fb88ded21432a8757f37ec3db5e71
I20250901 01:52:35.571229 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 260) took 4988 ms. Trace:
I20250901 01:52:35.571549 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e969c00770404996bc1844d96f5385d9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.572966 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 379) took 4305 ms. Trace:
I20250901 01:52:35.573060 2113 rpcz_store.cc:276] 0901 01:52:31.267388 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:31.268624 (+ 1236us) service_pool.cc:225] Handling call
0901 01:52:35.572872 (+4304248us) ts_tablet_manager.cc:1145] Deleting tablet 154574e17f214cc0ad56e6e6be3af899
0901 01:52:35.572882 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.572948 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4304078}
W20250901 01:52:35.573487 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 154574e17f214cc0ad56e6e6be3af899 because the tablet was not found. No further retry: Not found: Tablet not found: 154574e17f214cc0ad56e6e6be3af899
I20250901 01:52:35.573824 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_be7faf8f5fc54cde944fc7dbab8cafa2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.575690 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 380) took 4308 ms. Trace:
I20250901 01:52:35.575796 2113 rpcz_store.cc:276] 0901 01:52:31.267503 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:31.268830 (+ 1327us) service_pool.cc:225] Handling call
0901 01:52:35.575584 (+4306754us) ts_tablet_manager.cc:1145] Deleting tablet 9d886fa63f8f4996b087997b542e8bb1
0901 01:52:35.575597 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.575670 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4306636}
I20250901 01:52:35.576247 1482 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 45f6e8ba99b74898a2922b36768c6c36 (table default.Table_d1df9c0ae1a04b21a133e93cf4ae363a [id=9ce26dcf307149b595a43b6ac4562fba]). Expected version 2 got 0
W20250901 01:52:35.576301 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 9d886fa63f8f4996b087997b542e8bb1 because the tablet was not found. No further retry: Not found: Tablet not found: 9d886fa63f8f4996b087997b542e8bb1
I20250901 01:52:35.576625 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 381) took 4309 ms. Trace:
I20250901 01:52:35.576720 2113 rpcz_store.cc:276] 0901 01:52:31.267617 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:31.268969 (+ 1352us) service_pool.cc:225] Handling call
0901 01:52:35.576519 (+4307550us) ts_tablet_manager.cc:1145] Deleting tablet 23af154a6f1c4e2e9352fb17c9024f10
0901 01:52:35.576531 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.576606 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4307410}
W20250901 01:52:35.577165 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 23af154a6f1c4e2e9352fb17c9024f10 because the tablet was not found. No further retry: Not found: Tablet not found: 23af154a6f1c4e2e9352fb17c9024f10
I20250901 01:52:35.577351 2118 rpcz_store.cc:276] 0901 01:52:30.582287 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.582329 (+ 42us) service_pool.cc:225] Handling call
0901 01:52:35.571144 (+4988815us) ts_tablet_manager.cc:1145] Deleting tablet da5f69c5b0ca4257ad2ff7918ee2af2e
0901 01:52:35.571153 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.571214 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4988664}
W20250901 01:52:35.577762 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet da5f69c5b0ca4257ad2ff7918ee2af2e because the tablet was not found. No further retry: Not found: Tablet not found: da5f69c5b0ca4257ad2ff7918ee2af2e
I20250901 01:52:35.578123 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7659e8c7f6cb42d7af93c507c6aa9483"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.578627 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 384) took 4310 ms. Trace:
I20250901 01:52:35.578728 2113 rpcz_store.cc:276] 0901 01:52:31.267964 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:31.269424 (+ 1460us) service_pool.cc:225] Handling call
0901 01:52:35.578528 (+4309104us) ts_tablet_manager.cc:1145] Deleting tablet 7a2f6e2c7ee2402ea310c058e046416c
0901 01:52:35.578540 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.578610 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4308946}
I20250901 01:52:35.578512 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ca51e205013144c8bb43f835ebbf782f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.579401 2113 tablet_replica.cc:331] T 26c8ed083cea4dc0acca80d00b11b29b P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:35.579676 2113 raft_consensus.cc:2241] T 26c8ed083cea4dc0acca80d00b11b29b P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:35.579706 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8abb9216c409463881803f877a2c31a9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.579877 2113 pending_rounds.cc:70] T 26c8ed083cea4dc0acca80d00b11b29b P fd27ab6665304aaea53adbd102dd4f58: Trying to abort 1 pending ops.
I20250901 01:52:35.579921 2113 pending_rounds.cc:77] T 26c8ed083cea4dc0acca80d00b11b29b P fd27ab6665304aaea53adbd102dd4f58: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7195408602925350912 op_type: NO_OP noop_request { }
I20250901 01:52:35.580040 2113 raft_consensus.cc:2887] T 26c8ed083cea4dc0acca80d00b11b29b P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250901 01:52:35.580142 2113 raft_consensus.cc:2270] T 26c8ed083cea4dc0acca80d00b11b29b P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:35.580753 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 261) took 4996 ms. Trace:
I20250901 01:52:35.580850 2113 ts_tablet_manager.cc:1905] T 26c8ed083cea4dc0acca80d00b11b29b P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:35.581033 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_31b28a8d8d44497baab7c62c710ecb3f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.581766 2118 rpcz_store.cc:276] 0901 01:52:30.584440 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.584539 (+ 99us) service_pool.cc:225] Handling call
0901 01:52:35.578828 (+4994289us) ts_tablet_manager.cc:1145] Deleting tablet 36e10328c34242ebb4dbd8f294162696
0901 01:52:35.578837 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.580716 (+ 1879us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4994146}
I20250901 01:52:35.582325 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ae440a015b744e63b84a172893e319ef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:35.583269 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 36e10328c34242ebb4dbd8f294162696 because the tablet was not found. No further retry: Not found: Tablet not found: 36e10328c34242ebb4dbd8f294162696
I20250901 01:52:35.583606 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7a7cba0e87a94d8a8755bdad66ae6b39"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.584285 2118 tablet_replica.cc:331] T 75a703c8e20a4a2fa0ab6ec58d41ef92 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:35.584699 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_eed6aa04a51b44deb315592ba848362b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.585814 2918 tablet.cc:1722] T 3eba54e5ddd14c518f4cbd539268ae37 P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:35.586022 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_740a5ef73862476e873b1412640eaab1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.585482 2118 raft_consensus.cc:2241] T 75a703c8e20a4a2fa0ab6ec58d41ef92 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:35.588570 2118 raft_consensus.cc:2270] T 75a703c8e20a4a2fa0ab6ec58d41ef92 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:35.589395 2118 ts_tablet_manager.cc:1905] T 75a703c8e20a4a2fa0ab6ec58d41ef92 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:35.591181 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_810947879fdb46fda74c0ec93c532c75"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.592446 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_09f08f31a05d46ff8f3d531f96a67137"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.593377 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1c2a5cebf938404299b4e1dc84a7d326"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.596832 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2c8d21902c6e43629c4f0384b65523f7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.600219 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_798f770d1fec40b7b17dca3c96464b66"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.600620 2113 ts_tablet_manager.cc:1918] T 26c8ed083cea4dc0acca80d00b11b29b P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:35.600728 2113 log.cc:1199] T 26c8ed083cea4dc0acca80d00b11b29b P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/26c8ed083cea4dc0acca80d00b11b29b
I20250901 01:52:35.601137 2113 ts_tablet_manager.cc:1939] T 26c8ed083cea4dc0acca80d00b11b29b P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
W20250901 01:52:35.601325 858 consensus_peers.cc:489] T d0c3c363dda444519a17dca63ad9caa7 P 00c282f779c1477d920f24dc0dc16628 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:35.601972 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e68d8029933e42f4bd62f657463467ad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.602059 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 385) took 4334 ms. Trace:
I20250901 01:52:35.602157 2113 rpcz_store.cc:276] 0901 01:52:31.268063 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:31.269616 (+ 1553us) service_pool.cc:225] Handling call
0901 01:52:35.579343 (+4309727us) ts_tablet_manager.cc:1145] Deleting tablet 26c8ed083cea4dc0acca80d00b11b29b
0901 01:52:35.579355 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.597248 (+ 17893us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:35.600509 (+ 3261us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:35.602035 (+ 1526us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4309553}
I20250901 01:52:35.602648 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 26c8ed083cea4dc0acca80d00b11b29b (table default.Table_5a32379f33204b9e9b2dfe82995134dc [id=5e18d8cdb4fd4399b1a9f8518676f08f]) successfully deleted
I20250901 01:52:35.603026 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_16fbe0d3722e4990bf5c7250f65ca8d3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.603157 2918 tablet.cc:1722] T 615b2306fca84003b264da9d5f95ddab P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6
I20250901 01:52:35.604074 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_02771b5b878044f6b8ce81c401fdc6fb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.604990 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_6a8b9624752645a7ae6bed1cac882cc0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.606141 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d015c875bb674d5abdded49a70381598"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.607079 1490 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_fa623571fcac4341935b9cb46ef4818c" } modify_external_catalogs: true
I20250901 01:52:35.607295 1490 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_fa623571fcac4341935b9cb46ef4818c" } modify_external_catalogs: true
I20250901 01:52:35.608093 1482 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_97c44ea002094df8a6af3dccd0099c39" } new_table_name: "default.Table_f709774f621c483b8ee257273a1e3e38" modify_external_catalogs: true
E20250901 01:52:35.611452 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 494b6488aa0d4355b4caa46004267465, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7c21b68ef2e74136a807c1dd90e17f4e'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.611800 1486 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_c2ed44ae561c4b88b22e8975a61b77aa" } new_table_name: "default.Table_eaa0b271683343998d39a8ca0f151587" modify_external_catalogs: true
I20250901 01:52:35.617064 1489 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_a920a3ffbabf4d83a6563c3423cde12a" } modify_external_catalogs: true
I20250901 01:52:35.617244 1489 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_a920a3ffbabf4d83a6563c3423cde12a" } modify_external_catalogs: true
W20250901 01:52:35.615746 1482 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 415c9d7b552b42c384b444a20491f65c has no consensus state
E20250901 01:52:35.619845 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 01deaae6abb141789f0cd240f3058c73, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7baa92b46d98454a891222be49a6ca48'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.620077 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 0ee3f35392954817b350e5f48653be78, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_442e11b9211c4b209c68f653d7bfc313'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.621528 922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2dacb608897d47d28512df87dc320927" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:35.621791 922 raft_consensus.cc:2466] T 2dacb608897d47d28512df87dc320927 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:35.625080 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 386) took 4356 ms. Trace:
I20250901 01:52:35.625185 2113 rpcz_store.cc:276] 0901 01:52:31.268160 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:31.269822 (+ 1662us) service_pool.cc:225] Handling call
0901 01:52:35.624966 (+4355144us) ts_tablet_manager.cc:1145] Deleting tablet 98cca86164db4252b1b49de88747822a
0901 01:52:35.624978 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.625051 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4354956}
I20250901 01:52:35.626859 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 388) took 4358 ms. Trace:
I20250901 01:52:35.626964 2113 rpcz_store.cc:276] 0901 01:52:31.268323 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:31.270033 (+ 1710us) service_pool.cc:225] Handling call
0901 01:52:35.626757 (+4356724us) ts_tablet_manager.cc:1145] Deleting tablet d58f69fad01f403eab74a47d3695ed83
0901 01:52:35.626771 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.626841 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4356592}
I20250901 01:52:35.627454 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 389) took 4359 ms. Trace:
I20250901 01:52:35.627542 2113 rpcz_store.cc:276] 0901 01:52:31.268467 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:31.270185 (+ 1718us) service_pool.cc:225] Handling call
0901 01:52:35.627366 (+4357181us) ts_tablet_manager.cc:1145] Deleting tablet d77aadd7291647a0bae7c421f244c252
0901 01:52:35.627377 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.627438 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4357063}
I20250901 01:52:35.631191 2918 tablet.cc:1722] T 89d40f0331114490b51de3ba6d773571 P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250901 01:52:35.632586 2113 tablet_replica.cc:331] T 3d90c6ef06d1436d8e2bb1333e7d9985 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:35.632790 2113 raft_consensus.cc:2241] T 3d90c6ef06d1436d8e2bb1333e7d9985 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:35.633106 2911 raft_consensus.cc:491] T 7f46e9ddadaf4ff986d8c991178daa0c P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:35.633204 2911 raft_consensus.cc:513] T 7f46e9ddadaf4ff986d8c991178daa0c P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:35.633531 2113 raft_consensus.cc:2270] T 3d90c6ef06d1436d8e2bb1333e7d9985 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:35.633739 2911 leader_election.cc:290] T 7f46e9ddadaf4ff986d8c991178daa0c P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:35.634071 2113 ts_tablet_manager.cc:1905] T 3d90c6ef06d1436d8e2bb1333e7d9985 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:35.620362 1491 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_8cd45b1e166e4b0c80647bc3c11cb7ee" } modify_external_catalogs: true
W20250901 01:52:35.625725 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 98cca86164db4252b1b49de88747822a because the tablet was not found. No further retry: Not found: Tablet not found: 98cca86164db4252b1b49de88747822a
I20250901 01:52:35.636308 1491 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_8cd45b1e166e4b0c80647bc3c11cb7ee" } modify_external_catalogs: true
W20250901 01:52:35.636711 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet d58f69fad01f403eab74a47d3695ed83 because the tablet was not found. No further retry: Not found: Tablet not found: d58f69fad01f403eab74a47d3695ed83
W20250901 01:52:35.637027 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet d77aadd7291647a0bae7c421f244c252 because the tablet was not found. No further retry: Not found: Tablet not found: d77aadd7291647a0bae7c421f244c252
I20250901 01:52:35.634292 1483 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_5e84ae9a9c2a4084a1f9c5ffb4594de2" } new_table_name: "default.Table_4cb98468d9f44e16aab412c39b234c15" modify_external_catalogs: true
I20250901 01:52:35.635475 1482 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_3b46a0a1f77a4cea842e19b3c89de475" } new_table_name: "default.Table_360f4423e7f341339188b71cc82a1149" modify_external_catalogs: true
E20250901 01:52:35.640988 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 77911bcb98cf4d8890118eca9136f86e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_db61e055dbd7450898e82c4c0c4763be'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.641211 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae4844183a3c4a04a775e6fd878391a3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_4bd66ab99a454974aa0f7f4290edfda2'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.644670 1491 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 6c0abbfc8d904dce989ce51961e4237d
W20250901 01:52:35.649612 1491 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 6c0abbfc8d904dce989ce51961e4237d: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:35.645982 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_6830934323be4d32a03af4d8dbc36c8d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.650270 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 6c0abbfc8d904dce989ce51961e4237d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:35 UTC) from {username='slave'} at 127.0.0.1:35946
W20250901 01:52:35.648159 1483 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ae9a4ad4fdfb4771ba23a91f7007447e has no consensus state
I20250901 01:52:35.651338 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 6c0abbfc8d904dce989ce51961e4237d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:35 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:35.648272 1492 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_442e11b9211c4b209c68f653d7bfc313" } modify_external_catalogs: true
I20250901 01:52:35.651623 1492 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_442e11b9211c4b209c68f653d7bfc313" } modify_external_catalogs: true
I20250901 01:52:35.648900 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_543428ba89cf42d4b581abf5d5a9acbd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:35.653079 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9745582cebfe47b58c21d0191ab4cb9b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77728af9eee44f29a1e2131d5b75b7ca'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.656002 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_dc05999ede1b46dc8b8fd041c7b879c9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:35.653330 1696 catalog_manager.cc:6205] Error selecting replicas for tablet e2ed32b3292141278af28026528e96c2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b5077f55951d48be92a67407ffe26a08'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.657164 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9f62a9d97f514747b940b488907369f3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d53a456dbca64daf92aeb5fb4517b417'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.657416 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4caca784717a41e18d4a37bb6bd87cb0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.658336 2113 ts_tablet_manager.cc:1918] T 3d90c6ef06d1436d8e2bb1333e7d9985 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:35.658435 2113 log.cc:1199] T 3d90c6ef06d1436d8e2bb1333e7d9985 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/3d90c6ef06d1436d8e2bb1333e7d9985
I20250901 01:52:35.658833 2113 ts_tablet_manager.cc:1939] T 3d90c6ef06d1436d8e2bb1333e7d9985 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:35.659662 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 392) took 4313 ms. Trace:
I20250901 01:52:35.659744 2113 rpcz_store.cc:276] 0901 01:52:31.346060 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:31.346112 (+ 52us) service_pool.cc:225] Handling call
0901 01:52:35.632521 (+4286409us) ts_tablet_manager.cc:1145] Deleting tablet 3d90c6ef06d1436d8e2bb1333e7d9985
0901 01:52:35.632533 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.642831 (+ 10298us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:35.658232 (+ 15401us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:35.659643 (+ 1411us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":569,"tablet-delete.queue_time_us":4286220}
I20250901 01:52:35.660166 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 3d90c6ef06d1436d8e2bb1333e7d9985 (table default.Table_5facf725e0fc44cda7d77f8511030fa5 [id=892696e8040d4f2db8620d1d4ac90e0f]) successfully deleted
I20250901 01:52:35.657888 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ec14bce74c4849fab50e2d197286e5f0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:35.660960 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5a7fa68d29644ae8afd2711e39dde59a, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d24e3a18ef4843fdb7b74ce23d47cf9e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.661070 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 415c9d7b552b42c384b444a20491f65c, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_f709774f621c483b8ee257273a1e3e38'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.661180 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ff2bd017417c41b5b8225b8f00283e63, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0c7355975d6b44dbbe3a77df942b4d43'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.661270 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2821bf7750ba4e38b25ccba2e2bc0eab, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ce2f3b515c8f4c1bb557c5ed84b48dba'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.661370 1696 catalog_manager.cc:6205] Error selecting replicas for tablet dcc60bb000f440f1aca9133928b5ce20, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2fbca8463d6546c9b8f5b018fc60251d'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.661453 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 465c02a130c54e09b5a74fb4dd2ef762, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ed6fd284be6f4ae3a008e21e7ce17930'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.661572 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 75883d9fd00e429aa8853903a9fdd7e4, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2e67ad583a0b413e9190cb55137ccf3a'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.661661 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f8f923ddcd2740bcb7e521f4f57c4034, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_e09ac1a9050547038400d4b52299d22f'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.661763 1696 catalog_manager.cc:6205] Error selecting replicas for tablet b1ba0e19dc0b462e80c537bf2d183f5b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_66b111271c15410aab23682f8f6081ac'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.661850 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 8e10a01e77d14cf28c2c5e0205e595b9, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_83cf2d86a1284544ba2e7aa9ebb66bb7'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.661953 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5f0e054f62de4cf6bc6f69ad6664b192, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ef6832d0845044ed9f165ca3541b4c3f'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.662036 1696 catalog_manager.cc:6205] Error selecting replicas for tablet fd2ce82a4ee24d0faa44980a3967e187, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_63d0f2ead9d946d39ce40162ebd769d6'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.662132 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae9a4ad4fdfb4771ba23a91f7007447e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_4cb98468d9f44e16aab412c39b234c15'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.662212 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 086aa948f93644da8e5835b0fac87ddd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b4fbc34da806475c912052d0fab3fc53'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.662318 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 082db24bb3fd4089950504145a20a075, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_4473168010a84ac7a014b99dd10e35a9'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.662395 1696 catalog_manager.cc:6205] Error selecting replicas for tablet fefbf5ae030446b292d8068e95577534, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_3dde393d3a564e84b8ef08c0d25265ce'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.662488 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 93aa22afc0ad4d7fa24807ea3c5d5c21, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0f4b4094d2974b7591875375eda6bed0'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.662571 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9074543dea9949e7a085717b40913348, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_53e30205c5fd488eb3b48c685a159d04'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.662659 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 3a62034e688348ed9e2ac9de009ed3a2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_e225c82a2d6045f6b16915cdb72f5734'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.662734 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 18498c9cb09141d08c1c5c0854b63ebd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77f4810bc83c4d948531da2a45a3792e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.662822 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f5e3316adeae47cd848833dd84b972bb, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_fc962c8bc653498e85cc0bb41639e9ad'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.662899 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 81e8c6567953484ab000fe2eb4398301, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_a920a3ffbabf4d83a6563c3423cde12a'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.662993 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 6ce9521b925b4f97bb4a810ce77e147e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_bd74ae506bbe40489259a7be70c3d10b'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.663070 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2f61edb50d7a42198bfd70639c7bf4de, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_fe79329825c54cdf8c97a14aa8a30384'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.663501 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_cff4504da69b4e5dbad403b82b74dd65"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.664613 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2bd5353b6a0c4c179f7b2fa7b95b8778"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.665544 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2703dceb01334a27b439027a00b57068"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.667675 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_36ad5f7d84ef44b6a3381d25547473f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.668641 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9389fc2bdc4e4123ac829a544bb78461"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.669147 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 396) took 4146 ms. Trace:
I20250901 01:52:35.669391 1484 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_2e67ad583a0b413e9190cb55137ccf3a" } new_table_name: "default.Table_a9f6ac8609554a38ab632b2a3f7113d8" modify_external_catalogs: true
I20250901 01:52:35.673058 1486 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_ef6832d0845044ed9f165ca3541b4c3f" } modify_external_catalogs: true
I20250901 01:52:35.673297 1486 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_ef6832d0845044ed9f165ca3541b4c3f" } modify_external_catalogs: true
I20250901 01:52:35.669291 2113 rpcz_store.cc:276] 0901 01:52:31.523117 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:31.523163 (+ 46us) service_pool.cc:225] Handling call
0901 01:52:35.668985 (+4145822us) ts_tablet_manager.cc:1145] Deleting tablet bd9b59f0864a497faf281edd6077668d
0901 01:52:35.668999 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.669123 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4144673}
W20250901 01:52:35.674566 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet bd9b59f0864a497faf281edd6077668d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd9b59f0864a497faf281edd6077668d already in progress: creating tablet
I20250901 01:52:35.674949 2113 tablet_replica.cc:331] T fae2779fac164b9f8052c3666db803b8 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:35.675253 2113 raft_consensus.cc:2241] T fae2779fac164b9f8052c3666db803b8 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:35.675523 2113 raft_consensus.cc:2270] T fae2779fac164b9f8052c3666db803b8 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250901 01:52:35.676818 1484 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 75883d9fd00e429aa8853903a9fdd7e4 has no consensus state
I20250901 01:52:35.677212 2113 ts_tablet_manager.cc:1905] T fae2779fac164b9f8052c3666db803b8 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:35.683012 2118 ts_tablet_manager.cc:1918] T 75a703c8e20a4a2fa0ab6ec58d41ef92 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:35.684877 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_017350849a7c408d8015e25141b60fd5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.686101 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_509bf9d6bbb348e6a9e049f3b3fa6e6d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.687108 2118 log.cc:1199] T 75a703c8e20a4a2fa0ab6ec58d41ef92 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/75a703c8e20a4a2fa0ab6ec58d41ef92
I20250901 01:52:35.687433 2118 ts_tablet_manager.cc:1939] T 75a703c8e20a4a2fa0ab6ec58d41ef92 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:35.688081 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 262) took 5102 ms. Trace:
I20250901 01:52:35.688160 2118 rpcz_store.cc:276] 0901 01:52:30.585544 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.585608 (+ 64us) service_pool.cc:225] Handling call
0901 01:52:35.584238 (+4998630us) ts_tablet_manager.cc:1145] Deleting tablet 75a703c8e20a4a2fa0ab6ec58d41ef92
0901 01:52:35.584246 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.641450 (+ 57204us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:35.682887 (+ 41437us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:35.688059 (+ 5172us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4998486}
I20250901 01:52:35.689687 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7dfa0f40583e45c8819773cba18df002"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.689785 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 75a703c8e20a4a2fa0ab6ec58d41ef92 (table default.Table_77728af9eee44f29a1e2131d5b75b7ca [id=a90d1f10531042c3bb7a196b8be61c5f]) successfully deleted
I20250901 01:52:35.693127 2113 ts_tablet_manager.cc:1918] T fae2779fac164b9f8052c3666db803b8 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:35.693253 2113 log.cc:1199] T fae2779fac164b9f8052c3666db803b8 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/fae2779fac164b9f8052c3666db803b8
I20250901 01:52:35.693622 2113 ts_tablet_manager.cc:1939] T fae2779fac164b9f8052c3666db803b8 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:35.698953 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_160b8850bf794ef799a53e99cea1bb63"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.700134 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_edc1b2f29f69420da36277f6eec89791"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.701323 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_126177a26eef4894bb819768278cafa7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.702316 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f7cac842fbbe4fefbc1a0397bc6fb5c6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.703471 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_29eb0f3f75e240449507e20908732061"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.694413 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 398) took 4142 ms. Trace:
I20250901 01:52:35.704789 2113 rpcz_store.cc:276] 0901 01:52:31.551879 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:31.551937 (+ 58us) service_pool.cc:225] Handling call
0901 01:52:35.674887 (+4122950us) ts_tablet_manager.cc:1145] Deleting tablet fae2779fac164b9f8052c3666db803b8
0901 01:52:35.674900 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.688985 (+ 14085us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:35.693033 (+ 4048us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:35.694392 (+ 1359us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4122771}
I20250901 01:52:35.705267 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet fae2779fac164b9f8052c3666db803b8 (table default.Table_7736ed4b5a0d4b1a816500c525e68ad3 [id=7191ff77d44c4c86aa44473814570d09]) successfully deleted
I20250901 01:52:35.706363 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 400) took 3533 ms. Trace:
I20250901 01:52:35.706442 2113 rpcz_store.cc:276] 0901 01:52:32.172741 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:32.172800 (+ 59us) service_pool.cc:225] Handling call
0901 01:52:35.706270 (+3533470us) ts_tablet_manager.cc:1145] Deleting tablet 219fe3882d884ee49b08671b547cac85
0901 01:52:35.706281 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.706347 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3529340}
W20250901 01:52:35.706862 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 219fe3882d884ee49b08671b547cac85 because the tablet was not found. No further retry: Not found: Tablet not found: 219fe3882d884ee49b08671b547cac85
I20250901 01:52:35.707482 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9df3dfde6b8d44dba8ae0b32e69538f5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.708717 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_46240aa954b948a29007dc3838dd4ec2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:35.711218 1473 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet ae9a4ad4fdfb4771ba23a91f7007447e (table default.Table_4cb98468d9f44e16aab412c39b234c15 [id=9bebb2e486ab48888e6b523281ac1e88], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet ae9a4ad4fdfb4771ba23a91f7007447e has no consensus state
I20250901 01:52:35.714879 2113 tablet_replica.cc:331] T 6ba8241644984394ada95b561e76467e P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:35.715097 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 263) took 5122 ms. Trace:
I20250901 01:52:35.715190 2118 rpcz_store.cc:276] 0901 01:52:30.592836 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.592885 (+ 49us) service_pool.cc:225] Handling call
0901 01:52:35.714991 (+5122106us) ts_tablet_manager.cc:1145] Deleting tablet 06d5eeb70e664f73bf643922d445bc7a
0901 01:52:35.715003 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.715077 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5121938}
I20250901 01:52:35.715251 2113 raft_consensus.cc:2241] T 6ba8241644984394ada95b561e76467e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:35.715488 2113 raft_consensus.cc:2270] T 6ba8241644984394ada95b561e76467e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:35.715565 2118 tablet_replica.cc:331] T 438ae684af234c868a419fbab43b0b98 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:35.715739 2118 raft_consensus.cc:2241] T 438ae684af234c868a419fbab43b0b98 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:35.715876 2118 raft_consensus.cc:2270] T 438ae684af234c868a419fbab43b0b98 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:35.716324 2118 ts_tablet_manager.cc:1905] T 438ae684af234c868a419fbab43b0b98 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:35.716679 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 06d5eeb70e664f73bf643922d445bc7a because the tablet was not found. No further retry: Not found: Tablet not found: 06d5eeb70e664f73bf643922d445bc7a
I20250901 01:52:35.716713 2113 ts_tablet_manager.cc:1905] T 6ba8241644984394ada95b561e76467e P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:35.719874 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d4c9aa723aee4dc4b6237cb5c3722b52" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:35.722446 1054 raft_consensus.cc:1273] T d0c3c363dda444519a17dca63ad9caa7 P fd27ab6665304aaea53adbd102dd4f58 [term 3 FOLLOWER]: Refusing update from remote peer 00c282f779c1477d920f24dc0dc16628: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250901 01:52:35.722632 1490 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet e9f1f892658b4736859ff507536920a1
W20250901 01:52:35.722856 1490 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet e9f1f892658b4736859ff507536920a1: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:35.723680 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet e9f1f892658b4736859ff507536920a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:35 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:35.723768 1489 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 81e8c6567953484ab000fe2eb4398301
I20250901 01:52:35.724215 886 tablet_service.cc:1515] Processing DeleteTablet for tablet e9f1f892658b4736859ff507536920a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:35 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:35.724287 1492 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 0ee3f35392954817b350e5f48653be78
I20250901 01:52:35.724566 1486 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 5f0e054f62de4cf6bc6f69ad6664b192
I20250901 01:52:35.724711 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c64b0519cb8e43fbaf1c078e1c924724"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.728336 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1497ccfdf0ec4c169b2aa917a576654a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.732308 1151 tablet_bootstrap.cc:492] T 3dae00aa38204ee3ac77325eb334be42 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:35.732539 1151 ts_tablet_manager.cc:1397] T 3dae00aa38204ee3ac77325eb334be42 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.171s user 0.002s sys 0.001s
I20250901 01:52:35.733322 1151 raft_consensus.cc:357] T 3dae00aa38204ee3ac77325eb334be42 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:35.733552 1151 raft_consensus.cc:383] T 3dae00aa38204ee3ac77325eb334be42 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:35.733636 1151 raft_consensus.cc:738] T 3dae00aa38204ee3ac77325eb334be42 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:35.734423 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_bd02c8b798a04caca7c2284fadc8e616"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.734611 1151 consensus_queue.cc:260] T 3dae00aa38204ee3ac77325eb334be42 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:35.735145 1151 ts_tablet_manager.cc:1428] T 3dae00aa38204ee3ac77325eb334be42 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250901 01:52:35.735621 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_cd28fbd1dc6b48b39b925031269894e9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.736949 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_fdd743e1471d4e6ebd05daa5a1dd31b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:35.737423 992 leader_election.cc:336] T 6ba8241644984394ada95b561e76467e P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:35.738056 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a491bca760504376bb649d91b074e584"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.738876 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3d3459ed5c8345e092b55c1007492d89"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.736568 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_250d22984c0d471ab549dc094299b118"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.740473 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c7f158cf6a504f0ba8e888bb8fbde4c4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:35.740624 858 consensus_peers.cc:489] T 14b466b2a5a249aa841f1b8b06301edc P 00c282f779c1477d920f24dc0dc16628 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250901 01:52:35.741447 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d7a6672638d74d7fafea8c9af4254297"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.742995 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_411c8b47fdb645089f3271612ed467b1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.744297 1151 tablet_bootstrap.cc:492] T 88ceba7f48b04d7885cc0164668700a5 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
W20250901 01:52:35.745923 858 consensus_peers.cc:489] T 508406601c994e42a5c5b1f665e74957 P 00c282f779c1477d920f24dc0dc16628 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:35.746204 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9eda7e2dce534086ad250d5b82671706"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.746968 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f002646edec242cf975e6c74452cbc3e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.748068 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_010477a553ac4fd6bd6ed9e1fcf1f0bf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.748870 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_6038d8d1cd00432ba6e73463f602f518"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:35.753386 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 494b6488aa0d4355b4caa46004267465, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7c21b68ef2e74136a807c1dd90e17f4e'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.754783 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5f810e54fc95496b886810a512081d57"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.755240 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2e6f76aa4ed24a23a95e003e7f870b99"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.755693 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1acbd8d872d94810a62d293ed5908cd0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:35.757789 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 01deaae6abb141789f0cd240f3058c73, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7baa92b46d98454a891222be49a6ca48'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.758083 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_67d1c260e51f4153b23015602fd75eb1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:35.758996 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 77911bcb98cf4d8890118eca9136f86e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_db61e055dbd7450898e82c4c0c4763be'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.759764 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae4844183a3c4a04a775e6fd878391a3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_4bd66ab99a454974aa0f7f4290edfda2'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.759994 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9745582cebfe47b58c21d0191ab4cb9b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77728af9eee44f29a1e2131d5b75b7ca'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.759282 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1f65a4636d7f4df4a0d9b207c94d8db1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:35.767787 1696 catalog_manager.cc:6205] Error selecting replicas for tablet e2ed32b3292141278af28026528e96c2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b5077f55951d48be92a67407ffe26a08'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.768155 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_73d6db93d01a4204901f9a28c80fb78b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.768764 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_75ee31aaada64c53bfe8ebaaf752041f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:35.769582 859 consensus_peers.cc:489] T e87d1281f35648b0b115b67c95e244e7 P 00c282f779c1477d920f24dc0dc16628 -> Peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Couldn't send request to peer fd27ab6665304aaea53adbd102dd4f58. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e87d1281f35648b0b115b67c95e244e7. This is attempt 1: this message will repeat every 5th retry.
E20250901 01:52:35.770148 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9f62a9d97f514747b940b488907369f3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d53a456dbca64daf92aeb5fb4517b417'. Need at least 3 replicas, but only 2 tablet servers are available
W20250901 01:52:35.770236 859 consensus_peers.cc:489] T 3d90c6ef06d1436d8e2bb1333e7d9985 P 00c282f779c1477d920f24dc0dc16628 -> Peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Couldn't send request to peer fd27ab6665304aaea53adbd102dd4f58. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:35.770407 1490 master_service.cc:948] ReplaceTablet: received request to replace tablet fcb7b5b7d63d4f0b906fea97cb548de2 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:35.770917 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_aa60a9b2af104421bdc3f740c9db0e7b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.771664 921 raft_consensus.cc:1273] T 2973b59bb1df41d18d9a5b6a29ae336f P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Refusing update from remote peer fd27ab6665304aaea53adbd102dd4f58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
E20250901 01:52:35.772184 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5a7fa68d29644ae8afd2711e39dde59a, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d24e3a18ef4843fdb7b74ce23d47cf9e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.772382 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 415c9d7b552b42c384b444a20491f65c, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_f709774f621c483b8ee257273a1e3e38'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.778071 1151 tablet_bootstrap.cc:654] T 88ceba7f48b04d7885cc0164668700a5 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:35.772806 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e3fd9357e03244599fd2921c0df92c62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.780298 2113 ts_tablet_manager.cc:1918] T 6ba8241644984394ada95b561e76467e P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:35.773489 1486 master_service.cc:948] ReplaceTablet: received request to replace tablet 6ba8241644984394ada95b561e76467e from {username='slave'} at 127.0.0.1:33478
E20250901 01:52:35.781370 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ff2bd017417c41b5b8225b8f00283e63, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0c7355975d6b44dbbe3a77df942b4d43'. Need at least 3 replicas, but only 2 tablet servers are available
W20250901 01:52:35.781773 859 consensus_peers.cc:489] T d000ba05e12843b4925e622ab55471c0 P 00c282f779c1477d920f24dc0dc16628 -> Peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Couldn't send request to peer fd27ab6665304aaea53adbd102dd4f58. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d000ba05e12843b4925e622ab55471c0. This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:35.781803 2911 raft_consensus.cc:491] T 44af4764d4f64e44aec84ccb84be6f5f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:35.781919 2911 raft_consensus.cc:513] T 44af4764d4f64e44aec84ccb84be6f5f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:35.782073 1474 consensus_peers.cc:489] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f -> Peer d2ae2cb6af0f4b3a8f47ab1cdd6567d3 (127.0.104.61:42375): Couldn't send request to peer d2ae2cb6af0f4b3a8f47ab1cdd6567d3. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.61:42375: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
E20250901 01:52:35.782166 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2821bf7750ba4e38b25ccba2e2bc0eab, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ce2f3b515c8f4c1bb557c5ed84b48dba'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.782495 2911 leader_election.cc:290] T 44af4764d4f64e44aec84ccb84be6f5f P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:35.782765 2113 log.cc:1199] T 6ba8241644984394ada95b561e76467e P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/6ba8241644984394ada95b561e76467e
E20250901 01:52:35.782891 1696 catalog_manager.cc:6205] Error selecting replicas for tablet dcc60bb000f440f1aca9133928b5ce20, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2fbca8463d6546c9b8f5b018fc60251d'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.782752 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f9926cde2d3b40e0babd041116719d10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:35.783921 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 465c02a130c54e09b5a74fb4dd2ef762, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ed6fd284be6f4ae3a008e21e7ce17930'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.784971 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_462249aa7e9d48dcb41c0b6c7144e2cc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.786659 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3201085ec0044a17a77036244e5e8b08"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:35.785748 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 75883d9fd00e429aa8853903a9fdd7e4, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_a9f6ac8609554a38ab632b2a3f7113d8'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.789064 2871 raft_consensus.cc:491] T c83ce95fd6fa41478de0e2168be201b1 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:35.789191 2871 raft_consensus.cc:513] T c83ce95fd6fa41478de0e2168be201b1 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:35.789734 2871 leader_election.cc:290] T c83ce95fd6fa41478de0e2168be201b1 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:35.791081 2113 ts_tablet_manager.cc:1939] T 6ba8241644984394ada95b561e76467e P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:35.791535 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 402) took 3137 ms. Trace:
I20250901 01:52:35.791633 2113 rpcz_store.cc:276] 0901 01:52:32.653727 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:32.653789 (+ 62us) service_pool.cc:225] Handling call
0901 01:52:35.714710 (+3060921us) ts_tablet_manager.cc:1145] Deleting tablet 6ba8241644984394ada95b561e76467e
0901 01:52:35.714835 (+ 125us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.743897 (+ 29062us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:35.780162 (+ 36265us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:35.791514 (+ 11352us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3060723}
I20250901 01:52:35.783668 992 leader_election.cc:304] T 6ba8241644984394ada95b561e76467e P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:35.792140 2118 ts_tablet_manager.cc:1918] T 438ae684af234c868a419fbab43b0b98 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250901 01:52:35.792261 992 raft_consensus.cc:2684] T 6ba8241644984394ada95b561e76467e P fd27ab6665304aaea53adbd102dd4f58: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250901 01:52:35.792268 858 leader_election.cc:336] T c83ce95fd6fa41478de0e2168be201b1 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:35.792912 992 consensus_peers.cc:489] T 764bfeb2a7d14c40815b5ec1b07bdd62 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Couldn't send request to peer 00c282f779c1477d920f24dc0dc16628. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 764bfeb2a7d14c40815b5ec1b07bdd62. This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:35.792975 2118 log.cc:1199] T 438ae684af234c868a419fbab43b0b98 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/438ae684af234c868a419fbab43b0b98
W20250901 01:52:35.793227 992 leader_election.cc:343] T 088ef4861dc14f81ae470c3aaee5b4e1 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 088ef4861dc14f81ae470c3aaee5b4e1
I20250901 01:52:35.794093 2118 ts_tablet_manager.cc:1939] T 438ae684af234c868a419fbab43b0b98 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:35.792054 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 6ba8241644984394ada95b561e76467e (table default.Table_a96f01d4ce3c4694a635575ffaa4f02d [id=5dc1ab36f2974f07866b14fedbdabd57]) successfully deleted
E20250901 01:52:35.794457 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f8f923ddcd2740bcb7e521f4f57c4034, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_e09ac1a9050547038400d4b52299d22f'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.794806 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_45a01cca3d484016a065515397ad0012"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.795410 1486 master_service.cc:948] ReplaceTablet: received request to replace tablet b009cc95c819431ea9827a3e453b8da3 from {username='slave'} at 127.0.0.1:33478
E20250901 01:52:35.796662 1696 catalog_manager.cc:6205] Error selecting replicas for tablet b1ba0e19dc0b462e80c537bf2d183f5b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_66b111271c15410aab23682f8f6081ac'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.797369 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 8e10a01e77d14cf28c2c5e0205e595b9, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_83cf2d86a1284544ba2e7aa9ebb66bb7'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.797189 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 264) took 5184 ms. Trace:
I20250901 01:52:35.797572 1490 master_service.cc:948] ReplaceTablet: received request to replace tablet f6a8b425968e49deaecfa3adacfd3f1f from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:35.797737 2118 rpcz_store.cc:276] 0901 01:52:30.613054 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.613109 (+ 55us) service_pool.cc:225] Handling call
0901 01:52:35.715526 (+5102417us) ts_tablet_manager.cc:1145] Deleting tablet 438ae684af234c868a419fbab43b0b98
0901 01:52:35.715533 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.747140 (+ 31607us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:35.791762 (+ 44622us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:35.797170 (+ 5408us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5099960}
I20250901 01:52:35.799130 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 438ae684af234c868a419fbab43b0b98 (table default.Table_95515169b000454aa95025573560278f [id=1442b92adbbe43a6b8580cce1bbfe389]) successfully deleted
I20250901 01:52:35.799237 2113 tablet_replica.cc:331] T 44af4764d4f64e44aec84ccb84be6f5f P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:35.799424 2113 raft_consensus.cc:2241] T 44af4764d4f64e44aec84ccb84be6f5f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:35.799854 992 leader_election.cc:304] T ae21b9ae2e3b472eb67fb5f96388a50f P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:35.800058 2911 raft_consensus.cc:2802] T ae21b9ae2e3b472eb67fb5f96388a50f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:35.800125 2911 raft_consensus.cc:491] T ae21b9ae2e3b472eb67fb5f96388a50f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:35.800185 2911 raft_consensus.cc:3058] T ae21b9ae2e3b472eb67fb5f96388a50f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:35.800484 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 265) took 5183 ms. Trace:
E20250901 01:52:35.801209 1696 catalog_manager.cc:6205] Error selecting replicas for tablet fd2ce82a4ee24d0faa44980a3967e187, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_63d0f2ead9d946d39ce40162ebd769d6'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.801895 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae9a4ad4fdfb4771ba23a91f7007447e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_4cb98468d9f44e16aab412c39b234c15'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.802073 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 086aa948f93644da8e5835b0fac87ddd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b4fbc34da806475c912052d0fab3fc53'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.802906 2911 raft_consensus.cc:513] T ae21b9ae2e3b472eb67fb5f96388a50f P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:35.801046 2118 rpcz_store.cc:276] 0901 01:52:30.616024 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.616067 (+ 43us) service_pool.cc:225] Handling call
0901 01:52:35.799523 (+5183456us) ts_tablet_manager.cc:1145] Deleting tablet 606fac5c6efe48d1877104b01c23ba5b
0901 01:52:35.799536 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.799940 (+ 404us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5183153}
I20250901 01:52:35.803439 2911 leader_election.cc:290] T ae21b9ae2e3b472eb67fb5f96388a50f P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:35.804201 2113 raft_consensus.cc:2270] T 44af4764d4f64e44aec84ccb84be6f5f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:35.805442 2113 ts_tablet_manager.cc:1905] T 44af4764d4f64e44aec84ccb84be6f5f P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:35.797008 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e5672597a72d46dbb202c3af6c34d0c7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.804999 1490 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet f6a8b425968e49deaecfa3adacfd3f1f
I20250901 01:52:35.805078 1486 master_service.cc:948] ReplaceTablet: received request to replace tablet 49aaaec611784e6bac131efd58dbca51 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:35.806483 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 266) took 5190 ms. Trace:
W20250901 01:52:35.805877 1490 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet f6a8b425968e49deaecfa3adacfd3f1f: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:35.807041 2118 rpcz_store.cc:276] 0901 01:52:30.616164 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.616293 (+ 129us) service_pool.cc:225] Handling call
0901 01:52:35.804580 (+5188287us) ts_tablet_manager.cc:1145] Deleting tablet 6cf09e8f299548dba2cce16d7b565bad
0901 01:52:35.804594 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.806351 (+ 1757us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5188014}
I20250901 01:52:35.808349 1490 catalog_manager.cc:6652] ReplaceTablet: tablet f6a8b425968e49deaecfa3adacfd3f1f deleted and replaced by tablet 8f4f5563df6b4d0db890747ba032ef9f
W20250901 01:52:35.809267 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 6cf09e8f299548dba2cce16d7b565bad because the tablet was not found. No further retry: Not found: Tablet not found: 6cf09e8f299548dba2cce16d7b565bad
E20250901 01:52:35.805224 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 082db24bb3fd4089950504145a20a075, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_4473168010a84ac7a014b99dd10e35a9'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.809710 893 tablet_service.cc:1515] Processing DeleteTablet for tablet f6a8b425968e49deaecfa3adacfd3f1f with delete_type TABLET_DATA_DELETED (replaced by tablet 8f4f5563df6b4d0db890747ba032ef9f) from {username='slave'} at 127.0.0.1:34472
E20250901 01:52:35.810060 1696 catalog_manager.cc:6205] Error selecting replicas for tablet fefbf5ae030446b292d8068e95577534, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_3dde393d3a564e84b8ef08c0d25265ce'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.809535 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 268) took 5193 ms. Trace:
I20250901 01:52:35.810384 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e3cc585e96734778ba8ed7b78df8b06b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.810314 2118 rpcz_store.cc:276] 0901 01:52:30.616428 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.616475 (+ 47us) service_pool.cc:225] Handling call
0901 01:52:35.808857 (+5192382us) ts_tablet_manager.cc:1145] Deleting tablet f62d96c37cca4143a4b098b7c69979e6
0901 01:52:35.808870 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.809518 (+ 648us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5192134}
E20250901 01:52:35.811489 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 93aa22afc0ad4d7fa24807ea3c5d5c21, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0f4b4094d2974b7591875375eda6bed0'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.811673 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9074543dea9949e7a085717b40913348, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_53e30205c5fd488eb3b48c685a159d04'. Need at least 3 replicas, but only 2 tablet servers are available
W20250901 01:52:35.811851 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet f62d96c37cca4143a4b098b7c69979e6 because the tablet was not found. No further retry: Not found: Tablet not found: f62d96c37cca4143a4b098b7c69979e6
I20250901 01:52:35.811962 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_970749cac4d84ec3a19c56d2f0a965c1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:35.813081 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 3a62034e688348ed9e2ac9de009ed3a2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_e225c82a2d6045f6b16915cdb72f5734'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.812873 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 269) took 5196 ms. Trace:
I20250901 01:52:35.813374 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5c9f79db678d44e29d05fb6d95a7acb3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.813409 2118 rpcz_store.cc:276] 0901 01:52:30.616551 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.616689 (+ 138us) service_pool.cc:225] Handling call
0901 01:52:35.812361 (+5195672us) ts_tablet_manager.cc:1145] Deleting tablet 246b33338de647689d7404c70676e76f
0901 01:52:35.812374 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.812854 (+ 480us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5192172}
E20250901 01:52:35.814271 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 18498c9cb09141d08c1c5c0854b63ebd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77f4810bc83c4d948531da2a45a3792e'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.814599 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_19bfcdb9dfb340d88648eb5f4ff83a1c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.809136 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet f6a8b425968e49deaecfa3adacfd3f1f with delete_type TABLET_DATA_DELETED (replaced by tablet 8f4f5563df6b4d0db890747ba032ef9f) from {username='slave'} at 127.0.0.1:35946
W20250901 01:52:35.814695 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 246b33338de647689d7404c70676e76f because the tablet was not found. No further retry: Not found: Tablet not found: 246b33338de647689d7404c70676e76f
I20250901 01:52:35.814165 992 leader_election.cc:304] T 2dacb608897d47d28512df87dc320927 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
E20250901 01:52:35.815623 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f5e3316adeae47cd848833dd84b972bb, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_fc962c8bc653498e85cc0bb41639e9ad'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.815757 2911 raft_consensus.cc:2802] T 2dacb608897d47d28512df87dc320927 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:35.815873 2911 raft_consensus.cc:491] T 2dacb608897d47d28512df87dc320927 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:35.815974 2911 raft_consensus.cc:3058] T 2dacb608897d47d28512df87dc320927 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:35.815893 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 267) took 5199 ms. Trace:
E20250901 01:52:35.817106 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 6ce9521b925b4f97bb4a810ce77e147e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_bd74ae506bbe40489259a7be70c3d10b'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.816354 2118 rpcz_store.cc:276] 0901 01:52:30.616306 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.616709 (+ 403us) service_pool.cc:225] Handling call
0901 01:52:35.815258 (+5198549us) ts_tablet_manager.cc:1145] Deleting tablet 6c05eff658924ca4a9a1e7a37f917d67
0901 01:52:35.815381 (+ 123us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.815767 (+ 386us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5195070}
E20250901 01:52:35.817443 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2f61edb50d7a42198bfd70639c7bf4de, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_fe79329825c54cdf8c97a14aa8a30384'. Need at least 3 replicas, but only 2 tablet servers are available
W20250901 01:52:35.817840 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 6c05eff658924ca4a9a1e7a37f917d67 because the tablet was not found. No further retry: Not found: Tablet not found: 6c05eff658924ca4a9a1e7a37f917d67
I20250901 01:52:35.817839 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_cb132e890af04468b18ec6a49416d16a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.818751 2118 tablet_replica.cc:331] T f195b040a6ae4b7b83a8308ab0a12b37 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:35.819167 2911 raft_consensus.cc:513] T 2dacb608897d47d28512df87dc320927 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:35.819303 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8cba1c11563b4992b3f28893b0d559ed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.819826 2911 leader_election.cc:290] T 2dacb608897d47d28512df87dc320927 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:35.819939 2118 raft_consensus.cc:2241] T f195b040a6ae4b7b83a8308ab0a12b37 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:35.820744 2118 pending_rounds.cc:70] T f195b040a6ae4b7b83a8308ab0a12b37 P 00c282f779c1477d920f24dc0dc16628: Trying to abort 1 pending ops.
I20250901 01:52:35.821080 2118 pending_rounds.cc:77] T f195b040a6ae4b7b83a8308ab0a12b37 P 00c282f779c1477d920f24dc0dc16628: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7195408606088716288 op_type: NO_OP noop_request { }
I20250901 01:52:35.821902 2118 raft_consensus.cc:2887] T f195b040a6ae4b7b83a8308ab0a12b37 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250901 01:52:35.822633 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_eab6381f55fc4b7cb2565669779f45a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.822463 2118 raft_consensus.cc:2270] T f195b040a6ae4b7b83a8308ab0a12b37 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:35.823697 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_34d59546288e43f9a2910bc22efa5435"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.825322 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9555356e26bf405ba7e2a5a14a36ffaa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.827482 2878 raft_consensus.cc:491] T 3dae00aa38204ee3ac77325eb334be42 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:35.827627 2878 raft_consensus.cc:513] T 3dae00aa38204ee3ac77325eb334be42 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:35.828200 2878 leader_election.cc:290] T 3dae00aa38204ee3ac77325eb334be42 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:35.828608 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2273b7c15b9048fc8d204571b5eacc99"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:35.829787 992 leader_election.cc:343] T 50472a7aa2f04e0fbd65ac6c86c34b3e P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 50472a7aa2f04e0fbd65ac6c86c34b3e
I20250901 01:52:35.838029 2878 raft_consensus.cc:491] T d4c9aa723aee4dc4b6237cb5c3722b52 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:35.838176 2878 raft_consensus.cc:513] T d4c9aa723aee4dc4b6237cb5c3722b52 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:35.838784 2878 leader_election.cc:290] T d4c9aa723aee4dc4b6237cb5c3722b52 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:35.841037 2118 ts_tablet_manager.cc:1905] T f195b040a6ae4b7b83a8308ab0a12b37 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:35.844650 992 consensus_peers.cc:489] T e1c8fe96a6ba4edba4f09c00f7add75e P fd27ab6665304aaea53adbd102dd4f58 -> Peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Couldn't send request to peer 00c282f779c1477d920f24dc0dc16628. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e1c8fe96a6ba4edba4f09c00f7add75e. This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:35.847720 2118 ts_tablet_manager.cc:1918] T f195b040a6ae4b7b83a8308ab0a12b37 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:35.847837 2118 log.cc:1199] T f195b040a6ae4b7b83a8308ab0a12b37 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/f195b040a6ae4b7b83a8308ab0a12b37
I20250901 01:52:35.848201 2118 ts_tablet_manager.cc:1939] T f195b040a6ae4b7b83a8308ab0a12b37 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:35.848996 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 270) took 5221 ms. Trace:
I20250901 01:52:35.851464 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d82e3f2e580044bb9c092b6a00afdd3e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.852272 1052 raft_consensus.cc:2466] T d4c9aa723aee4dc4b6237cb5c3722b52 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00c282f779c1477d920f24dc0dc16628 in term 0.
I20250901 01:52:35.853192 2118 rpcz_store.cc:276] 0901 01:52:30.627655 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.627703 (+ 48us) service_pool.cc:225] Handling call
0901 01:52:35.818386 (+5190683us) ts_tablet_manager.cc:1145] Deleting tablet f195b040a6ae4b7b83a8308ab0a12b37
0901 01:52:35.818401 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.844730 (+ 26329us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:35.847647 (+ 2917us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:35.848968 (+ 1321us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5190352}
I20250901 01:52:35.853379 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1efa89ece1464866b2ffc8ced417dc9b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.853556 2118 tablet_replica.cc:331] T abf6e3be60f74872b2308dc95bf9cd37 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:35.853715 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet f195b040a6ae4b7b83a8308ab0a12b37 (table default.Table_8b1086c9a4c44239851c586ea7267e3c [id=523de440fbab4bf687c0f6168e48513c]) successfully deleted
I20250901 01:52:35.853775 2118 raft_consensus.cc:2241] T abf6e3be60f74872b2308dc95bf9cd37 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:35.853946 2118 raft_consensus.cc:2270] T abf6e3be60f74872b2308dc95bf9cd37 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:35.854518 2118 ts_tablet_manager.cc:1905] T abf6e3be60f74872b2308dc95bf9cd37 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:35.854766 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_cc3c2da23aa84fc9805083ce841bffa2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.856709 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7f1efcb90ee7455ea8bf164b94de7b11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.860126 2118 ts_tablet_manager.cc:1918] T abf6e3be60f74872b2308dc95bf9cd37 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:35.860232 2118 log.cc:1199] T abf6e3be60f74872b2308dc95bf9cd37 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/abf6e3be60f74872b2308dc95bf9cd37
I20250901 01:52:35.861420 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_16092a6d140a4ffc91c554932dab3cb2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.861860 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c843f91e706f431ab758acda809d88d4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.861974 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e153d58f284f4345aa8f3096738c363a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.862826 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5fb77057f89144848a8a2cecbaecd47d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.865650 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_db299fd5d9424753aef4419b97a704c1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.866439 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a96ea83e2d9b4b81993e0725a7973545"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:35.853919 992 leader_election.cc:343] T 7f46e9ddadaf4ff986d8c991178daa0c P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 7f46e9ddadaf4ff986d8c991178daa0c
I20250901 01:52:35.867298 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_07be754daa014dd48b29a685a0dd50bc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.867573 2118 ts_tablet_manager.cc:1939] T abf6e3be60f74872b2308dc95bf9cd37 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:35.867744 2613 consensus_queue.cc:1035] T 2973b59bb1df41d18d9a5b6a29ae336f P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:35.864537 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b5b62e7b7dc649d1bc48a53a83bf670b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.868978 2613 consensus_queue.cc:1035] T b2b172757e50404aae78e818921a7d9d P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:35.869773 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 271) took 5171 ms. Trace:
I20250901 01:52:35.869879 2118 rpcz_store.cc:276] 0901 01:52:30.698573 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.698624 (+ 51us) service_pool.cc:225] Handling call
0901 01:52:35.853495 (+5154871us) ts_tablet_manager.cc:1145] Deleting tablet abf6e3be60f74872b2308dc95bf9cd37
0901 01:52:35.853507 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.857479 (+ 3972us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:35.860036 (+ 2557us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:35.869753 (+ 9717us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5154673}
I20250901 01:52:35.869992 2613 consensus_queue.cc:1035] T 81ce448b148447ddbc41875d863e0518 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:35.870226 2118 tablet_replica.cc:331] T 3618a3eb4f174c43805ce7dc0e5315cb P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:35.870388 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet abf6e3be60f74872b2308dc95bf9cd37 (table default.Table_fea398de1f1948809aebf95c387f4b17 [id=6975ac6754db4664a168e25ad61db164]) successfully deleted
I20250901 01:52:35.870433 2118 raft_consensus.cc:2241] T 3618a3eb4f174c43805ce7dc0e5315cb P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
E20250901 01:52:35.870472 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 494b6488aa0d4355b4caa46004267465, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7c21b68ef2e74136a807c1dd90e17f4e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.870731 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 01deaae6abb141789f0cd240f3058c73, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7baa92b46d98454a891222be49a6ca48'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.870903 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 77911bcb98cf4d8890118eca9136f86e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_db61e055dbd7450898e82c4c0c4763be'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.871070 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae4844183a3c4a04a775e6fd878391a3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_4bd66ab99a454974aa0f7f4290edfda2'. Need at least 3 replicas, but only 2 tablet servers are available
W20250901 01:52:35.871066 992 leader_election.cc:336] T 2dacb608897d47d28512df87dc320927 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
E20250901 01:52:35.871301 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9745582cebfe47b58c21d0191ab4cb9b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77728af9eee44f29a1e2131d5b75b7ca'. Need at least 3 replicas, but only 2 tablet servers are available
W20250901 01:52:35.871322 992 consensus_peers.cc:489] T aec951f2cc84499b9162b9fa24d5440e P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
E20250901 01:52:35.871529 1696 catalog_manager.cc:6205] Error selecting replicas for tablet e2ed32b3292141278af28026528e96c2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b5077f55951d48be92a67407ffe26a08'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.871681 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9f62a9d97f514747b940b488907369f3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d53a456dbca64daf92aeb5fb4517b417'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.871690 2753 consensus_queue.cc:1035] T 508406601c994e42a5c5b1f665e74957 P 00c282f779c1477d920f24dc0dc16628 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:35.861091 1151 tablet_bootstrap.cc:492] T 88ceba7f48b04d7885cc0164668700a5 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:35.871973 1151 ts_tablet_manager.cc:1397] T 88ceba7f48b04d7885cc0164668700a5 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.128s user 0.009s sys 0.020s
E20250901 01:52:35.872090 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5a7fa68d29644ae8afd2711e39dde59a, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d24e3a18ef4843fdb7b74ce23d47cf9e'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.872073 2753 consensus_queue.cc:1035] T d0c3c363dda444519a17dca63ad9caa7 P 00c282f779c1477d920f24dc0dc16628 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
E20250901 01:52:35.872318 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 415c9d7b552b42c384b444a20491f65c, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_f709774f621c483b8ee257273a1e3e38'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.872488 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ff2bd017417c41b5b8225b8f00283e63, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0c7355975d6b44dbbe3a77df942b4d43'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.872639 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2821bf7750ba4e38b25ccba2e2bc0eab, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ce2f3b515c8f4c1bb557c5ed84b48dba'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.872813 1696 catalog_manager.cc:6205] Error selecting replicas for tablet dcc60bb000f440f1aca9133928b5ce20, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2fbca8463d6546c9b8f5b018fc60251d'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.872763 1151 raft_consensus.cc:357] T 88ceba7f48b04d7885cc0164668700a5 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:35.872965 1151 raft_consensus.cc:383] T 88ceba7f48b04d7885cc0164668700a5 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:35.873036 1151 raft_consensus.cc:738] T 88ceba7f48b04d7885cc0164668700a5 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
E20250901 01:52:35.873063 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 465c02a130c54e09b5a74fb4dd2ef762, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ed6fd284be6f4ae3a008e21e7ce17930'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.873289 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 75883d9fd00e429aa8853903a9fdd7e4, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_a9f6ac8609554a38ab632b2a3f7113d8'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.873255 1151 consensus_queue.cc:260] T 88ceba7f48b04d7885cc0164668700a5 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
E20250901 01:52:35.873520 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 8f4f5563df6b4d0db890747ba032ef9f, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_eaa0b271683343998d39a8ca0f151587'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.873688 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f8f923ddcd2740bcb7e521f4f57c4034, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_e09ac1a9050547038400d4b52299d22f'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.873839 1696 catalog_manager.cc:6205] Error selecting replicas for tablet b1ba0e19dc0b462e80c537bf2d183f5b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_66b111271c15410aab23682f8f6081ac'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.874009 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 8e10a01e77d14cf28c2c5e0205e595b9, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_83cf2d86a1284544ba2e7aa9ebb66bb7'. Need at least 3 replicas, but only 2 tablet servers are available
W20250901 01:52:35.871415 992 leader_election.cc:336] T ae21b9ae2e3b472eb67fb5f96388a50f P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
E20250901 01:52:35.874221 1696 catalog_manager.cc:6205] Error selecting replicas for tablet fd2ce82a4ee24d0faa44980a3967e187, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_63d0f2ead9d946d39ce40162ebd769d6'. Need at least 3 replicas, but only 2 tablet servers are available
W20250901 01:52:35.874289 992 leader_election.cc:336] T 088ef4861dc14f81ae470c3aaee5b4e1 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
E20250901 01:52:35.874439 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae9a4ad4fdfb4771ba23a91f7007447e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_4cb98468d9f44e16aab412c39b234c15'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.874588 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 086aa948f93644da8e5835b0fac87ddd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b4fbc34da806475c912052d0fab3fc53'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.874389 992 leader_election.cc:304] T 088ef4861dc14f81ae470c3aaee5b4e1 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
E20250901 01:52:35.874958 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 082db24bb3fd4089950504145a20a075, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_4473168010a84ac7a014b99dd10e35a9'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.875180 1696 catalog_manager.cc:6205] Error selecting replicas for tablet fefbf5ae030446b292d8068e95577534, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_3dde393d3a564e84b8ef08c0d25265ce'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.875384 2613 raft_consensus.cc:2747] T 088ef4861dc14f81ae470c3aaee5b4e1 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:35.875682 992 consensus_peers.cc:489] T e1c8fe96a6ba4edba4f09c00f7add75e P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:35.876545 1151 ts_tablet_manager.cc:1428] T 88ceba7f48b04d7885cc0164668700a5 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250901 01:52:35.876389 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a9d1613e93b8427997dd682a6dc1dce6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.877353 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_157df3c286494de3a70166c330d436c7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.877665 1151 tablet_bootstrap.cc:492] T b8ca4e66c6a74f9d8d90a69be5d6a58f P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:35.878594 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_0bac84bc5bee45c9b11017e9d0ab477b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.878419 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e453ce743d974178bcd84554bb57129c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:35.880584 992 leader_election.cc:336] T 50472a7aa2f04e0fbd65ac6c86c34b3e P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:35.880538 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_43b56717caeb494c84a2807dea677b2f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.880687 992 leader_election.cc:304] T 50472a7aa2f04e0fbd65ac6c86c34b3e P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:35.880872 2613 raft_consensus.cc:2747] T 50472a7aa2f04e0fbd65ac6c86c34b3e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:35.882400 918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2dacb608897d47d28512df87dc320927" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:35.882571 918 raft_consensus.cc:3058] T 2dacb608897d47d28512df87dc320927 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:35.882784 921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "44af4764d4f64e44aec84ccb84be6f5f" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:35.882946 921 raft_consensus.cc:2466] T 44af4764d4f64e44aec84ccb84be6f5f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:35.883466 917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ae21b9ae2e3b472eb67fb5f96388a50f" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:35.883615 917 raft_consensus.cc:3058] T ae21b9ae2e3b472eb67fb5f96388a50f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:35.883950 2118 raft_consensus.cc:2270] T 3618a3eb4f174c43805ce7dc0e5315cb P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:35.884559 2118 ts_tablet_manager.cc:1905] T 3618a3eb4f174c43805ce7dc0e5315cb P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:35.885095 918 raft_consensus.cc:2466] T 2dacb608897d47d28512df87dc320927 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:35.885757 1151 tablet_bootstrap.cc:654] T b8ca4e66c6a74f9d8d90a69be5d6a58f P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:35.885746 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_0c9736a8ee724c079bd75430985cb34b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.886149 917 raft_consensus.cc:2466] T ae21b9ae2e3b472eb67fb5f96388a50f P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:35.886413 2113 ts_tablet_manager.cc:1918] T 44af4764d4f64e44aec84ccb84be6f5f P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:35.886502 2113 log.cc:1199] T 44af4764d4f64e44aec84ccb84be6f5f P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/44af4764d4f64e44aec84ccb84be6f5f
I20250901 01:52:35.886797 2113 ts_tablet_manager.cc:1939] T 44af4764d4f64e44aec84ccb84be6f5f P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
E20250901 01:52:35.886982 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 93aa22afc0ad4d7fa24807ea3c5d5c21, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0f4b4094d2974b7591875375eda6bed0'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.887089 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9074543dea9949e7a085717b40913348, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_53e30205c5fd488eb3b48c685a159d04'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.887203 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 403) took 3223 ms. Trace:
I20250901 01:52:35.887260 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3dae00aa38204ee3ac77325eb334be42" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:35.887274 2113 rpcz_store.cc:276] 0901 01:52:32.663315 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:32.663361 (+ 46us) service_pool.cc:225] Handling call
0901 01:52:35.799185 (+3135824us) ts_tablet_manager.cc:1145] Deleting tablet 44af4764d4f64e44aec84ccb84be6f5f
0901 01:52:35.799196 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.832502 (+ 33306us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:35.886310 (+ 53808us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:35.887187 (+ 877us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4798,"tablet-delete.queue_time_us":3135634}
I20250901 01:52:35.887392 916 raft_consensus.cc:2466] T 3dae00aa38204ee3ac77325eb334be42 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:35.888680 1055 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c83ce95fd6fa41478de0e2168be201b1" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:35.889019 2916 tablet.cc:1722] T 6c0abbfc8d904dce989ce51961e4237d P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250901 01:52:35.887763 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 44af4764d4f64e44aec84ccb84be6f5f (table default.Table_9896b980158948339a0b91f5882ed56b [id=a541b55be0dc420a8353ef4793b925b5]) successfully deleted
I20250901 01:52:35.888034 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ca90343ccc124424a56117b0d84db354"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.888556 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e144ab1fbc6c4b9e97d41cf4103531ab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.887648 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 406) took 3099 ms. Trace:
I20250901 01:52:35.898399 2113 rpcz_store.cc:276] 0901 01:52:32.788020 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:32.788188 (+ 168us) service_pool.cc:225] Handling call
0901 01:52:35.887545 (+3099357us) ts_tablet_manager.cc:1145] Deleting tablet 4902062977ea42149e3466e999117322
0901 01:52:35.887554 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.887634 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3099151}
W20250901 01:52:35.899185 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 4902062977ea42149e3466e999117322 because the tablet was not found. No further retry: Not found: Tablet not found: 4902062977ea42149e3466e999117322
I20250901 01:52:35.899724 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 407) took 3111 ms. Trace:
I20250901 01:52:35.900027 859 leader_election.cc:304] T d4c9aa723aee4dc4b6237cb5c3722b52 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
W20250901 01:52:35.900214 859 raft_consensus.cc:2684] T d4c9aa723aee4dc4b6237cb5c3722b52 P 00c282f779c1477d920f24dc0dc16628: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250901 01:52:35.900218 2909 tablet.cc:1722] T 6c0abbfc8d904dce989ce51961e4237d P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250901 01:52:35.900568 2113 rpcz_store.cc:276] 0901 01:52:32.788137 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:32.788441 (+ 304us) service_pool.cc:225] Handling call
0901 01:52:35.899625 (+3111184us) ts_tablet_manager.cc:1145] Deleting tablet 48a211e174054866b030cdfb0cf947f4
0901 01:52:35.899638 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.899705 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3111035}
W20250901 01:52:35.900951 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 48a211e174054866b030cdfb0cf947f4 because the tablet was not found. No further retry: Not found: Tablet not found: 48a211e174054866b030cdfb0cf947f4
E20250901 01:52:35.903118 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 3a62034e688348ed9e2ac9de009ed3a2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_e225c82a2d6045f6b16915cdb72f5734'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.903373 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 18498c9cb09141d08c1c5c0854b63ebd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77f4810bc83c4d948531da2a45a3792e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.903563 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f5e3316adeae47cd848833dd84b972bb, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_fc962c8bc653498e85cc0bb41639e9ad'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.903726 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 6ce9521b925b4f97bb4a810ce77e147e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_bd74ae506bbe40489259a7be70c3d10b'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.904677 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 405) took 3116 ms. Trace:
I20250901 01:52:35.905064 2113 rpcz_store.cc:276] 0901 01:52:32.787918 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:32.788739 (+ 821us) service_pool.cc:225] Handling call
0901 01:52:35.904587 (+3115848us) ts_tablet_manager.cc:1145] Deleting tablet 31614db4c53a4f3c96fd190d326ce601
0901 01:52:35.904598 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.904660 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3115715}
E20250901 01:52:35.905757 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2f61edb50d7a42198bfd70639c7bf4de, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_fe79329825c54cdf8c97a14aa8a30384'. Need at least 3 replicas, but only 2 tablet servers are available
W20250901 01:52:35.905838 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 31614db4c53a4f3c96fd190d326ce601 because the tablet was not found. No further retry: Not found: Tablet not found: 31614db4c53a4f3c96fd190d326ce601
W20250901 01:52:35.906077 992 consensus_peers.cc:489] T 2973b59bb1df41d18d9a5b6a29ae336f P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250901 01:52:35.907012 992 consensus_peers.cc:489] T f65ec84958eb488d8b0f812b59b5b3d5 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20250901 01:52:35.907510 992 leader_election.cc:336] T 7f46e9ddadaf4ff986d8c991178daa0c P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:35.907873 992 leader_election.cc:304] T 7f46e9ddadaf4ff986d8c991178daa0c P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:35.908434 2613 raft_consensus.cc:2747] T 7f46e9ddadaf4ff986d8c991178daa0c P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:35.906138 2113 tablet_replica.cc:331] T 410d44e5f1974ed69e5dee65e98856a4 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:35.909912 2113 raft_consensus.cc:2241] T 410d44e5f1974ed69e5dee65e98856a4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:35.910075 2113 raft_consensus.cc:2270] T 410d44e5f1974ed69e5dee65e98856a4 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:35.913353 2113 ts_tablet_manager.cc:1905] T 410d44e5f1974ed69e5dee65e98856a4 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:35.914495 1492 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_e5753556ca8b4720a2e3cb3c5f99c27f" } new_table_name: "default.Table_655ebd0a95a745c2a952006ada10fc75" modify_external_catalogs: true
W20250901 01:52:35.913496 992 consensus_peers.cc:489] T 81ce448b148447ddbc41875d863e0518 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:35.913574 1151 tablet_bootstrap.cc:492] T b8ca4e66c6a74f9d8d90a69be5d6a58f P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:35.915400 1151 ts_tablet_manager.cc:1397] T b8ca4e66c6a74f9d8d90a69be5d6a58f P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.038s user 0.002s sys 0.004s
I20250901 01:52:35.916141 1151 raft_consensus.cc:357] T b8ca4e66c6a74f9d8d90a69be5d6a58f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:35.916344 1151 raft_consensus.cc:383] T b8ca4e66c6a74f9d8d90a69be5d6a58f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:35.916457 1151 raft_consensus.cc:738] T b8ca4e66c6a74f9d8d90a69be5d6a58f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:35.916693 1151 consensus_queue.cc:260] T b8ca4e66c6a74f9d8d90a69be5d6a58f P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:35.917358 1151 ts_tablet_manager.cc:1428] T b8ca4e66c6a74f9d8d90a69be5d6a58f P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250901 01:52:35.917811 1151 tablet_bootstrap.cc:492] T eee34044c39d4013baf78d3705489626 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:35.918196 1483 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_4e6fae563c5843f4953527abbe862adc" } new_table_name: "default.Table_a47ce7aa50f34c2a9c204a5f8376e6ee" modify_external_catalogs: true
I20250901 01:52:35.919325 1151 tablet_bootstrap.cc:654] T eee34044c39d4013baf78d3705489626 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:35.921916 1492 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5d68ea97c71942c7b55b967ba870e0d7 consensus state has no leader
I20250901 01:52:35.923007 2118 ts_tablet_manager.cc:1918] T 3618a3eb4f174c43805ce7dc0e5315cb P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250901 01:52:35.923053 992 leader_election.cc:304] T 44af4764d4f64e44aec84ccb84be6f5f P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
W20250901 01:52:35.923234 992 raft_consensus.cc:2684] T 44af4764d4f64e44aec84ccb84be6f5f P fd27ab6665304aaea53adbd102dd4f58: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250901 01:52:35.923707 992 leader_election.cc:304] T 2dacb608897d47d28512df87dc320927 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:35.923908 2118 log.cc:1199] T 3618a3eb4f174c43805ce7dc0e5315cb P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/3618a3eb4f174c43805ce7dc0e5315cb
I20250901 01:52:35.924770 2118 ts_tablet_manager.cc:1939] T 3618a3eb4f174c43805ce7dc0e5315cb P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:35.925436 2780 raft_consensus.cc:2802] T 2dacb608897d47d28512df87dc320927 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:35.925590 2780 raft_consensus.cc:695] T 2dacb608897d47d28512df87dc320927 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:35.925849 2780 consensus_queue.cc:237] T 2dacb608897d47d28512df87dc320927 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:35.926800 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 272) took 5085 ms. Trace:
E20250901 01:52:35.927256 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 494b6488aa0d4355b4caa46004267465, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7c21b68ef2e74136a807c1dd90e17f4e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.927448 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 01deaae6abb141789f0cd240f3058c73, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7baa92b46d98454a891222be49a6ca48'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.927572 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 77911bcb98cf4d8890118eca9136f86e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_db61e055dbd7450898e82c4c0c4763be'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.927722 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae4844183a3c4a04a775e6fd878391a3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_4bd66ab99a454974aa0f7f4290edfda2'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.927827 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9745582cebfe47b58c21d0191ab4cb9b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77728af9eee44f29a1e2131d5b75b7ca'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.928731 1696 catalog_manager.cc:6205] Error selecting replicas for tablet e2ed32b3292141278af28026528e96c2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b5077f55951d48be92a67407ffe26a08'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.928992 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9f62a9d97f514747b940b488907369f3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d53a456dbca64daf92aeb5fb4517b417'. Need at least 3 replicas, but only 2 tablet servers are available
W20250901 01:52:35.928210 859 leader_election.cc:343] T c83ce95fd6fa41478de0e2168be201b1 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:35.932004 859 leader_election.cc:304] T c83ce95fd6fa41478de0e2168be201b1 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
E20250901 01:52:35.932346 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5a7fa68d29644ae8afd2711e39dde59a, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d24e3a18ef4843fdb7b74ce23d47cf9e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.932546 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 415c9d7b552b42c384b444a20491f65c, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_f709774f621c483b8ee257273a1e3e38'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.931947 2118 rpcz_store.cc:276] 0901 01:52:30.841477 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.841545 (+ 68us) service_pool.cc:225] Handling call
0901 01:52:35.870180 (+5028635us) ts_tablet_manager.cc:1145] Deleting tablet 3618a3eb4f174c43805ce7dc0e5315cb
0901 01:52:35.870189 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.897724 (+ 27535us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:35.922895 (+ 25171us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:35.926781 (+ 3886us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5028415}
I20250901 01:52:35.932852 2357 raft_consensus.cc:2747] T c83ce95fd6fa41478de0e2168be201b1 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:35.932965 2118 tablet_replica.cc:331] T 15e4d972e8254cf899c3e0b71b6e2bd9 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:35.933161 2118 raft_consensus.cc:2241] T 15e4d972e8254cf899c3e0b71b6e2bd9 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:35.933355 2118 raft_consensus.cc:2270] T 15e4d972e8254cf899c3e0b71b6e2bd9 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:35.935664 992 leader_election.cc:304] T ae21b9ae2e3b472eb67fb5f96388a50f P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:35.935951 2613 raft_consensus.cc:2802] T ae21b9ae2e3b472eb67fb5f96388a50f P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:35.936079 2613 raft_consensus.cc:695] T ae21b9ae2e3b472eb67fb5f96388a50f P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:35.936303 2613 consensus_queue.cc:237] T ae21b9ae2e3b472eb67fb5f96388a50f P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:35.936631 2118 ts_tablet_manager.cc:1905] T 15e4d972e8254cf899c3e0b71b6e2bd9 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:35.938203 992 leader_election.cc:304] T 3dae00aa38204ee3ac77325eb334be42 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:35.938366 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 3618a3eb4f174c43805ce7dc0e5315cb (table default.Table_d24e3a18ef4843fdb7b74ce23d47cf9e [id=31a2e50c32644321825da40176127015]) successfully deleted
W20250901 01:52:35.939155 992 leader_election.cc:343] T d4c9aa723aee4dc4b6237cb5c3722b52 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: d4c9aa723aee4dc4b6237cb5c3722b52
I20250901 01:52:35.939117 1485 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_d1df9c0ae1a04b21a133e93cf4ae363a" } new_table_name: "default.Table_a2e84eb190f440afbb9dff19c87fda4c" modify_external_catalogs: true
I20250901 01:52:35.938360 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_6e6221ab678945c19ab6ae10609a193c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.943205 2118 ts_tablet_manager.cc:1918] T 15e4d972e8254cf899c3e0b71b6e2bd9 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:35.943595 2856 raft_consensus.cc:2802] T 3dae00aa38204ee3ac77325eb334be42 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:35.943702 2856 raft_consensus.cc:491] T 3dae00aa38204ee3ac77325eb334be42 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:35.943783 2856 raft_consensus.cc:3058] T 3dae00aa38204ee3ac77325eb334be42 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:35.946684 2856 raft_consensus.cc:513] T 3dae00aa38204ee3ac77325eb334be42 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:35.947233 2856 leader_election.cc:290] T 3dae00aa38204ee3ac77325eb334be42 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:35.947854 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_40abace9f3f04060b88907a0ddbb6259"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.947851 1486 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_91214af783ff4aa3b6c3cbdb0358307e" } new_table_name: "default.Table_9b9afa82d7de43048ee139b228df93a5" modify_external_catalogs: true
I20250901 01:52:35.948827 1489 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_359a26011f684089ac6da2a62e032680" } modify_external_catalogs: true
I20250901 01:52:35.948953 1491 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_ed6fd284be6f4ae3a008e21e7ce17930" } new_table_name: "default.Table_bfddafe8e3204ce8bc9dd5934343ec59" modify_external_catalogs: true
I20250901 01:52:35.949035 1489 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_359a26011f684089ac6da2a62e032680" } modify_external_catalogs: true
W20250901 01:52:35.950374 1485 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 45f6e8ba99b74898a2922b36768c6c36 consensus state has no leader
W20250901 01:52:35.951959 992 leader_election.cc:336] T d4c9aa723aee4dc4b6237cb5c3722b52 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:35.952248 992 leader_election.cc:304] T d4c9aa723aee4dc4b6237cb5c3722b52 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:35.952741 2911 raft_consensus.cc:2747] T d4c9aa723aee4dc4b6237cb5c3722b52 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:35.953019 992 leader_election.cc:336] T 3dae00aa38204ee3ac77325eb334be42 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:35.953562 992 leader_election.cc:336] T 2dacb608897d47d28512df87dc320927 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:35.954666 992 leader_election.cc:336] T ae21b9ae2e3b472eb67fb5f96388a50f P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:35.955003 992 consensus_peers.cc:489] T 4b4aa063ffe948e69e47f03102581899 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250901 01:52:35.955355 992 leader_election.cc:336] T 44af4764d4f64e44aec84ccb84be6f5f P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:35.957127 1151 tablet_bootstrap.cc:492] T eee34044c39d4013baf78d3705489626 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:35.957288 1151 ts_tablet_manager.cc:1397] T eee34044c39d4013baf78d3705489626 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.040s user 0.002s sys 0.002s
I20250901 01:52:35.957985 1151 raft_consensus.cc:357] T eee34044c39d4013baf78d3705489626 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:35.958189 1151 raft_consensus.cc:383] T eee34044c39d4013baf78d3705489626 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:35.958269 1151 raft_consensus.cc:738] T eee34044c39d4013baf78d3705489626 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:35.958511 1151 consensus_queue.cc:260] T eee34044c39d4013baf78d3705489626 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
E20250901 01:52:35.947484 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ff2bd017417c41b5b8225b8f00283e63, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0c7355975d6b44dbbe3a77df942b4d43'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.958750 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2821bf7750ba4e38b25ccba2e2bc0eab, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ce2f3b515c8f4c1bb557c5ed84b48dba'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.959177 1696 catalog_manager.cc:6205] Error selecting replicas for tablet dcc60bb000f440f1aca9133928b5ce20, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2fbca8463d6546c9b8f5b018fc60251d'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.959256 1151 ts_tablet_manager.cc:1428] T eee34044c39d4013baf78d3705489626 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
E20250901 01:52:35.959488 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 465c02a130c54e09b5a74fb4dd2ef762, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ed6fd284be6f4ae3a008e21e7ce17930'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.959511 1151 tablet_bootstrap.cc:492] T 75a703c8e20a4a2fa0ab6ec58d41ef92 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
E20250901 01:52:35.959744 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 75883d9fd00e429aa8853903a9fdd7e4, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_a9f6ac8609554a38ab632b2a3f7113d8'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.960031 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 8f4f5563df6b4d0db890747ba032ef9f, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_eaa0b271683343998d39a8ca0f151587'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.960275 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f8f923ddcd2740bcb7e521f4f57c4034, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_e09ac1a9050547038400d4b52299d22f'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.960515 1696 catalog_manager.cc:6205] Error selecting replicas for tablet b1ba0e19dc0b462e80c537bf2d183f5b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_66b111271c15410aab23682f8f6081ac'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.960790 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 8e10a01e77d14cf28c2c5e0205e595b9, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_83cf2d86a1284544ba2e7aa9ebb66bb7'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.961083 1696 catalog_manager.cc:6205] Error selecting replicas for tablet fd2ce82a4ee24d0faa44980a3967e187, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_63d0f2ead9d946d39ce40162ebd769d6'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.961342 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae9a4ad4fdfb4771ba23a91f7007447e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_4cb98468d9f44e16aab412c39b234c15'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.961588 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 086aa948f93644da8e5835b0fac87ddd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b4fbc34da806475c912052d0fab3fc53'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.961799 1151 tablet_bootstrap.cc:654] T 75a703c8e20a4a2fa0ab6ec58d41ef92 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
E20250901 01:52:35.961887 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 082db24bb3fd4089950504145a20a075, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_4473168010a84ac7a014b99dd10e35a9'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.962374 1696 catalog_manager.cc:6205] Error selecting replicas for tablet fefbf5ae030446b292d8068e95577534, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_3dde393d3a564e84b8ef08c0d25265ce'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.962779 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 93aa22afc0ad4d7fa24807ea3c5d5c21, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0f4b4094d2974b7591875375eda6bed0'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.963099 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9074543dea9949e7a085717b40913348, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_53e30205c5fd488eb3b48c685a159d04'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.963397 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 3a62034e688348ed9e2ac9de009ed3a2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_e225c82a2d6045f6b16915cdb72f5734'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.963673 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 18498c9cb09141d08c1c5c0854b63ebd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77f4810bc83c4d948531da2a45a3792e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.963968 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f5e3316adeae47cd848833dd84b972bb, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_fc962c8bc653498e85cc0bb41639e9ad'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.964253 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 6ce9521b925b4f97bb4a810ce77e147e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_bd74ae506bbe40489259a7be70c3d10b'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:35.964540 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2f61edb50d7a42198bfd70639c7bf4de, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_fe79329825c54cdf8c97a14aa8a30384'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:35.962935 2113 ts_tablet_manager.cc:1918] T 410d44e5f1974ed69e5dee65e98856a4 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:35.965490 2909 tablet.cc:1722] T f6a8b425968e49deaecfa3adacfd3f1f P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:35.968593 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b5a6fa1a9b1f4abba06711bf7cd837eb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.970669 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8901bb3c38584a84b3f52d6954c2a976"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:35.972754 2113 log.cc:1199] T 410d44e5f1974ed69e5dee65e98856a4 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/410d44e5f1974ed69e5dee65e98856a4
I20250901 01:52:35.973088 2113 ts_tablet_manager.cc:1939] T 410d44e5f1974ed69e5dee65e98856a4 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
W20250901 01:52:35.974499 992 consensus_peers.cc:489] T b2b172757e50404aae78e818921a7d9d P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:35.955370 2916 tablet.cc:1722] T f6a8b425968e49deaecfa3adacfd3f1f P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:35.963989 918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3dae00aa38204ee3ac77325eb334be42" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:35.975116 918 raft_consensus.cc:3058] T 3dae00aa38204ee3ac77325eb334be42 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:35.975387 2118 log.cc:1199] T 15e4d972e8254cf899c3e0b71b6e2bd9 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/15e4d972e8254cf899c3e0b71b6e2bd9
I20250901 01:52:35.975719 2118 ts_tablet_manager.cc:1939] T 15e4d972e8254cf899c3e0b71b6e2bd9 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:35.977764 918 raft_consensus.cc:2466] T 3dae00aa38204ee3ac77325eb334be42 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:35.980082 2911 raft_consensus.cc:491] T 45f6e8ba99b74898a2922b36768c6c36 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:35.980202 2911 raft_consensus.cc:513] T 45f6e8ba99b74898a2922b36768c6c36 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:35.980744 2911 leader_election.cc:290] T 45f6e8ba99b74898a2922b36768c6c36 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:35.981578 2911 consensus_queue.cc:1035] T 4b4aa063ffe948e69e47f03102581899 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:35.982486 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 411) took 3192 ms. Trace:
I20250901 01:52:35.982635 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 273) took 5140 ms. Trace:
I20250901 01:52:35.982731 2118 rpcz_store.cc:276] 0901 01:52:30.841597 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.841801 (+ 204us) service_pool.cc:225] Handling call
0901 01:52:35.932914 (+5091113us) ts_tablet_manager.cc:1145] Deleting tablet 15e4d972e8254cf899c3e0b71b6e2bd9
0901 01:52:35.932925 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.940218 (+ 7293us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:35.943116 (+ 2898us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:35.982466 (+ 39350us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5090934}
I20250901 01:52:35.983366 2118 tablet_replica.cc:331] T d1f2d98712ff4ea6aec665512de068bd P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:35.983629 2118 raft_consensus.cc:2241] T d1f2d98712ff4ea6aec665512de068bd P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:35.983812 2118 raft_consensus.cc:2270] T d1f2d98712ff4ea6aec665512de068bd P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:35.984465 2118 ts_tablet_manager.cc:1905] T d1f2d98712ff4ea6aec665512de068bd P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:35.985159 2909 tablet.cc:1722] T 615b2306fca84003b264da9d5f95ddab P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 7
I20250901 01:52:35.990729 2113 rpcz_store.cc:276] 0901 01:52:32.789663 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:32.789725 (+ 62us) service_pool.cc:225] Handling call
0901 01:52:35.906091 (+3116366us) ts_tablet_manager.cc:1145] Deleting tablet 410d44e5f1974ed69e5dee65e98856a4
0901 01:52:35.906101 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.940844 (+ 34743us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:35.962835 (+ 21991us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:35.982465 (+ 19630us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3116058}
I20250901 01:52:35.992456 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 410d44e5f1974ed69e5dee65e98856a4 (table default.Table_bec114a616504798bede9a9066e57192 [id=ae79f2c40fd7483ea400a36cf3c48888]) successfully deleted
I20250901 01:52:35.991230 992 leader_election.cc:304] T 3dae00aa38204ee3ac77325eb334be42 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:35.993367 2911 raft_consensus.cc:2802] T 3dae00aa38204ee3ac77325eb334be42 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:35.993498 2911 raft_consensus.cc:695] T 3dae00aa38204ee3ac77325eb334be42 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:35.993747 2911 consensus_queue.cc:237] T 3dae00aa38204ee3ac77325eb334be42 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:36.004334 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 409) took 3214 ms. Trace:
I20250901 01:52:36.005048 2909 tablet.cc:1722] T eb19e3a424324a24824f6caaf5848952 P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:36.005736 2113 rpcz_store.cc:276] 0901 01:52:32.789470 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:32.789757 (+ 287us) service_pool.cc:225] Handling call
0901 01:52:35.993017 (+3203260us) ts_tablet_manager.cc:1145] Deleting tablet 56475ba491a24301a534304811f30140
0901 01:52:35.993031 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:36.004306 (+ 11275us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3202089}
I20250901 01:52:36.007584 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_abbb9acf0e804bb8831abd3b4e4028f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.010593 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_970975ef2a52446e9ee708c94cd10ad3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.012455 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_02120a08d8d94d219505b296b09d66a7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.016337 1485 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_3d5afc22c4e14d6990aea31564dfde20" } modify_external_catalogs: true
W20250901 01:52:36.017503 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 56475ba491a24301a534304811f30140 because the tablet was not found. No further retry: Not found: Tablet not found: 56475ba491a24301a534304811f30140
I20250901 01:52:36.012693 2909 tablet.cc:1722] T b4278a013ae84082bc7a8497b04ac147 P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:36.019079 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 410) took 3229 ms. Trace:
I20250901 01:52:36.021059 2113 rpcz_store.cc:276] 0901 01:52:32.789567 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:32.789819 (+ 252us) service_pool.cc:225] Handling call
0901 01:52:36.018963 (+3229144us) ts_tablet_manager.cc:1145] Deleting tablet b81a12b38ac64efabd44aeb192d181a0
0901 01:52:36.018975 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:36.019058 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3227923}
I20250901 01:52:36.022336 919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45f6e8ba99b74898a2922b36768c6c36" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:36.022538 919 raft_consensus.cc:2466] T 45f6e8ba99b74898a2922b36768c6c36 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
W20250901 01:52:36.023110 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet b81a12b38ac64efabd44aeb192d181a0 because the tablet was not found. No further retry: Not found: Tablet not found: b81a12b38ac64efabd44aeb192d181a0
I20250901 01:52:36.023113 2916 tablet.cc:1722] T d0c3c363dda444519a17dca63ad9caa7 P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250901 01:52:36.023801 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 408) took 3234 ms. Trace:
I20250901 01:52:36.024928 2118 ts_tablet_manager.cc:1918] T d1f2d98712ff4ea6aec665512de068bd P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:36.025336 2113 rpcz_store.cc:276] 0901 01:52:32.789373 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:32.789902 (+ 529us) service_pool.cc:225] Handling call
0901 01:52:36.023704 (+3233802us) ts_tablet_manager.cc:1145] Deleting tablet 1a60c10e42104502ac2d3b80b8126ceb
0901 01:52:36.023715 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:36.023781 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3232619}
I20250901 01:52:36.026448 2918 tablet.cc:1722] T 615b2306fca84003b264da9d5f95ddab P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 7
I20250901 01:52:36.027004 2118 log.cc:1199] T d1f2d98712ff4ea6aec665512de068bd P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/d1f2d98712ff4ea6aec665512de068bd
I20250901 01:52:36.027539 2118 ts_tablet_manager.cc:1939] T d1f2d98712ff4ea6aec665512de068bd P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
W20250901 01:52:36.028752 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 1a60c10e42104502ac2d3b80b8126ceb because the tablet was not found. No further retry: Not found: Tablet not found: 1a60c10e42104502ac2d3b80b8126ceb
I20250901 01:52:36.028947 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 274) took 5187 ms. Trace:
I20250901 01:52:36.029307 2113 tablet_replica.cc:331] T 61ddcf86c9f34726939cc116434f0822 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:36.030776 2918 tablet.cc:1722] T eb19e3a424324a24824f6caaf5848952 P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250901 01:52:36.032984 2942 log.cc:927] Time spent T b2b172757e50404aae78e818921a7d9d P 00c282f779c1477d920f24dc0dc16628: Append to log took a long time: real 0.060s user 0.001s sys 0.000s
I20250901 01:52:36.031061 2118 rpcz_store.cc:276] 0901 01:52:30.841685 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.842006 (+ 321us) service_pool.cc:225] Handling call
0901 01:52:35.983320 (+5141314us) ts_tablet_manager.cc:1145] Deleting tablet d1f2d98712ff4ea6aec665512de068bd
0901 01:52:35.983330 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:35.999165 (+ 15835us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:36.024821 (+ 25656us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:36.028931 (+ 4110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5141131}
I20250901 01:52:36.034195 2118 tablet_replica.cc:331] T 5d794471dbed4477b9d52fd411fa4d10 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:36.034525 2118 raft_consensus.cc:2241] T 5d794471dbed4477b9d52fd411fa4d10 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:36.037758 2118 raft_consensus.cc:2270] T 5d794471dbed4477b9d52fd411fa4d10 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:36.038426 2118 ts_tablet_manager.cc:1905] T 5d794471dbed4477b9d52fd411fa4d10 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:36.039530 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b28a6ccdc80844df914cf7a82b2d6617"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.043471 2956 tablet.cc:1722] T b4278a013ae84082bc7a8497b04ac147 P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:36.050326 1489 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet dabebc77f72e4bb19e264be2f496202a
W20250901 01:52:36.050531 1489 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet dabebc77f72e4bb19e264be2f496202a: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:36.051476 893 tablet_service.cc:1515] Processing DeleteTablet for tablet dabebc77f72e4bb19e264be2f496202a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:35 UTC) from {username='slave'} at 127.0.0.1:34472
W20250901 01:52:36.052050 1491 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 465c02a130c54e09b5a74fb4dd2ef762 has no consensus state
I20250901 01:52:36.053016 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet dabebc77f72e4bb19e264be2f496202a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:35 UTC) from {username='slave'} at 127.0.0.1:35946
E20250901 01:52:36.053063 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 494b6488aa0d4355b4caa46004267465, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7c21b68ef2e74136a807c1dd90e17f4e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.053179 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 01deaae6abb141789f0cd240f3058c73, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7baa92b46d98454a891222be49a6ca48'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.053241 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 77911bcb98cf4d8890118eca9136f86e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_db61e055dbd7450898e82c4c0c4763be'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:36.053296 1483 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 2973b59bb1df41d18d9a5b6a29ae336f (table default.Table_3c2306df46ec4c14b64b5fbd9d21eb98 [id=746845bd84c6444b8a1e5683eaba4a68]). Expected version 2 got 0
I20250901 01:52:36.053413 1485 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_3d5afc22c4e14d6990aea31564dfde20" } modify_external_catalogs: true
W20250901 01:52:36.054826 1486 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2dacb608897d47d28512df87dc320927 consensus state has no leader
I20250901 01:52:36.057726 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_cd098cb98a9e44cfba46625ead2d144b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.065485 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_61afae1d44054e919d3a563655cacd91"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.068967 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9a0c98867f2840c2a160006e3a01af29"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.068382 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_6f4bed72c50b4e5bbf5b6fce7c7b6cb9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.068564 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_70a00fa7d9bc41dc972038f0bd808827"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.071146 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a8824af483234e08963c0746c674c363"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.074626 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b25ebdacd3bc4d5aa38535545c05a1cc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.075472 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_284d33792f964d3ba7a6f4430ffe446e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.077019 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e6ed7cee20b043339ebfe231d6a9a9dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.077561 1484 master_service.cc:948] ReplaceTablet: received request to replace tablet 3dae00aa38204ee3ac77325eb334be42 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:36.078217 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_dc1f825569cb45c58f0210af8ee253c2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.078979 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_272a6eb786fa467fb2ee6dcfe438a856"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.079701 2118 ts_tablet_manager.cc:1918] T 5d794471dbed4477b9d52fd411fa4d10 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:36.079828 2118 log.cc:1199] T 5d794471dbed4477b9d52fd411fa4d10 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/5d794471dbed4477b9d52fd411fa4d10
I20250901 01:52:36.080482 1491 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_f709774f621c483b8ee257273a1e3e38" } modify_external_catalogs: true
I20250901 01:52:36.080610 1491 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_f709774f621c483b8ee257273a1e3e38" } modify_external_catalogs: true
I20250901 01:52:36.081275 2118 ts_tablet_manager.cc:1939] T 5d794471dbed4477b9d52fd411fa4d10 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:36.081562 1489 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_685b7ca5d6e14076ad261d156b30858b" } modify_external_catalogs: true
I20250901 01:52:36.081687 1489 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_685b7ca5d6e14076ad261d156b30858b" } modify_external_catalogs: true
I20250901 01:52:36.082114 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 275) took 5240 ms. Trace:
I20250901 01:52:36.082193 2118 rpcz_store.cc:276] 0901 01:52:30.841850 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:30.842197 (+ 347us) service_pool.cc:225] Handling call
0901 01:52:36.034136 (+5191939us) ts_tablet_manager.cc:1145] Deleting tablet 5d794471dbed4477b9d52fd411fa4d10
0901 01:52:36.034149 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:36.045219 (+ 11070us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:36.079589 (+ 34370us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:36.082093 (+ 2504us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5191799}
I20250901 01:52:36.082840 2118 tablet_replica.cc:331] T f8d9bbf0c5be462a8ddc539238cbdf97 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:36.083040 2118 raft_consensus.cc:2241] T f8d9bbf0c5be462a8ddc539238cbdf97 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:36.083181 2118 raft_consensus.cc:2270] T f8d9bbf0c5be462a8ddc539238cbdf97 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:36.083694 2118 ts_tablet_manager.cc:1905] T f8d9bbf0c5be462a8ddc539238cbdf97 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:36.084126 1490 master_service.cc:948] ReplaceTablet: received request to replace tablet b13afacd37474781877b9f2cd20b313a from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:36.086140 1483 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_e09ac1a9050547038400d4b52299d22f" } modify_external_catalogs: true
I20250901 01:52:36.086342 1483 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_e09ac1a9050547038400d4b52299d22f" } modify_external_catalogs: true
I20250901 01:52:36.087308 2113 raft_consensus.cc:2241] T 61ddcf86c9f34726939cc116434f0822 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:36.089008 2113 raft_consensus.cc:2270] T 61ddcf86c9f34726939cc116434f0822 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250901 01:52:36.089375 992 consensus_peers.cc:489] T e1d18e999b5e42b1873631e0f1445081 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250901 01:52:36.088138 1151 tablet_bootstrap.cc:492] T 75a703c8e20a4a2fa0ab6ec58d41ef92 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:36.089841 1151 ts_tablet_manager.cc:1397] T 75a703c8e20a4a2fa0ab6ec58d41ef92 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.130s user 0.003s sys 0.003s
I20250901 01:52:36.090899 1151 raft_consensus.cc:357] T 75a703c8e20a4a2fa0ab6ec58d41ef92 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:36.091884 1151 raft_consensus.cc:383] T 75a703c8e20a4a2fa0ab6ec58d41ef92 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:36.092001 1151 raft_consensus.cc:738] T 75a703c8e20a4a2fa0ab6ec58d41ef92 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:36.094751 1485 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 33f637f6a8134215bb1995ec38440073
I20250901 01:52:36.094775 1151 consensus_queue.cc:260] T 75a703c8e20a4a2fa0ab6ec58d41ef92 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
W20250901 01:52:36.094903 1485 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 33f637f6a8134215bb1995ec38440073: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:36.095288 1151 ts_tablet_manager.cc:1428] T 75a703c8e20a4a2fa0ab6ec58d41ef92 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250901 01:52:36.095726 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 33f637f6a8134215bb1995ec38440073 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:36 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:36.096177 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 33f637f6a8134215bb1995ec38440073 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:36 UTC) from {username='slave'} at 127.0.0.1:34472
W20250901 01:52:36.090194 992 leader_election.cc:336] T 3dae00aa38204ee3ac77325eb334be42 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:36.097860 1489 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet e1d18e999b5e42b1873631e0f1445081
I20250901 01:52:36.091794 2113 ts_tablet_manager.cc:1905] T 61ddcf86c9f34726939cc116434f0822 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:36.098048 1489 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet e1d18e999b5e42b1873631e0f1445081: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:36.098970 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet e1d18e999b5e42b1873631e0f1445081 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:36 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:36.099234 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f25be4ef989d43e883ab5f12743c9598"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.099354 2118 ts_tablet_manager.cc:1918] T f8d9bbf0c5be462a8ddc539238cbdf97 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:36.099628 2118 log.cc:1199] T f8d9bbf0c5be462a8ddc539238cbdf97 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/f8d9bbf0c5be462a8ddc539238cbdf97
I20250901 01:52:36.099968 2118 ts_tablet_manager.cc:1939] T f8d9bbf0c5be462a8ddc539238cbdf97 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:36.100088 2954 tablet.cc:1722] T d0c3c363dda444519a17dca63ad9caa7 P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250901 01:52:36.100172 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_41a750d73fde41ac9219b369cade01e3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.101008 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_cd666e0f5f9146129cbb5260066f82a5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.101972 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1d8a3dbcfd4d417e9f3770f532f14df8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.102710 1482 master_service.cc:948] ReplaceTablet: received request to replace tablet aec951f2cc84499b9162b9fa24d5440e from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:36.103358 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e1257a3c56034f67b6826f2867d19067"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.099335 893 tablet_service.cc:1515] Processing DeleteTablet for tablet e1d18e999b5e42b1873631e0f1445081 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:36 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:36.103729 992 leader_election.cc:304] T 45f6e8ba99b74898a2922b36768c6c36 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:36.104003 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8483a346d76b4708b65affc69e726d3d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.104066 2878 raft_consensus.cc:2802] T 45f6e8ba99b74898a2922b36768c6c36 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:36.104137 2878 raft_consensus.cc:491] T 45f6e8ba99b74898a2922b36768c6c36 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:36.104193 2878 raft_consensus.cc:3058] T 45f6e8ba99b74898a2922b36768c6c36 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:36.104431 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 278) took 5099 ms. Trace:
I20250901 01:52:36.104511 2118 rpcz_store.cc:276] 0901 01:52:31.004676 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:31.005015 (+ 339us) service_pool.cc:225] Handling call
0901 01:52:36.082775 (+5077760us) ts_tablet_manager.cc:1145] Deleting tablet f8d9bbf0c5be462a8ddc539238cbdf97
0901 01:52:36.082785 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:36.092449 (+ 9664us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:36.095332 (+ 2883us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:36.104389 (+ 9057us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5077562}
I20250901 01:52:36.104681 1482 master_service.cc:948] ReplaceTablet: received request to replace tablet eb19e3a424324a24824f6caaf5848952 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:36.104928 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet f8d9bbf0c5be462a8ddc539238cbdf97 (table default.Table_dfb6e260190a40df8db77152b962cb8e [id=0e10c4b90e884aaeb0a75dbcf3dfd1bc]) successfully deleted
I20250901 01:52:36.105265 2118 tablet_replica.cc:331] T e87d1281f35648b0b115b67c95e244e7 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:36.105501 2118 raft_consensus.cc:2241] T e87d1281f35648b0b115b67c95e244e7 P 00c282f779c1477d920f24dc0dc16628 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:36.105903 2118 raft_consensus.cc:2270] T e87d1281f35648b0b115b67c95e244e7 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:36.106468 2118 ts_tablet_manager.cc:1905] T e87d1281f35648b0b115b67c95e244e7 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:36.106649 2878 raft_consensus.cc:513] T 45f6e8ba99b74898a2922b36768c6c36 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:36.108139 2878 leader_election.cc:290] T 45f6e8ba99b74898a2922b36768c6c36 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:36.109535 1151 tablet_bootstrap.cc:492] T 3a2bc5ad1cd243daa3c92fceca8ba414 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:36.112190 1482 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet eb19e3a424324a24824f6caaf5848952
W20250901 01:52:36.112519 1482 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet eb19e3a424324a24824f6caaf5848952: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:36.113974 1482 catalog_manager.cc:6652] ReplaceTablet: tablet eb19e3a424324a24824f6caaf5848952 deleted and replaced by tablet 97418fffe6614dfba4ebd13a6ed57e67
I20250901 01:52:36.114001 893 tablet_service.cc:1515] Processing DeleteTablet for tablet eb19e3a424324a24824f6caaf5848952 with delete_type TABLET_DATA_DELETED (replaced by tablet 97418fffe6614dfba4ebd13a6ed57e67) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:36.112802 2780 raft_consensus.cc:491] T e9f1f892658b4736859ff507536920a1 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:36.113258 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet eb19e3a424324a24824f6caaf5848952 with delete_type TABLET_DATA_DELETED (replaced by tablet 97418fffe6614dfba4ebd13a6ed57e67) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:36.114166 2780 raft_consensus.cc:513] T e9f1f892658b4736859ff507536920a1 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:36.114501 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_99e1680b548844adbaa22b8b034eead1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.115197 2780 leader_election.cc:290] T e9f1f892658b4736859ff507536920a1 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:36.115808 1492 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: default.Table_a47ce7aa50f34c2a9c204a5f8376e6ee [id=cecdd8d6ca6d445c9fd82decd2a95366] alter complete (version 7)
I20250901 01:52:36.117344 2954 tablet.cc:1722] T 2973b59bb1df41d18d9a5b6a29ae336f P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250901 01:52:36.117383 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_eb89575e96f54814be208aca0ba2aa54"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.118326 1488 master_service.cc:948] ReplaceTablet: received request to replace tablet 6ba88af177874544b764cc32420e7ac9 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:36.120314 2118 ts_tablet_manager.cc:1918] T e87d1281f35648b0b115b67c95e244e7 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 01:52:36.122830 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4421ebe4eef6488b90438ec66220c88f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.120568 2118 log.cc:1199] T e87d1281f35648b0b115b67c95e244e7 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/e87d1281f35648b0b115b67c95e244e7
I20250901 01:52:36.123734 2118 ts_tablet_manager.cc:1939] T e87d1281f35648b0b115b67c95e244e7 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:36.124148 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8360bdd411cd4e2ea29e26007dc3b80e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.124881 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 279) took 5120 ms. Trace:
I20250901 01:52:36.125602 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c7d16f6ec3d64038ae050dffcad9403a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.125905 2118 rpcz_store.cc:276] 0901 01:52:31.004796 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:31.005391 (+ 595us) service_pool.cc:225] Handling call
0901 01:52:36.105221 (+5099830us) ts_tablet_manager.cc:1145] Deleting tablet e87d1281f35648b0b115b67c95e244e7
0901 01:52:36.105230 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:36.110948 (+ 5718us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:36.120185 (+ 9237us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:36.124853 (+ 4668us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5099648}
I20250901 01:52:36.126312 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2e0d266c1c9f4c83991eaf37950d5863"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.126502 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet e87d1281f35648b0b115b67c95e244e7 (table default.Table_f840789a58984799bd8057128f6ea439 [id=4453959e273c473fadd737f56f7db3dd]) successfully deleted
I20250901 01:52:36.126890 2118 tablet_replica.cc:331] T 2df2240631844defa433270a3829d65b P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:36.127029 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_48179cd042804028920fe58f6a3a9019"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.127642 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_dcf98a961294461baaccfeb4adec8bd3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.128335 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_fe45599dc0d349ccbd6cebb864b7048d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.128691 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_642300c83bee4c2386815ebe0b5a40bd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.130216 2118 raft_consensus.cc:2241] T 2df2240631844defa433270a3829d65b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:36.131085 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_782d717d09ea41f3a599e9126c0ca026"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.130430 2118 raft_consensus.cc:2270] T 2df2240631844defa433270a3829d65b P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:36.131482 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_0624119c361a4cf799d2ea5afd80dedb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.132625 2118 ts_tablet_manager.cc:1905] T 2df2240631844defa433270a3829d65b P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:36.133751 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_58c5843ccd3f4672a2732ebefce57371"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.134126 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_afc7eca10f8a428985d267ad4aeaf55e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.135635 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8f49a214927e44c38f4ac2d76aa49d93"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.136751 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_05715a21eb5d48bd8447ca5ba51c25f3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.137475 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_6846f20191d24eda8b01f985c6c1aa51"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.137866 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_622e1a7708304465898ee1e8154753c5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:36.053301 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae4844183a3c4a04a775e6fd878391a3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_4bd66ab99a454974aa0f7f4290edfda2'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.138978 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9745582cebfe47b58c21d0191ab4cb9b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77728af9eee44f29a1e2131d5b75b7ca'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.139119 1696 catalog_manager.cc:6205] Error selecting replicas for tablet e2ed32b3292141278af28026528e96c2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b5077f55951d48be92a67407ffe26a08'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.139223 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9f62a9d97f514747b940b488907369f3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d53a456dbca64daf92aeb5fb4517b417'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:36.139232 2118 ts_tablet_manager.cc:1918] T 2df2240631844defa433270a3829d65b P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
E20250901 01:52:36.139343 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5a7fa68d29644ae8afd2711e39dde59a, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d24e3a18ef4843fdb7b74ce23d47cf9e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.139416 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 415c9d7b552b42c384b444a20491f65c, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_f709774f621c483b8ee257273a1e3e38'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.139500 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ff2bd017417c41b5b8225b8f00283e63, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0c7355975d6b44dbbe3a77df942b4d43'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.139565 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2821bf7750ba4e38b25ccba2e2bc0eab, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ce2f3b515c8f4c1bb557c5ed84b48dba'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.139632 1696 catalog_manager.cc:6205] Error selecting replicas for tablet dcc60bb000f440f1aca9133928b5ce20, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2fbca8463d6546c9b8f5b018fc60251d'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.139696 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 465c02a130c54e09b5a74fb4dd2ef762, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_bfddafe8e3204ce8bc9dd5934343ec59'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.139758 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 75883d9fd00e429aa8853903a9fdd7e4, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_a9f6ac8609554a38ab632b2a3f7113d8'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.139835 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 8f4f5563df6b4d0db890747ba032ef9f, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_eaa0b271683343998d39a8ca0f151587'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.139899 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f8f923ddcd2740bcb7e521f4f57c4034, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_e09ac1a9050547038400d4b52299d22f'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.139958 1696 catalog_manager.cc:6205] Error selecting replicas for tablet b1ba0e19dc0b462e80c537bf2d183f5b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_66b111271c15410aab23682f8f6081ac'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.140035 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 8e10a01e77d14cf28c2c5e0205e595b9, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_83cf2d86a1284544ba2e7aa9ebb66bb7'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.140108 1696 catalog_manager.cc:6205] Error selecting replicas for tablet fd2ce82a4ee24d0faa44980a3967e187, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_63d0f2ead9d946d39ce40162ebd769d6'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.140169 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae9a4ad4fdfb4771ba23a91f7007447e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_4cb98468d9f44e16aab412c39b234c15'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.140229 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 086aa948f93644da8e5835b0fac87ddd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b4fbc34da806475c912052d0fab3fc53'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.140290 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 082db24bb3fd4089950504145a20a075, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_4473168010a84ac7a014b99dd10e35a9'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.140350 1696 catalog_manager.cc:6205] Error selecting replicas for tablet fefbf5ae030446b292d8068e95577534, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_3dde393d3a564e84b8ef08c0d25265ce'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.140431 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 93aa22afc0ad4d7fa24807ea3c5d5c21, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0f4b4094d2974b7591875375eda6bed0'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.140496 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9074543dea9949e7a085717b40913348, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_53e30205c5fd488eb3b48c685a159d04'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.140556 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 3a62034e688348ed9e2ac9de009ed3a2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_e225c82a2d6045f6b16915cdb72f5734'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.140617 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 18498c9cb09141d08c1c5c0854b63ebd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77f4810bc83c4d948531da2a45a3792e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.140676 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f5e3316adeae47cd848833dd84b972bb, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_fc962c8bc653498e85cc0bb41639e9ad'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.140739 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 6ce9521b925b4f97bb4a810ce77e147e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_bd74ae506bbe40489259a7be70c3d10b'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.140807 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2f61edb50d7a42198bfd70639c7bf4de, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_fe79329825c54cdf8c97a14aa8a30384'. Need at least 3 replicas, but only 2 tablet servers are available
W20250901 01:52:36.141839 992 leader_election.cc:336] T 45f6e8ba99b74898a2922b36768c6c36 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:36.141849 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3e7c73e2bb7e456a9d9d6ae0503a29e4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.142583 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f3fcec377c7042b9bed7dc2bf85621de"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.143863 1490 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_77f4810bc83c4d948531da2a45a3792e" } new_table_name: "default.Table_e44ed688d0654ad5996005dc0b4b4b16" modify_external_catalogs: true
W20250901 01:52:36.148029 992 consensus_peers.cc:489] T 5adf6b492640417ca071c94702b48cb9 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20250901 01:52:36.148332 2118 log.cc:1199] T 2df2240631844defa433270a3829d65b P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/2df2240631844defa433270a3829d65b
I20250901 01:52:36.148758 2118 ts_tablet_manager.cc:1939] T 2df2240631844defa433270a3829d65b P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:36.148190 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_cbee76dcfeb4408688d5c2c7b0980b0b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:36.151319 1490 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 18498c9cb09141d08c1c5c0854b63ebd has no consensus state
I20250901 01:52:36.151468 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_0f70326b803f4265853919e2898c01dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.152815 1151 tablet_bootstrap.cc:654] T 3a2bc5ad1cd243daa3c92fceca8ba414 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:36.154593 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3beb3fe4f4f14d3687aec68be5625bac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.155826 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_6962a76439d2472ebbff2d65a5765ed6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.156363 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 281) took 4970 ms. Trace:
I20250901 01:52:36.156539 2118 rpcz_store.cc:276] 0901 01:52:31.185998 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:31.186055 (+ 57us) service_pool.cc:225] Handling call
0901 01:52:36.126837 (+4940782us) ts_tablet_manager.cc:1145] Deleting tablet 2df2240631844defa433270a3829d65b
0901 01:52:36.126848 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:36.135996 (+ 9148us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:36.139142 (+ 3146us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:36.156340 (+ 17198us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4940580}
I20250901 01:52:36.156356 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_0c98c0d85b2847abaf2d5fe0d6a13a55"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.157106 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 2df2240631844defa433270a3829d65b (table default.Table_6295dbf840ff483a91edb67301e9b439 [id=597ca217b3f945d88f32297e970378a1]) successfully deleted
I20250901 01:52:36.157455 2118 tablet_replica.cc:331] T 616ff34e8d314e0aa0e11ce60be11789 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:36.157694 2118 raft_consensus.cc:2241] T 616ff34e8d314e0aa0e11ce60be11789 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:36.157828 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b7be7eb11d284a18babbc91edafd2efb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.157964 2118 raft_consensus.cc:2270] T 616ff34e8d314e0aa0e11ce60be11789 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:36.158501 2118 ts_tablet_manager.cc:1905] T 616ff34e8d314e0aa0e11ce60be11789 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:36.158702 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a297f229423d4b2799415796baabc140"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.159516 919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45f6e8ba99b74898a2922b36768c6c36" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:36.159703 919 raft_consensus.cc:3058] T 45f6e8ba99b74898a2922b36768c6c36 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:36.159894 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_eb48575aa8de492b9eadb23e7e9672e8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.161967 911 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9f1f892658b4736859ff507536920a1" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:36.162122 911 raft_consensus.cc:2466] T e9f1f892658b4736859ff507536920a1 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:36.162289 919 raft_consensus.cc:2466] T 45f6e8ba99b74898a2922b36768c6c36 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:36.164523 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f519563e6f3546c496a682b85b2950ad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.164902 2118 ts_tablet_manager.cc:1918] T 616ff34e8d314e0aa0e11ce60be11789 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250901 01:52:36.165001 2118 log.cc:1199] T 616ff34e8d314e0aa0e11ce60be11789 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/616ff34e8d314e0aa0e11ce60be11789
I20250901 01:52:36.165304 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b866d312aba146a4bd4b97746d97f4c8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.165997 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f6d4317518f94f95bb35fb3263305372"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.167734 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_df9596e7d4d54d1e9cf71a300ae99474"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.171303 2118 ts_tablet_manager.cc:1939] T 616ff34e8d314e0aa0e11ce60be11789 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:36.171977 2956 tablet.cc:1722] T 2973b59bb1df41d18d9a5b6a29ae336f P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250901 01:52:36.172093 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 282) took 4960 ms. Trace:
I20250901 01:52:36.173573 2118 rpcz_store.cc:276] 0901 01:52:31.211282 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:31.211518 (+ 236us) service_pool.cc:225] Handling call
0901 01:52:36.157406 (+4945888us) ts_tablet_manager.cc:1145] Deleting tablet 616ff34e8d314e0aa0e11ce60be11789
0901 01:52:36.157418 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:36.161777 (+ 4359us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:36.164813 (+ 3036us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:36.172076 (+ 7263us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4945685}
I20250901 01:52:36.174041 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 616ff34e8d314e0aa0e11ce60be11789 (table default.Table_0f4b4094d2974b7591875375eda6bed0 [id=5ee13d46e06a4fa59b1070547573e523]) successfully deleted
I20250901 01:52:36.174343 2118 tablet_replica.cc:331] T fcb7b5b7d63d4f0b906fea97cb548de2 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:36.174577 2118 raft_consensus.cc:2241] T fcb7b5b7d63d4f0b906fea97cb548de2 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:36.174786 2118 raft_consensus.cc:2270] T fcb7b5b7d63d4f0b906fea97cb548de2 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:36.175329 2118 ts_tablet_manager.cc:1905] T fcb7b5b7d63d4f0b906fea97cb548de2 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:36.177915 1491 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 415c9d7b552b42c384b444a20491f65c
I20250901 01:52:36.179242 1486 catalog_manager.cc:5582] T 2dacb608897d47d28512df87dc320927 P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: UNKNOWN } } }
I20250901 01:52:36.179641 1486 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 2dacb608897d47d28512df87dc320927 (table default.Table_9b9afa82d7de43048ee139b228df93a5 [id=94d314a62cf24de19d6e1fff9b3794ce]). Expected version 1 got 0
I20250901 01:52:36.179875 1486 catalog_manager.cc:5582] T ae21b9ae2e3b472eb67fb5f96388a50f P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: UNKNOWN } } }
I20250901 01:52:36.180168 1486 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet ae21b9ae2e3b472eb67fb5f96388a50f (table default.Table_046289f393ff432aae0b033bdec1fb17 [id=db57b18a9ce24e068b362f5fbe940531]). Expected version 1 got 0
I20250901 01:52:36.188356 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet b8ca4e66c6a74f9d8d90a69be5d6a58f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:29 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:36.195845 992 leader_election.cc:304] T e9f1f892658b4736859ff507536920a1 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:36.196130 2118 ts_tablet_manager.cc:1918] T fcb7b5b7d63d4f0b906fea97cb548de2 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 01:52:36.196094 2948 raft_consensus.cc:2802] T e9f1f892658b4736859ff507536920a1 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:36.196928 2948 raft_consensus.cc:491] T e9f1f892658b4736859ff507536920a1 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:36.197492 2948 raft_consensus.cc:3058] T e9f1f892658b4736859ff507536920a1 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:36.197677 2113 ts_tablet_manager.cc:1918] T 61ddcf86c9f34726939cc116434f0822 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:36.197767 2113 log.cc:1199] T 61ddcf86c9f34726939cc116434f0822 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/61ddcf86c9f34726939cc116434f0822
I20250901 01:52:36.198110 2113 ts_tablet_manager.cc:1939] T 61ddcf86c9f34726939cc116434f0822 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:36.198784 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 412) took 3359 ms. Trace:
I20250901 01:52:36.198876 2113 rpcz_store.cc:276] 0901 01:52:32.839714 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:32.839775 (+ 61us) service_pool.cc:225] Handling call
0901 01:52:36.029254 (+3189479us) ts_tablet_manager.cc:1145] Deleting tablet 61ddcf86c9f34726939cc116434f0822
0901 01:52:36.029265 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:36.149349 (+120084us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:36.197585 (+ 48236us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:36.198767 (+ 1182us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3189261}
I20250901 01:52:36.199595 2118 log.cc:1199] T fcb7b5b7d63d4f0b906fea97cb548de2 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/fcb7b5b7d63d4f0b906fea97cb548de2
I20250901 01:52:36.199975 2118 ts_tablet_manager.cc:1939] T fcb7b5b7d63d4f0b906fea97cb548de2 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:36.200877 2948 raft_consensus.cc:513] T e9f1f892658b4736859ff507536920a1 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:36.201360 1151 tablet_bootstrap.cc:492] T 3a2bc5ad1cd243daa3c92fceca8ba414 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:36.201490 1151 ts_tablet_manager.cc:1397] T 3a2bc5ad1cd243daa3c92fceca8ba414 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.092s user 0.002s sys 0.002s
I20250901 01:52:36.192025 1696 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 415c9d7b552b42c384b444a20491f65c
I20250901 01:52:36.199258 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 61ddcf86c9f34726939cc116434f0822 (table default.Table_63d0f2ead9d946d39ce40162ebd769d6 [id=f9dcf4937b1143248bfa9d88053fce17]) successfully deleted
I20250901 01:52:36.201529 2948 leader_election.cc:290] T e9f1f892658b4736859ff507536920a1 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:36.201262 1483 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet f8f923ddcd2740bcb7e521f4f57c4034
I20250901 01:52:36.201987 992 leader_election.cc:304] T 45f6e8ba99b74898a2922b36768c6c36 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:36.201433 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_37c2a01673774da5901a81fef53fb97f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.202085 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_720e06522f50472080a4b582d3074187"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.202606 2878 raft_consensus.cc:2802] T 45f6e8ba99b74898a2922b36768c6c36 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:36.201452 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f2c2c034d9ca4753b3187f1e91ed1db5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.202884 2878 raft_consensus.cc:695] T 45f6e8ba99b74898a2922b36768c6c36 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:36.203269 1151 raft_consensus.cc:357] T 3a2bc5ad1cd243daa3c92fceca8ba414 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:36.203382 2878 consensus_queue.cc:237] T 45f6e8ba99b74898a2922b36768c6c36 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:36.203421 1151 raft_consensus.cc:383] T 3a2bc5ad1cd243daa3c92fceca8ba414 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:36.203537 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_583c82daa96a4c5db03ba975fe6fc22f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.203980 1151 raft_consensus.cc:738] T 3a2bc5ad1cd243daa3c92fceca8ba414 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:36.204172 1151 consensus_queue.cc:260] T 3a2bc5ad1cd243daa3c92fceca8ba414 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:36.204650 1151 ts_tablet_manager.cc:1428] T 3a2bc5ad1cd243daa3c92fceca8ba414 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250901 01:52:36.204689 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_036008cedecc434b838f87486e8f41eb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.204936 1151 tablet_bootstrap.cc:492] T f8d9bbf0c5be462a8ddc539238cbdf97 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:36.206700 1151 tablet_bootstrap.cc:654] T f8d9bbf0c5be462a8ddc539238cbdf97 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:36.206880 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_dcbcce1a1c5341dfa95d15a9d9077605"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.207369 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9b66b3462b27467cb58728d283a2f4b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.207499 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a12208a60fe14ea982417b36f9344f5c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.208009 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5c6158d041c4448b9199714c38fac4f7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.210979 2956 tablet.cc:1717] T d0c3c363dda444519a17dca63ad9caa7 P 00c282f779c1477d920f24dc0dc16628: Skipping requested alter to schema version 5, tablet already version 5
I20250901 01:52:36.212927 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_cd33aa5f3ffd41389d3e9625255fc303"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.214849 1486 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: default.Table_3c2306df46ec4c14b64b5fbd9d21eb98 [id=746845bd84c6444b8a1e5683eaba4a68] alter complete (version 2)
I20250901 01:52:36.215961 2113 tablet_replica.cc:331] T 4cde5f4f02044b7bbd7d43b49aac4ae8 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:36.201653 1696 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet f8f923ddcd2740bcb7e521f4f57c4034
I20250901 01:52:36.216596 2113 raft_consensus.cc:2241] T 4cde5f4f02044b7bbd7d43b49aac4ae8 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:36.216320 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 283) took 4982 ms. Trace:
I20250901 01:52:36.216730 2118 rpcz_store.cc:276] 0901 01:52:31.233771 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:31.233826 (+ 55us) service_pool.cc:225] Handling call
0901 01:52:36.174303 (+4940477us) ts_tablet_manager.cc:1145] Deleting tablet fcb7b5b7d63d4f0b906fea97cb548de2
0901 01:52:36.174311 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:36.178442 (+ 4131us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:36.196025 (+ 17583us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:36.216302 (+ 20277us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4940280}
I20250901 01:52:36.217178 2113 raft_consensus.cc:2270] T 4cde5f4f02044b7bbd7d43b49aac4ae8 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:36.217387 2118 tablet_replica.cc:331] T d000ba05e12843b4925e622ab55471c0 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:36.217748 2118 raft_consensus.cc:2241] T d000ba05e12843b4925e622ab55471c0 P 00c282f779c1477d920f24dc0dc16628 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:36.217876 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet fcb7b5b7d63d4f0b906fea97cb548de2 (table default.Table_2ceaff93c8f44f00aab53e48d8466dfd [id=4d9ea39016ca4bf4859f4f80ea4e9ed6]) successfully deleted
I20250901 01:52:36.218312 2113 ts_tablet_manager.cc:1905] T 4cde5f4f02044b7bbd7d43b49aac4ae8 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:36.218686 2118 raft_consensus.cc:2270] T d000ba05e12843b4925e622ab55471c0 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:36.219781 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5486197c95914b4abd94a9b5a4c7cf15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.220163 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_db5506fe654c41008f7664dbc7570a83"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.220747 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3f2809560bd64615ab3b7694aae0e292"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.220777 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_67df1c59dab2419aa6b931d3264991c3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.219931 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_91e93056c5fa474d87003221b437403b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.222577 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_fd7504d785f9482c9798f391df99acca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.224164 2118 ts_tablet_manager.cc:1905] T d000ba05e12843b4925e622ab55471c0 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:36.224133 919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9f1f892658b4736859ff507536920a1" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:36.224156 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2ed8ac1005fa4ef5ab44c9b3c03dff7f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.224218 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_727175cfa02d4d75a5e51230d664a618"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.224334 919 raft_consensus.cc:3058] T e9f1f892658b4736859ff507536920a1 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:36.224682 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_57efc2b8b4464114adf5edaf5826366b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.225198 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_dde1e9a437ec4df1aff5c8230b6627f3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.226315 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_755d2b72fd4a4c2fb25535a1cddc2f50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.226970 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ca07d3813fd748b28b95a7dc7301a600"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.227124 919 raft_consensus.cc:2466] T e9f1f892658b4736859ff507536920a1 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:36.227300 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_054d3fb839fb4da4af30509dd0b1fbf3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.227325 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_73cc708effdd4d0fb6733dae616c8f3e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.228145 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_95f5c87f6cb047a787265d1cd5e4b9fc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.228735 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_eff69b8493a14937b1b65eb3707ecc81"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.230453 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_807546400a2844cd8749455524b1d49a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.230464 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_37954df26aba4958b6b4bcd4a2faddf5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.231583 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4584303d36134084861325318d323c39"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.232182 2954 tablet.cc:1717] T d0c3c363dda444519a17dca63ad9caa7 P fd27ab6665304aaea53adbd102dd4f58: Skipping requested alter to schema version 5, tablet already version 5
W20250901 01:52:36.232285 992 leader_election.cc:336] T 45f6e8ba99b74898a2922b36768c6c36 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:36.232491 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_466813e96de04fa48cdcb188d440e670"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:36.232623 992 leader_election.cc:336] T e9f1f892658b4736859ff507536920a1 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:36.233506 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3747b084d5404c239867ed11e4119e92"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.233727 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b7c3f9b9efb9435f8857295edf29086d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.234187 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5b1f4e512b054ef79b7cc0bcdb5c0131"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.234344 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f1018bde7ef64dff9f2c14486be58033"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.237141 992 leader_election.cc:304] T e9f1f892658b4736859ff507536920a1 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:36.237825 2780 raft_consensus.cc:2802] T e9f1f892658b4736859ff507536920a1 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:36.238006 2780 raft_consensus.cc:695] T e9f1f892658b4736859ff507536920a1 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:36.238379 2113 ts_tablet_manager.cc:1918] T 4cde5f4f02044b7bbd7d43b49aac4ae8 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 01:52:36.241487 2118 ts_tablet_manager.cc:1918] T d000ba05e12843b4925e622ab55471c0 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
E20250901 01:52:36.243649 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 494b6488aa0d4355b4caa46004267465, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7c21b68ef2e74136a807c1dd90e17f4e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.243755 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 01deaae6abb141789f0cd240f3058c73, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7baa92b46d98454a891222be49a6ca48'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.243829 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 77911bcb98cf4d8890118eca9136f86e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_db61e055dbd7450898e82c4c0c4763be'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.243888 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae4844183a3c4a04a775e6fd878391a3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_4bd66ab99a454974aa0f7f4290edfda2'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.243949 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9745582cebfe47b58c21d0191ab4cb9b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77728af9eee44f29a1e2131d5b75b7ca'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.244012 1696 catalog_manager.cc:6205] Error selecting replicas for tablet e2ed32b3292141278af28026528e96c2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b5077f55951d48be92a67407ffe26a08'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.244073 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9f62a9d97f514747b940b488907369f3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d53a456dbca64daf92aeb5fb4517b417'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.244135 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5a7fa68d29644ae8afd2711e39dde59a, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d24e3a18ef4843fdb7b74ce23d47cf9e'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:36.244102 2113 log.cc:1199] T 4cde5f4f02044b7bbd7d43b49aac4ae8 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/4cde5f4f02044b7bbd7d43b49aac4ae8
E20250901 01:52:36.244196 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ff2bd017417c41b5b8225b8f00283e63, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0c7355975d6b44dbbe3a77df942b4d43'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.244249 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2821bf7750ba4e38b25ccba2e2bc0eab, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ce2f3b515c8f4c1bb557c5ed84b48dba'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.244313 1696 catalog_manager.cc:6205] Error selecting replicas for tablet dcc60bb000f440f1aca9133928b5ce20, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2fbca8463d6546c9b8f5b018fc60251d'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:36.244807 2113 ts_tablet_manager.cc:1939] T 4cde5f4f02044b7bbd7d43b49aac4ae8 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:36.246182 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7019ad9ce482483fb845ed0be494977d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.246253 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_fcfec4ceaed341ef9ea0b205c4a0bd9a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:36.244374 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 465c02a130c54e09b5a74fb4dd2ef762, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_bfddafe8e3204ce8bc9dd5934343ec59'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.247367 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 75883d9fd00e429aa8853903a9fdd7e4, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_a9f6ac8609554a38ab632b2a3f7113d8'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.247483 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 8f4f5563df6b4d0db890747ba032ef9f, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_eaa0b271683343998d39a8ca0f151587'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.247592 1696 catalog_manager.cc:6205] Error selecting replicas for tablet b1ba0e19dc0b462e80c537bf2d183f5b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_66b111271c15410aab23682f8f6081ac'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.247690 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 8e10a01e77d14cf28c2c5e0205e595b9, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_83cf2d86a1284544ba2e7aa9ebb66bb7'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.247787 1696 catalog_manager.cc:6205] Error selecting replicas for tablet fd2ce82a4ee24d0faa44980a3967e187, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_63d0f2ead9d946d39ce40162ebd769d6'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.247879 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae9a4ad4fdfb4771ba23a91f7007447e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_4cb98468d9f44e16aab412c39b234c15'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.247974 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 086aa948f93644da8e5835b0fac87ddd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b4fbc34da806475c912052d0fab3fc53'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.248123 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 082db24bb3fd4089950504145a20a075, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_4473168010a84ac7a014b99dd10e35a9'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.248241 1696 catalog_manager.cc:6205] Error selecting replicas for tablet fefbf5ae030446b292d8068e95577534, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_3dde393d3a564e84b8ef08c0d25265ce'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.248962 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 93aa22afc0ad4d7fa24807ea3c5d5c21, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0f4b4094d2974b7591875375eda6bed0'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.249097 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 97418fffe6614dfba4ebd13a6ed57e67, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_360f4423e7f341339188b71cc82a1149'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.249235 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9074543dea9949e7a085717b40913348, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_53e30205c5fd488eb3b48c685a159d04'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.249356 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 3a62034e688348ed9e2ac9de009ed3a2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_e225c82a2d6045f6b16915cdb72f5734'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.249495 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 18498c9cb09141d08c1c5c0854b63ebd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_e44ed688d0654ad5996005dc0b4b4b16'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.249634 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f5e3316adeae47cd848833dd84b972bb, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_fc962c8bc653498e85cc0bb41639e9ad'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:36.238307 2780 consensus_queue.cc:237] T e9f1f892658b4736859ff507536920a1 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
E20250901 01:52:36.249763 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 6ce9521b925b4f97bb4a810ce77e147e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_bd74ae506bbe40489259a7be70c3d10b'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.249856 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2f61edb50d7a42198bfd70639c7bf4de, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_fe79329825c54cdf8c97a14aa8a30384'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:36.249948 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 413) took 3408 ms. Trace:
I20250901 01:52:36.250068 2113 rpcz_store.cc:276] 0901 01:52:32.841182 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:32.841233 (+ 51us) service_pool.cc:225] Handling call
0901 01:52:36.215910 (+3374677us) ts_tablet_manager.cc:1145] Deleting tablet 4cde5f4f02044b7bbd7d43b49aac4ae8
0901 01:52:36.215920 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:36.221881 (+ 5961us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:36.238287 (+ 16406us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:36.249929 (+ 11642us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3374516}
I20250901 01:52:36.250694 2118 log.cc:1199] T d000ba05e12843b4925e622ab55471c0 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/d000ba05e12843b4925e622ab55471c0
I20250901 01:52:36.251129 2118 ts_tablet_manager.cc:1939] T d000ba05e12843b4925e622ab55471c0 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:36.245926 1151 tablet_bootstrap.cc:492] T f8d9bbf0c5be462a8ddc539238cbdf97 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:36.251518 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 4cde5f4f02044b7bbd7d43b49aac4ae8 (table default.Table_d53a456dbca64daf92aeb5fb4517b417 [id=77ae3376281349e087a3b3cee41f2a09]) successfully deleted
I20250901 01:52:36.251461 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_0c9c356eb25247d9a774839d5b77fc32"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.251626 1151 ts_tablet_manager.cc:1397] T f8d9bbf0c5be462a8ddc539238cbdf97 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.004s
I20250901 01:52:36.251974 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 414) took 3376 ms. Trace:
I20250901 01:52:36.252043 2113 rpcz_store.cc:276] 0901 01:52:32.875054 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:32.875107 (+ 53us) service_pool.cc:225] Handling call
0901 01:52:36.251896 (+3376789us) ts_tablet_manager.cc:1145] Deleting tablet afc002ae2486476b9208e1cf8215157c
0901 01:52:36.251903 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:36.251960 (+ 57us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3376587}
I20250901 01:52:36.252375 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_521f924a49344f4f81f786e8547beec8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.252558 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9625f3a99e924c7e9749347874ec9250"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.253424 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d20de11d8df849f0b83d66e4051d2024"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.253495 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_aaba35694e974c2aaa5febf659501232"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.253532 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 415) took 3377 ms. Trace:
I20250901 01:52:36.254143 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8458ded48bbf460fbcdb70095b4b0fb5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.254220 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_186cdb6a0aad49469c898048289f95ad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.254030 2113 rpcz_store.cc:276] 0901 01:52:32.875660 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:32.875856 (+ 196us) service_pool.cc:225] Handling call
0901 01:52:36.253225 (+3377369us) ts_tablet_manager.cc:1145] Deleting tablet f9b293bf6a17453b9ea9892a6970d0b0
0901 01:52:36.253261 (+ 36us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:36.253422 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3377192}
W20250901 01:52:36.252959 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet afc002ae2486476b9208e1cf8215157c because the tablet was not found. No further retry: Not found: Tablet not found: afc002ae2486476b9208e1cf8215157c
I20250901 01:52:36.255067 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_160fd712a93346cf9efa93f1ed4f35dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.255324 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 284) took 5021 ms. Trace:
I20250901 01:52:36.255402 2118 rpcz_store.cc:276] 0901 01:52:31.233907 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:31.234046 (+ 139us) service_pool.cc:225] Handling call
0901 01:52:36.217347 (+4983301us) ts_tablet_manager.cc:1145] Deleting tablet d000ba05e12843b4925e622ab55471c0
0901 01:52:36.217356 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:36.233841 (+ 16485us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:36.241391 (+ 7550us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:36.255304 (+ 13913us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4983183}
W20250901 01:52:36.255939 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet f9b293bf6a17453b9ea9892a6970d0b0 because the tablet was not found. No further retry: Not found: Tablet not found: f9b293bf6a17453b9ea9892a6970d0b0
I20250901 01:52:36.256007 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 416) took 3380 ms. Trace:
I20250901 01:52:36.256067 2113 rpcz_store.cc:276] 0901 01:52:32.875785 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:32.875870 (+ 85us) service_pool.cc:225] Handling call
0901 01:52:36.255927 (+3380057us) ts_tablet_manager.cc:1145] Deleting tablet 6c05eff658924ca4a9a1e7a37f917d67
0901 01:52:36.255938 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:36.255993 (+ 55us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3379835}
I20250901 01:52:36.256332 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 417) took 3378 ms. Trace:
W20250901 01:52:36.256542 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 6c05eff658924ca4a9a1e7a37f917d67 because the tablet was not found. No further retry: Not found: Tablet not found: 6c05eff658924ca4a9a1e7a37f917d67
I20250901 01:52:36.257257 2118 tablet_replica.cc:331] T 3d90c6ef06d1436d8e2bb1333e7d9985 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:36.256392 2113 rpcz_store.cc:276] 0901 01:52:32.877397 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:32.877455 (+ 58us) service_pool.cc:225] Handling call
0901 01:52:36.256260 (+3378805us) ts_tablet_manager.cc:1145] Deleting tablet 606fac5c6efe48d1877104b01c23ba5b
0901 01:52:36.256270 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:36.256318 (+ 48us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3378620}
W20250901 01:52:36.257968 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 606fac5c6efe48d1877104b01c23ba5b because the tablet was not found. No further retry: Not found: Tablet not found: 606fac5c6efe48d1877104b01c23ba5b
I20250901 01:52:36.258347 2118 raft_consensus.cc:2241] T 3d90c6ef06d1436d8e2bb1333e7d9985 P 00c282f779c1477d920f24dc0dc16628 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:36.258598 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 419) took 3381 ms. Trace:
I20250901 01:52:36.258342 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a686037e9a804b56b987f7184f71b142"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.258944 2118 raft_consensus.cc:2270] T 3d90c6ef06d1436d8e2bb1333e7d9985 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:36.259438 2118 ts_tablet_manager.cc:1905] T 3d90c6ef06d1436d8e2bb1333e7d9985 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:36.259614 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ffb6cdee665b41f596b89bc2b6f98330"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.260681 2113 rpcz_store.cc:276] 0901 01:52:32.877607 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:32.877672 (+ 65us) service_pool.cc:225] Handling call
0901 01:52:36.258524 (+3380852us) ts_tablet_manager.cc:1145] Deleting tablet 05630a72ddae4de0b890dece2feacc31
0901 01:52:36.258532 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:36.258583 (+ 51us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3380732}
W20250901 01:52:36.261693 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 05630a72ddae4de0b890dece2feacc31 because the tablet was not found. No further retry: Not found: Tablet not found: 05630a72ddae4de0b890dece2feacc31
W20250901 01:52:36.242255 992 leader_election.cc:336] T e9f1f892658b4736859ff507536920a1 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:36.262087 1151 raft_consensus.cc:357] T f8d9bbf0c5be462a8ddc539238cbdf97 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:36.262251 1151 raft_consensus.cc:383] T f8d9bbf0c5be462a8ddc539238cbdf97 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:36.262315 1151 raft_consensus.cc:738] T f8d9bbf0c5be462a8ddc539238cbdf97 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:36.262547 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 421) took 3384 ms. Trace:
I20250901 01:52:36.262409 1490 catalog_manager.cc:5582] T 45f6e8ba99b74898a2922b36768c6c36 P fd27ab6665304aaea53adbd102dd4f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "fd27ab6665304aaea53adbd102dd4f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } health_report { overall_health: UNKNOWN } } }
I20250901 01:52:36.262935 1490 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 45f6e8ba99b74898a2922b36768c6c36 (table default.Table_a2e84eb190f440afbb9dff19c87fda4c [id=9ce26dcf307149b595a43b6ac4562fba]). Expected version 3 got 0
I20250901 01:52:36.266922 1485 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_f91f5f37b4fb40bea1190b3eb31f45d1" } modify_external_catalogs: true
I20250901 01:52:36.267095 1486 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_e225c82a2d6045f6b16915cdb72f5734" } new_table_name: "default.Table_8fda82fcfd3d4e079289989b3bffb99a" modify_external_catalogs: true
I20250901 01:52:36.267143 1485 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_f91f5f37b4fb40bea1190b3eb31f45d1" } modify_external_catalogs: true
I20250901 01:52:36.268142 1489 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_fe79329825c54cdf8c97a14aa8a30384" } new_table_name: "default.Table_cd56686217424ab8b95e6bb776d4177a" modify_external_catalogs: true
I20250901 01:52:36.268345 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2d81ef0719b945d386bc122743060be5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.262499 1151 consensus_queue.cc:260] T f8d9bbf0c5be462a8ddc539238cbdf97 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:36.270941 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_6601b59b9a884ea984f6c2c381a721fc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.270999 2113 rpcz_store.cc:276] 0901 01:52:32.877782 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:32.877815 (+ 33us) service_pool.cc:225] Handling call
0901 01:52:36.262470 (+3384655us) ts_tablet_manager.cc:1145] Deleting tablet 06d5eeb70e664f73bf643922d445bc7a
0901 01:52:36.262478 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:36.262533 (+ 55us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3384278}
W20250901 01:52:36.271526 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 06d5eeb70e664f73bf643922d445bc7a because the tablet was not found. No further retry: Not found: Tablet not found: 06d5eeb70e664f73bf643922d445bc7a
I20250901 01:52:36.271839 1151 ts_tablet_manager.cc:1428] T f8d9bbf0c5be462a8ddc539238cbdf97 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.020s user 0.001s sys 0.001s
I20250901 01:52:36.272245 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 420) took 3394 ms. Trace:
I20250901 01:52:36.272331 2113 rpcz_store.cc:276] 0901 01:52:32.877692 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:32.878217 (+ 525us) service_pool.cc:225] Handling call
0901 01:52:36.272151 (+3393934us) ts_tablet_manager.cc:1145] Deleting tablet a289a66b2dfa44dc8c8547ff83c1716e
0901 01:52:36.272160 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:36.272227 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3393818}
W20250901 01:52:36.272784 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet a289a66b2dfa44dc8c8547ff83c1716e because the tablet was not found. No further retry: Not found: Tablet not found: a289a66b2dfa44dc8c8547ff83c1716e
I20250901 01:52:36.273397 1151 tablet_bootstrap.cc:492] T 0cba5329ab31440999faec599d704633 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:36.276234 1151 tablet_bootstrap.cc:654] T 0cba5329ab31440999faec599d704633 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:36.273504 2113 tablet_replica.cc:331] T fa7fb88ded21432a8757f37ec3db5e71 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:36.277458 2113 raft_consensus.cc:2241] T fa7fb88ded21432a8757f37ec3db5e71 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:36.277701 2113 raft_consensus.cc:2270] T fa7fb88ded21432a8757f37ec3db5e71 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250901 01:52:36.278731 1486 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3a62034e688348ed9e2ac9de009ed3a2 has no consensus state
I20250901 01:52:36.278264 2113 ts_tablet_manager.cc:1905] T fa7fb88ded21432a8757f37ec3db5e71 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:36.280010 1485 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 669d26c6551c4fc58283b67a501d9bf1
W20250901 01:52:36.280238 1485 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 669d26c6551c4fc58283b67a501d9bf1: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
W20250901 01:52:36.280378 858 consensus_peers.cc:489] T 24ddc13d1bb04db4a7850acbff091577 P 00c282f779c1477d920f24dc0dc16628 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20250901 01:52:36.280979 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 669d26c6551c4fc58283b67a501d9bf1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:36 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:36.282446 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 669d26c6551c4fc58283b67a501d9bf1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:36 UTC) from {username='slave'} at 127.0.0.1:34472
W20250901 01:52:36.282572 1489 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2f61edb50d7a42198bfd70639c7bf4de has no consensus state
I20250901 01:52:36.282754 1492 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_e7126b48b6034b0f81ce1c1bef8ffb51" } new_table_name: "default.Table_290a3dc98f0943fd816d3cf2966436a4" modify_external_catalogs: true
I20250901 01:52:36.285048 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 3a2bc5ad1cd243daa3c92fceca8ba414 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:29 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:36.290035 1151 tablet_bootstrap.cc:492] T 0cba5329ab31440999faec599d704633 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:36.292254 2118 ts_tablet_manager.cc:1918] T 3d90c6ef06d1436d8e2bb1333e7d9985 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:36.292368 2118 log.cc:1199] T 3d90c6ef06d1436d8e2bb1333e7d9985 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/3d90c6ef06d1436d8e2bb1333e7d9985
I20250901 01:52:36.292742 2118 ts_tablet_manager.cc:1939] T 3d90c6ef06d1436d8e2bb1333e7d9985 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:36.295360 1151 ts_tablet_manager.cc:1397] T 0cba5329ab31440999faec599d704633 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.022s user 0.000s sys 0.003s
I20250901 01:52:36.296284 1151 raft_consensus.cc:357] T 0cba5329ab31440999faec599d704633 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:36.296447 1492 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 61d8f4252767490f842fb28f3edc1a96 consensus state has no leader
I20250901 01:52:36.296674 1151 raft_consensus.cc:383] T 0cba5329ab31440999faec599d704633 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:36.296802 1151 raft_consensus.cc:738] T 0cba5329ab31440999faec599d704633 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:36.296949 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_764c678e02f34704ae4a80fcbb4cd76c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.297046 1151 consensus_queue.cc:260] T 0cba5329ab31440999faec599d704633 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:36.297719 1151 ts_tablet_manager.cc:1428] T 0cba5329ab31440999faec599d704633 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250901 01:52:36.298076 1151 tablet_bootstrap.cc:492] T c83ce95fd6fa41478de0e2168be201b1 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:36.300035 1151 tablet_bootstrap.cc:654] T c83ce95fd6fa41478de0e2168be201b1 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:36.301060 919 raft_consensus.cc:1273] T 2dacb608897d47d28512df87dc320927 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Refusing update from remote peer fd27ab6665304aaea53adbd102dd4f58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250901 01:52:36.302349 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_685d1a46b226458a97e8134f74fea814"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.303352 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 287) took 4912 ms. Trace:
I20250901 01:52:36.303680 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_6ccbf9d734154c109060a53e7d9f9791"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.303474 2118 rpcz_store.cc:276] 0901 01:52:31.390564 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:31.390611 (+ 47us) service_pool.cc:225] Handling call
0901 01:52:36.257207 (+4866596us) ts_tablet_manager.cc:1145] Deleting tablet 3d90c6ef06d1436d8e2bb1333e7d9985
0901 01:52:36.257217 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:36.278041 (+ 20824us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:36.292155 (+ 14114us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:36.303333 (+ 11178us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4864254}
I20250901 01:52:36.304638 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1110d9e576e941b084facccd66c57fc8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.304737 2948 consensus_queue.cc:1035] T 2dacb608897d47d28512df87dc320927 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:36.304963 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 3d90c6ef06d1436d8e2bb1333e7d9985 (table default.Table_5facf725e0fc44cda7d77f8511030fa5 [id=892696e8040d4f2db8620d1d4ac90e0f]) successfully deleted
I20250901 01:52:36.305375 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 288) took 4812 ms. Trace:
I20250901 01:52:36.305533 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5e92d4978e3a4bc19c4215b6960fc2df"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.305591 2118 rpcz_store.cc:276] 0901 01:52:31.492914 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:31.492990 (+ 76us) service_pool.cc:225] Handling call
0901 01:52:36.305282 (+4812292us) ts_tablet_manager.cc:1145] Deleting tablet 435b1cca5a7d4d49959a310cb250b2a0
0901 01:52:36.305291 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:36.305359 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4679849}
W20250901 01:52:36.305728 992 consensus_peers.cc:489] T 2dacb608897d47d28512df87dc320927 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250901 01:52:36.306046 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 435b1cca5a7d4d49959a310cb250b2a0 because the tablet was not found. No further retry: Not found: Tablet not found: 435b1cca5a7d4d49959a310cb250b2a0
I20250901 01:52:36.306725 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ffbeb531c1f54a5c8fba48c13eeba606"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.307662 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_52864b6742f14231a970a8b903a62f34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.316198 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7a042e3bd00c452fb016e75c635df8a5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.317373 1485 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_4473168010a84ac7a014b99dd10e35a9" } new_table_name: "default.Table_ebba044988d5435aba124b1f971356d1" modify_external_catalogs: true
I20250901 01:52:36.322206 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 289) took 4827 ms. Trace:
I20250901 01:52:36.322369 2118 rpcz_store.cc:276] 0901 01:52:31.494286 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:31.494340 (+ 54us) service_pool.cc:225] Handling call
0901 01:52:36.322099 (+4827759us) ts_tablet_manager.cc:1145] Deleting tablet 061b760d8e6647c6ab0b5d538840a230
0901 01:52:36.322111 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:36.322185 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4696598}
I20250901 01:52:36.322901 2118 tablet_replica.cc:331] T bd9b59f0864a497faf281edd6077668d P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:36.323499 1486 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_4cb98468d9f44e16aab412c39b234c15" } modify_external_catalogs: true
W20250901 01:52:36.323619 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 061b760d8e6647c6ab0b5d538840a230 because the tablet was not found. No further retry: Not found: Tablet not found: 061b760d8e6647c6ab0b5d538840a230
I20250901 01:52:36.323742 1486 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_4cb98468d9f44e16aab412c39b234c15" } modify_external_catalogs: true
I20250901 01:52:36.329088 2113 ts_tablet_manager.cc:1918] T fa7fb88ded21432a8757f37ec3db5e71 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250901 01:52:36.329118 1485 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 082db24bb3fd4089950504145a20a075 has no consensus state
I20250901 01:52:36.329229 2113 log.cc:1199] T fa7fb88ded21432a8757f37ec3db5e71 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/fa7fb88ded21432a8757f37ec3db5e71
I20250901 01:52:36.329325 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2459cd0d170241548faee3db6ed02b34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.329567 2113 ts_tablet_manager.cc:1939] T fa7fb88ded21432a8757f37ec3db5e71 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:36.330308 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 422) took 3452 ms. Trace:
I20250901 01:52:36.330391 2113 rpcz_store.cc:276] 0901 01:52:32.877894 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:32.878350 (+ 456us) service_pool.cc:225] Handling call
0901 01:52:36.273453 (+3395103us) ts_tablet_manager.cc:1145] Deleting tablet fa7fb88ded21432a8757f37ec3db5e71
0901 01:52:36.273465 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:36.282825 (+ 9360us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:36.289391 (+ 6566us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:36.330290 (+ 40899us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3394950}
I20250901 01:52:36.330677 2113 tablet_replica.cc:331] T 764bfeb2a7d14c40815b5ec1b07bdd62 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:36.331137 1486 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet ae9a4ad4fdfb4771ba23a91f7007447e
I20250901 01:52:36.332746 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet fa7fb88ded21432a8757f37ec3db5e71 (table default.Table_22d8d38d0b624912a0d8968fb764420e [id=808a8849c99e485b8be7cd17397874ec]) successfully deleted
I20250901 01:52:36.332942 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5255617d462f40ffa0defb475215a8c2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.333684 2934 consensus_queue.cc:1035] T 34ebb441c27747ec860ff2e25f9a0adc P 00c282f779c1477d920f24dc0dc16628 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:36.334913 1483 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_046289f393ff432aae0b033bdec1fb17" } modify_external_catalogs: true
I20250901 01:52:36.334964 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet f8d9bbf0c5be462a8ddc539238cbdf97 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:30 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:36.335093 1483 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_046289f393ff432aae0b033bdec1fb17" } modify_external_catalogs: true
I20250901 01:52:36.338743 2113 raft_consensus.cc:2241] T 764bfeb2a7d14c40815b5ec1b07bdd62 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:36.339107 2118 raft_consensus.cc:2241] T bd9b59f0864a497faf281edd6077668d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:36.339845 2118 raft_consensus.cc:2270] T bd9b59f0864a497faf281edd6077668d P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:36.339674 2113 raft_consensus.cc:2270] T 764bfeb2a7d14c40815b5ec1b07bdd62 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:36.340523 2118 ts_tablet_manager.cc:1905] T bd9b59f0864a497faf281edd6077668d P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:36.341980 1483 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet ae21b9ae2e3b472eb67fb5f96388a50f
W20250901 01:52:36.342259 1483 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet ae21b9ae2e3b472eb67fb5f96388a50f: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:36.343111 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet ae21b9ae2e3b472eb67fb5f96388a50f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:36 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:36.343734 893 tablet_service.cc:1515] Processing DeleteTablet for tablet ae21b9ae2e3b472eb67fb5f96388a50f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:36 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:36.357028 2780 raft_consensus.cc:491] T eee34044c39d4013baf78d3705489626 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:36.357192 2780 raft_consensus.cc:513] T eee34044c39d4013baf78d3705489626 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:36.356885 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_fb43cb4a91da48d99f7ace90c1443198"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.357748 2780 leader_election.cc:290] T eee34044c39d4013baf78d3705489626 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
E20250901 01:52:36.355839 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 494b6488aa0d4355b4caa46004267465, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7c21b68ef2e74136a807c1dd90e17f4e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.358484 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 01deaae6abb141789f0cd240f3058c73, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7baa92b46d98454a891222be49a6ca48'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.358659 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 77911bcb98cf4d8890118eca9136f86e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_db61e055dbd7450898e82c4c0c4763be'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.358805 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae4844183a3c4a04a775e6fd878391a3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_4bd66ab99a454974aa0f7f4290edfda2'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.358959 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9745582cebfe47b58c21d0191ab4cb9b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77728af9eee44f29a1e2131d5b75b7ca'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.359093 1696 catalog_manager.cc:6205] Error selecting replicas for tablet e2ed32b3292141278af28026528e96c2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b5077f55951d48be92a67407ffe26a08'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.359246 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9f62a9d97f514747b940b488907369f3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d53a456dbca64daf92aeb5fb4517b417'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.359381 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5a7fa68d29644ae8afd2711e39dde59a, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d24e3a18ef4843fdb7b74ce23d47cf9e'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:36.364063 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3ac1d9a3f3044b698c1a5821fced657e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.364645 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_961c9f7052bd437aa4af4c6ec0b75e30"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.364833 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_50fdd788d61448109873a9a284ee81c0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.365188 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a51d4516974146cd9c2cb18f514d1307"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.365260 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a93c8e67f17b4701934f4f4fab028a08"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.365644 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_254461c207704a669a3d2b7c401ed943"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:36.359532 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ff2bd017417c41b5b8225b8f00283e63, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0c7355975d6b44dbbe3a77df942b4d43'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:36.370209 1485 master_service.cc:948] ReplaceTablet: received request to replace tablet 5f5396045979450193d461ba1334a5c9 from {username='slave'} at 127.0.0.1:33478
E20250901 01:52:36.370334 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2821bf7750ba4e38b25ccba2e2bc0eab, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ce2f3b515c8f4c1bb557c5ed84b48dba'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:36.366082 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e76a0d1b5e0d45db8bb67db2faa0d699"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:36.370699 1696 catalog_manager.cc:6205] Error selecting replicas for tablet dcc60bb000f440f1aca9133928b5ce20, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2fbca8463d6546c9b8f5b018fc60251d'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.370827 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 465c02a130c54e09b5a74fb4dd2ef762, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_bfddafe8e3204ce8bc9dd5934343ec59'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.370949 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 75883d9fd00e429aa8853903a9fdd7e4, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_a9f6ac8609554a38ab632b2a3f7113d8'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.371048 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 8f4f5563df6b4d0db890747ba032ef9f, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_eaa0b271683343998d39a8ca0f151587'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:36.366150 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_70c7adb6b634409c8cca40f569241461"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:36.371176 1696 catalog_manager.cc:6205] Error selecting replicas for tablet b1ba0e19dc0b462e80c537bf2d183f5b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_66b111271c15410aab23682f8f6081ac'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.371297 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 8e10a01e77d14cf28c2c5e0205e595b9, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_83cf2d86a1284544ba2e7aa9ebb66bb7'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.371412 1696 catalog_manager.cc:6205] Error selecting replicas for tablet fd2ce82a4ee24d0faa44980a3967e187, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_63d0f2ead9d946d39ce40162ebd769d6'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.371511 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 086aa948f93644da8e5835b0fac87ddd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b4fbc34da806475c912052d0fab3fc53'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:36.366413 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_bdee770396f44a18bdf7a0f7ec7f4fce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:36.371634 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 082db24bb3fd4089950504145a20a075, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ebba044988d5435aba124b1f971356d1'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.371735 1696 catalog_manager.cc:6205] Error selecting replicas for tablet fefbf5ae030446b292d8068e95577534, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_3dde393d3a564e84b8ef08c0d25265ce'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.371863 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 93aa22afc0ad4d7fa24807ea3c5d5c21, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0f4b4094d2974b7591875375eda6bed0'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.371966 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 97418fffe6614dfba4ebd13a6ed57e67, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_360f4423e7f341339188b71cc82a1149'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:36.369840 1484 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_b5077f55951d48be92a67407ffe26a08" } new_table_name: "default.Table_3ee9ea283e004e2197db0e093db416bb" modify_external_catalogs: true
E20250901 01:52:36.372082 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9074543dea9949e7a085717b40913348, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_53e30205c5fd488eb3b48c685a159d04'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.372182 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 3a62034e688348ed9e2ac9de009ed3a2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_8fda82fcfd3d4e079289989b3bffb99a'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.372318 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 18498c9cb09141d08c1c5c0854b63ebd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_e44ed688d0654ad5996005dc0b4b4b16'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.372440 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f5e3316adeae47cd848833dd84b972bb, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_fc962c8bc653498e85cc0bb41639e9ad'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.372560 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 6ce9521b925b4f97bb4a810ce77e147e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_bd74ae506bbe40489259a7be70c3d10b'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:36.372810 1492 master_service.cc:948] ReplaceTablet: received request to replace tablet 7954393c8c1c4457ab8bb723c03a9c8b from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:36.374509 2118 ts_tablet_manager.cc:1918] T bd9b59f0864a497faf281edd6077668d P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:36.374691 2118 log.cc:1199] T bd9b59f0864a497faf281edd6077668d P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/bd9b59f0864a497faf281edd6077668d
I20250901 01:52:36.375092 2118 ts_tablet_manager.cc:1939] T bd9b59f0864a497faf281edd6077668d P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:36.377287 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 290) took 4854 ms. Trace:
I20250901 01:52:36.377408 2118 rpcz_store.cc:276] 0901 01:52:31.522834 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:31.522885 (+ 51us) service_pool.cc:225] Handling call
0901 01:52:36.322855 (+4799970us) ts_tablet_manager.cc:1145] Deleting tablet bd9b59f0864a497faf281edd6077668d
0901 01:52:36.322863 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:36.366121 (+ 43258us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:36.374409 (+ 8288us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:36.377270 (+ 2861us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4694360}
I20250901 01:52:36.378008 2118 tablet_replica.cc:331] T fae2779fac164b9f8052c3666db803b8 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:36.378422 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet bd9b59f0864a497faf281edd6077668d (table default.Table_0c9bfebabc22494489b1c1251a412c81 [id=8b74044a78e0493dbaf1835f33334212]) successfully deleted
W20250901 01:52:36.378417 858 consensus_peers.cc:489] T 34ebb441c27747ec860ff2e25f9a0adc P 00c282f779c1477d920f24dc0dc16628 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:36.379737 2118 raft_consensus.cc:2241] T fae2779fac164b9f8052c3666db803b8 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:36.380267 2118 raft_consensus.cc:2270] T fae2779fac164b9f8052c3666db803b8 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:36.380523 919 raft_consensus.cc:1273] T 45f6e8ba99b74898a2922b36768c6c36 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Refusing update from remote peer fd27ab6665304aaea53adbd102dd4f58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250901 01:52:36.380995 2118 ts_tablet_manager.cc:1905] T fae2779fac164b9f8052c3666db803b8 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:36.382067 992 leader_election.cc:343] T eee34044c39d4013baf78d3705489626 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: eee34044c39d4013baf78d3705489626
I20250901 01:52:36.382200 2878 consensus_queue.cc:1035] T 45f6e8ba99b74898a2922b36768c6c36 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:36.383203 2954 tablet.cc:1722] T 2dacb608897d47d28512df87dc320927 P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:36.383476 2113 ts_tablet_manager.cc:1905] T 764bfeb2a7d14c40815b5ec1b07bdd62 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:36.384975 1484 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e2ed32b3292141278af28026528e96c2 has no consensus state
I20250901 01:52:36.389971 1484 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 2dacb608897d47d28512df87dc320927 (table default.Table_9b9afa82d7de43048ee139b228df93a5 [id=94d314a62cf24de19d6e1fff9b3794ce]). Expected version 1 got 0
I20250901 01:52:36.385193 2956 tablet.cc:1722] T 2dacb608897d47d28512df87dc320927 P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
E20250901 01:52:36.372676 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2f61edb50d7a42198bfd70639c7bf4de, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_cd56686217424ab8b95e6bb776d4177a'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:36.400832 1484 master_service.cc:948] ReplaceTablet: received request to replace tablet e9f1f892658b4736859ff507536920a1 from {username='slave'} at 127.0.0.1:33478
W20250901 01:52:36.403333 992 leader_election.cc:336] T eee34044c39d4013baf78d3705489626 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:36.403458 992 leader_election.cc:304] T eee34044c39d4013baf78d3705489626 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:36.403735 2856 raft_consensus.cc:2747] T eee34044c39d4013baf78d3705489626 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:36.403770 992 consensus_peers.cc:489] T 45f6e8ba99b74898a2922b36768c6c36 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:36.409603 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 0cba5329ab31440999faec599d704633 with delete_type TABLET_DATA_DELETED (replaced by tablet 2f61edb50d7a42198bfd70639c7bf4de) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:36.410427 2956 tablet.cc:1722] T 0e4009963cfe42d6ba164fc828897f09 P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:36.412921 1484 master_service.cc:948] ReplaceTablet: received request to replace tablet 5dd0c6b806574caf9126dad94833fcf7 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:36.414141 2113 ts_tablet_manager.cc:1918] T 764bfeb2a7d14c40815b5ec1b07bdd62 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:36.414250 2113 log.cc:1199] T 764bfeb2a7d14c40815b5ec1b07bdd62 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/764bfeb2a7d14c40815b5ec1b07bdd62
I20250901 01:52:36.414089 1484 master_service.cc:948] ReplaceTablet: received request to replace tablet 61ddcf86c9f34726939cc116434f0822 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:36.415532 1151 tablet_bootstrap.cc:492] T c83ce95fd6fa41478de0e2168be201b1 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:36.415746 1151 ts_tablet_manager.cc:1397] T c83ce95fd6fa41478de0e2168be201b1 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.118s user 0.005s sys 0.004s
I20250901 01:52:36.416287 2113 ts_tablet_manager.cc:1939] T 764bfeb2a7d14c40815b5ec1b07bdd62 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:36.417901 2954 tablet.cc:1722] T 0e4009963cfe42d6ba164fc828897f09 P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:36.421343 1151 raft_consensus.cc:357] T c83ce95fd6fa41478de0e2168be201b1 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:36.421643 1151 raft_consensus.cc:383] T c83ce95fd6fa41478de0e2168be201b1 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:36.421795 1151 raft_consensus.cc:738] T c83ce95fd6fa41478de0e2168be201b1 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:36.422236 1151 consensus_queue.cc:260] T c83ce95fd6fa41478de0e2168be201b1 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:36.423560 1151 ts_tablet_manager.cc:1428] T c83ce95fd6fa41478de0e2168be201b1 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250901 01:52:36.424136 1151 tablet_bootstrap.cc:492] T 5dd0c6b806574caf9126dad94833fcf7 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:36.427528 1484 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_83cf2d86a1284544ba2e7aa9ebb66bb7" } modify_external_catalogs: true
I20250901 01:52:36.427898 1484 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_83cf2d86a1284544ba2e7aa9ebb66bb7" } modify_external_catalogs: true
I20250901 01:52:36.431016 1492 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: default.Table_9b9afa82d7de43048ee139b228df93a5 [id=94d314a62cf24de19d6e1fff9b3794ce] alter complete (version 1)
I20250901 01:52:36.433467 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_6a655b445566484780b99be15670fd08"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.436924 2956 tablet.cc:1722] T 45f6e8ba99b74898a2922b36768c6c36 P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250901 01:52:36.438441 2954 tablet.cc:1722] T 45f6e8ba99b74898a2922b36768c6c36 P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20250901 01:52:36.439867 992 consensus_peers.cc:489] T 49aaaec611784e6bac131efd58dbca51 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250901 01:52:36.441102 1151 tablet_bootstrap.cc:654] T 5dd0c6b806574caf9126dad94833fcf7 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:36.435303 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_66b18a2650014dca8039dcde9abf5711"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.435719 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1ee80d9f97b54b939ce542323937036b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.436085 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a6e9638008ea4de28f8b681bc7c8eb02"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.436380 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_62cc9a4051e44418a19c0092b15076d7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.443480 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 424) took 3564 ms. Trace:
I20250901 01:52:36.436714 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b13f827d16e14cef888ed84c04dad509"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.443646 2113 rpcz_store.cc:276] 0901 01:52:32.878482 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:32.878522 (+ 40us) service_pool.cc:225] Handling call
0901 01:52:36.330635 (+3452113us) ts_tablet_manager.cc:1145] Deleting tablet 764bfeb2a7d14c40815b5ec1b07bdd62
0901 01:52:36.330644 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:36.407453 (+ 76809us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:36.414036 (+ 6583us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:36.443435 (+ 29399us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":33472,"tablet-delete.queue_time_us":3452008}
I20250901 01:52:36.444445 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 423) took 3566 ms. Trace:
I20250901 01:52:36.444525 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 764bfeb2a7d14c40815b5ec1b07bdd62 (table default.Table_4401d83f75f8464da49d5b811d7dd89d [id=16143c8e6f484f9a873437e709d8466d]) successfully deleted
I20250901 01:52:36.444590 2113 rpcz_store.cc:276] 0901 01:52:32.878387 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:32.878643 (+ 256us) service_pool.cc:225] Handling call
0901 01:52:36.444264 (+3565621us) ts_tablet_manager.cc:1145] Deleting tablet 246b33338de647689d7404c70676e76f
0901 01:52:36.444277 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:36.444392 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3565495}
I20250901 01:52:36.444947 2753 raft_consensus.cc:491] T 315c15cf0afc44d4a79cb29294281d60 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:36.445077 2753 raft_consensus.cc:513] T 315c15cf0afc44d4a79cb29294281d60 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:36.445186 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 425) took 3566 ms. Trace:
I20250901 01:52:36.445297 2113 rpcz_store.cc:276] 0901 01:52:32.878609 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:32.878755 (+ 146us) service_pool.cc:225] Handling call
0901 01:52:36.445039 (+3566284us) ts_tablet_manager.cc:1145] Deleting tablet 438ae684af234c868a419fbab43b0b98
0901 01:52:36.445048 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:36.445171 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3566157}
I20250901 01:52:36.445772 2113 tablet_replica.cc:331] T 75a703c8e20a4a2fa0ab6ec58d41ef92 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:36.445752 2753 leader_election.cc:290] T 315c15cf0afc44d4a79cb29294281d60 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:36.446228 2113 raft_consensus.cc:2241] T 75a703c8e20a4a2fa0ab6ec58d41ef92 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250901 01:52:36.446353 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 246b33338de647689d7404c70676e76f because the tablet was not found. No further retry: Not found: Tablet not found: 246b33338de647689d7404c70676e76f
I20250901 01:52:36.446329 1055 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "315c15cf0afc44d4a79cb29294281d60" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:36.446529 2113 raft_consensus.cc:2270] T 75a703c8e20a4a2fa0ab6ec58d41ef92 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250901 01:52:36.446647 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 438ae684af234c868a419fbab43b0b98 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 438ae684af234c868a419fbab43b0b98 already in progress: creating tablet
W20250901 01:52:36.446971 859 leader_election.cc:343] T 315c15cf0afc44d4a79cb29294281d60 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:36.447693 2113 ts_tablet_manager.cc:1905] T 75a703c8e20a4a2fa0ab6ec58d41ef92 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:36.448699 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_bee3cfe11415410bbb2a38d32fb12b10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.449101 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_89e624f1e1c04e38aedac13d0be77205"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.449820 1488 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_d56ff20d129b440cbab7b15c4af6fa46" } modify_external_catalogs: true
I20250901 01:52:36.450027 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1db72eb8d8214a5b971f37a91ed3b562"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.450831 1488 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_d56ff20d129b440cbab7b15c4af6fa46" } modify_external_catalogs: true
I20250901 01:52:36.453325 2118 ts_tablet_manager.cc:1918] T fae2779fac164b9f8052c3666db803b8 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:36.453433 2118 log.cc:1199] T fae2779fac164b9f8052c3666db803b8 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/fae2779fac164b9f8052c3666db803b8
I20250901 01:52:36.453799 2118 ts_tablet_manager.cc:1939] T fae2779fac164b9f8052c3666db803b8 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:36.454530 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 291) took 4903 ms. Trace:
I20250901 01:52:36.449790 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f469b05ef5134df5b85556983c007340"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.458818 1488 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet df01d5df9fc2424eaf9de4e68e781803
W20250901 01:52:36.459055 1488 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet df01d5df9fc2424eaf9de4e68e781803: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:36.460856 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ac988228732f4363940a20ab31080663"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.460992 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f1e60800b6184cbf9d04b6b6821fa74f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.462287 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3ea2c7afbe924454ae500f84a190551f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.462205 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_557e155b68834535808b6cc6cce517d0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.462976 2878 raft_consensus.cc:491] T 88ceba7f48b04d7885cc0164668700a5 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:36.463142 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_56166191a2b7462eb7e7aa4fd3905e16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.463138 2878 raft_consensus.cc:513] T 88ceba7f48b04d7885cc0164668700a5 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:36.463804 2878 leader_election.cc:290] T 88ceba7f48b04d7885cc0164668700a5 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:36.462239 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a8c2ae1f425643408d81f359aa524d20"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.463980 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3fc052da47e6470e932cd95be1353ca1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.464450 2118 rpcz_store.cc:276] 0901 01:52:31.551240 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:31.551310 (+ 70us) service_pool.cc:225] Handling call
0901 01:52:36.377957 (+4826647us) ts_tablet_manager.cc:1145] Deleting tablet fae2779fac164b9f8052c3666db803b8
0901 01:52:36.377969 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:36.419517 (+ 41548us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:36.453201 (+ 33684us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:36.454509 (+ 1308us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4746972}
I20250901 01:52:36.464905 2118 tablet_replica.cc:331] T 3dae00aa38204ee3ac77325eb334be42 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:36.464845 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_bd7760796dce41b8a519f95cd9df7e0e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.465134 2118 raft_consensus.cc:2241] T 3dae00aa38204ee3ac77325eb334be42 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:36.465273 2118 raft_consensus.cc:2270] T 3dae00aa38204ee3ac77325eb334be42 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:36.465989 1489 catalog_manager.cc:5674] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561) does not have the latest schema for tablet 45f6e8ba99b74898a2922b36768c6c36 (table default.Table_a2e84eb190f440afbb9dff19c87fda4c [id=9ce26dcf307149b595a43b6ac4562fba]). Expected version 3 got 0
I20250901 01:52:36.465847 2118 ts_tablet_manager.cc:1905] T 3dae00aa38204ee3ac77325eb334be42 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:36.466328 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8cbc07fd0f9c445fbbdfdda37c114ab8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.464845 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1983088021814198980e6e37ac894002"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.469056 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d8d10072ee6a4fa5bd9dff76991ed6e6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.469205 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_419b75a4c70a4100b7f3e8128edc59b1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.469302 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_12b77ceff2d141bdb717618aebd56a6e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.470103 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_123a1f7d13f74273b5a8b08bfcd7aa10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.470304 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_bd6e94f4382f4fa486414f818c88abf3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.470937 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a3a967e36ef2417188eac196e388f068"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.471637 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet df01d5df9fc2424eaf9de4e68e781803 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:36 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:36.472045 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b83f7f84f65944a881cbc8c311e3c0ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.472100 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_01d8e80e1b43482184424348cdb2226c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:36.473307 858 leader_election.cc:336] T 315c15cf0afc44d4a79cb29294281d60 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:36.473438 858 leader_election.cc:304] T 315c15cf0afc44d4a79cb29294281d60 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:36.474174 893 tablet_service.cc:1515] Processing DeleteTablet for tablet df01d5df9fc2424eaf9de4e68e781803 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:36 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:36.474504 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet fae2779fac164b9f8052c3666db803b8 (table default.Table_7736ed4b5a0d4b1a816500c525e68ad3 [id=7191ff77d44c4c86aa44473814570d09]) successfully deleted
I20250901 01:52:36.474858 2753 raft_consensus.cc:2747] T 315c15cf0afc44d4a79cb29294281d60 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:36.476807 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f58da4edc83849e3992c3992a3d3ea2e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.476972 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e9250f30827a43cc922702978f5a32cf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.477785 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7d8cda9e26be41758f6c165f6828dfd4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.478179 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d0c1687cbddd475f99997a71c134e39e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.478410 2118 ts_tablet_manager.cc:1918] T 3dae00aa38204ee3ac77325eb334be42 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:36.478332 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_89087dd7a55a4241b751753a7d11caa9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.478503 2118 log.cc:1199] T 3dae00aa38204ee3ac77325eb334be42 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/3dae00aa38204ee3ac77325eb334be42
I20250901 01:52:36.478642 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c464ef40020e49a88d998540e27420d7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.478830 2118 ts_tablet_manager.cc:1939] T 3dae00aa38204ee3ac77325eb334be42 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:36.478914 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_38773d4b33a542a6b499da3bad1a9b5a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.479712 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e5b440c3fd3d4b33808025940bcfc20a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.480607 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2006106d98214ee6a4159c1fc15e8163"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.481508 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_034d284a71984f149045a988336f0b0f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.482893 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c7676a5c33ff41719a3a7127dd901a3d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.482929 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_154f23cebbdb48d8b0aaec1031ee0fed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.483100 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_94d351d0e89a40a19d7a003996f7d08e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.483866 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_cd2a28c23c7048f0af86210b19a4f4c7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.484032 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d84b8382fbab4cb2ad9d84cff530937b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.479604 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 292) took 4859 ms. Trace:
I20250901 01:52:36.484344 2118 rpcz_store.cc:276] 0901 01:52:31.620082 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:31.620136 (+ 54us) service_pool.cc:225] Handling call
0901 01:52:36.464848 (+4844712us) ts_tablet_manager.cc:1145] Deleting tablet 3dae00aa38204ee3ac77325eb334be42
0901 01:52:36.464860 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:36.470912 (+ 6052us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:36.478318 (+ 7406us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:36.479585 (+ 1267us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4833406}
I20250901 01:52:36.484738 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 293) took 4863 ms. Trace:
I20250901 01:52:36.484901 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5e31a10fa221481e98975b89d9392871"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.484874 2118 rpcz_store.cc:276] 0901 01:52:31.621238 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:31.621290 (+ 52us) service_pool.cc:225] Handling call
0901 01:52:36.484639 (+4863349us) ts_tablet_manager.cc:1145] Deleting tablet 3363fd67b1ad44849ecb7b4090b61a72
0901 01:52:36.484650 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:36.484722 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4852149}
I20250901 01:52:36.485193 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 295) took 4862 ms. Trace:
I20250901 01:52:36.485248 2118 rpcz_store.cc:276] 0901 01:52:31.622520 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:31.622561 (+ 41us) service_pool.cc:225] Handling call
0901 01:52:36.485113 (+4862552us) ts_tablet_manager.cc:1145] Deleting tablet 42c1156e8dd84c6fa1396b46de13eb6f
0901 01:52:36.485122 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:36.485178 (+ 56us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4849731}
I20250901 01:52:36.485544 2118 tablet_replica.cc:331] T 008d23f236c348b6be9aee718f8698af P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:36.485724 2118 raft_consensus.cc:2241] T 008d23f236c348b6be9aee718f8698af P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:36.485700 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ee1b9d1980944ef88a9203179914d7cf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.485869 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1a88de112c9340b398d120020e4cbd9b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.486126 2118 raft_consensus.cc:2270] T 008d23f236c348b6be9aee718f8698af P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:36.487140 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 3dae00aa38204ee3ac77325eb334be42 (table default.Table_faf7ef12324f454396e5ae64d64ebe51 [id=5812902df9ea406ba59ff0ff23f130b6]) successfully deleted
W20250901 01:52:36.487500 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 3363fd67b1ad44849ecb7b4090b61a72 because the tablet was not found. No further retry: Not found: Tablet not found: 3363fd67b1ad44849ecb7b4090b61a72
W20250901 01:52:36.487754 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 42c1156e8dd84c6fa1396b46de13eb6f because the tablet was not found. No further retry: Not found: Tablet not found: 42c1156e8dd84c6fa1396b46de13eb6f
I20250901 01:52:36.488134 2118 ts_tablet_manager.cc:1905] T 008d23f236c348b6be9aee718f8698af P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:36.489272 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_473c95422bd14797a2d060b99c39dbc3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.489383 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_0c8b2c1966f243f680a8329794981742"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.490345 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_193556fd57534e0ab9dc13981ac67733"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.490506 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d178b998bb2b4249908098d5c73a143d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.490617 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5112c969a57a4a629dadfc96b86fb5be"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.491355 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b7f8d4a2a4f24bda83a2364b5b994b19"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.491541 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_06be56e012a44c45a6daab7e9257b3f7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.491374 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_78bca7bd107d4c5da5baf7774d14e826"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.492276 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_dbf8ec696ede4cb3bec18373e40801e2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.493170 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a7b51311ef0b4ab69eff2575ca1f2b1e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.493278 1490 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet c83ce95fd6fa41478de0e2168be201b1 (table default.Table_a525265208584713b1b3465d63130cd9 [id=bfd87b9b708449d3b16e3e0639561a49]). Expected version 1 got 0
E20250901 01:52:36.494730 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 494b6488aa0d4355b4caa46004267465, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7c21b68ef2e74136a807c1dd90e17f4e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.494899 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 01deaae6abb141789f0cd240f3058c73, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7baa92b46d98454a891222be49a6ca48'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.495034 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 77911bcb98cf4d8890118eca9136f86e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_db61e055dbd7450898e82c4c0c4763be'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.495138 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae4844183a3c4a04a775e6fd878391a3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_4bd66ab99a454974aa0f7f4290edfda2'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.495254 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9745582cebfe47b58c21d0191ab4cb9b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77728af9eee44f29a1e2131d5b75b7ca'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.495360 1696 catalog_manager.cc:6205] Error selecting replicas for tablet e2ed32b3292141278af28026528e96c2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_3ee9ea283e004e2197db0e093db416bb'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.495487 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9f62a9d97f514747b940b488907369f3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d53a456dbca64daf92aeb5fb4517b417'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.495597 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5a7fa68d29644ae8afd2711e39dde59a, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d24e3a18ef4843fdb7b74ce23d47cf9e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.495705 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ff2bd017417c41b5b8225b8f00283e63, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0c7355975d6b44dbbe3a77df942b4d43'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.495820 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2821bf7750ba4e38b25ccba2e2bc0eab, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ce2f3b515c8f4c1bb557c5ed84b48dba'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.495967 1696 catalog_manager.cc:6205] Error selecting replicas for tablet dcc60bb000f440f1aca9133928b5ce20, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2fbca8463d6546c9b8f5b018fc60251d'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.496076 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 465c02a130c54e09b5a74fb4dd2ef762, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_bfddafe8e3204ce8bc9dd5934343ec59'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.496203 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 75883d9fd00e429aa8853903a9fdd7e4, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_a9f6ac8609554a38ab632b2a3f7113d8'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.496312 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 8f4f5563df6b4d0db890747ba032ef9f, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_eaa0b271683343998d39a8ca0f151587'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.496490 1696 catalog_manager.cc:6205] Error selecting replicas for tablet b1ba0e19dc0b462e80c537bf2d183f5b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_66b111271c15410aab23682f8f6081ac'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.496608 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 8e10a01e77d14cf28c2c5e0205e595b9, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_83cf2d86a1284544ba2e7aa9ebb66bb7'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.496738 1696 catalog_manager.cc:6205] Error selecting replicas for tablet fd2ce82a4ee24d0faa44980a3967e187, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_63d0f2ead9d946d39ce40162ebd769d6'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.496850 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 086aa948f93644da8e5835b0fac87ddd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b4fbc34da806475c912052d0fab3fc53'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.496963 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 082db24bb3fd4089950504145a20a075, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ebba044988d5435aba124b1f971356d1'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.497046 1696 catalog_manager.cc:6205] Error selecting replicas for tablet fefbf5ae030446b292d8068e95577534, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_3dde393d3a564e84b8ef08c0d25265ce'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.497160 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 93aa22afc0ad4d7fa24807ea3c5d5c21, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0f4b4094d2974b7591875375eda6bed0'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.497248 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 97418fffe6614dfba4ebd13a6ed57e67, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_360f4423e7f341339188b71cc82a1149'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.497359 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9074543dea9949e7a085717b40913348, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_53e30205c5fd488eb3b48c685a159d04'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.497457 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 3a62034e688348ed9e2ac9de009ed3a2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_8fda82fcfd3d4e079289989b3bffb99a'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.497582 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 18498c9cb09141d08c1c5c0854b63ebd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_e44ed688d0654ad5996005dc0b4b4b16'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.497686 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f5e3316adeae47cd848833dd84b972bb, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_fc962c8bc653498e85cc0bb41639e9ad'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.497836 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 6ce9521b925b4f97bb4a810ce77e147e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_bd74ae506bbe40489259a7be70c3d10b'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.498353 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2f61edb50d7a42198bfd70639c7bf4de, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_cd56686217424ab8b95e6bb776d4177a'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:36.500989 1482 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_a47ce7aa50f34c2a9c204a5f8376e6ee" } new_table_name: "default.Table_19fb556a9c4941709db9cd00c545c0eb" modify_external_catalogs: true
I20250901 01:52:36.502902 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_15ae6d81c988410fb17f7b4c9e4f6dc1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.504444 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d97b1efbb36b4b34a55f4d249f1255c7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.505506 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d46c0450cccb494f93ffad1dd4bf9d0a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.505952 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_477ccfc0e31e494b9f7971b5291bc475"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.507076 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2c51afd3b4844991a075dc38b93da269"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.508001 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_95d6c7b41af6453d940e2c758b9d4dcd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.510341 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_0102f414933e401a9aa7f37a0bf5a710"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.511332 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_911e2570d4a14965862e1ed6804727a8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.511693 1489 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_62eb2015c3dc4feda347dae9fb970abd" } new_table_name: "default.Table_297125e9cb7f407588dcb89fcafdfb1e" modify_external_catalogs: true
I20250901 01:52:36.512224 2118 ts_tablet_manager.cc:1918] T 008d23f236c348b6be9aee718f8698af P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:36.512344 2118 log.cc:1199] T 008d23f236c348b6be9aee718f8698af P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/008d23f236c348b6be9aee718f8698af
I20250901 01:52:36.512738 2118 ts_tablet_manager.cc:1939] T 008d23f236c348b6be9aee718f8698af P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:36.512794 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b133fc66c5be47318c37ab86dd081bf4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.513556 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 296) took 4889 ms. Trace:
I20250901 01:52:36.513695 2118 rpcz_store.cc:276] 0901 01:52:31.624003 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:31.624051 (+ 48us) service_pool.cc:225] Handling call
0901 01:52:36.485500 (+4861449us) ts_tablet_manager.cc:1145] Deleting tablet 008d23f236c348b6be9aee718f8698af
0901 01:52:36.485508 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:36.497670 (+ 12162us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:36.512101 (+ 14431us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:36.513534 (+ 1433us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4850058}
I20250901 01:52:36.516686 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_07c829654eb84750bed92623a58f0df9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.519937 1486 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_53e30205c5fd488eb3b48c685a159d04" } modify_external_catalogs: true
I20250901 01:52:36.520164 1486 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_53e30205c5fd488eb3b48c685a159d04" } modify_external_catalogs: true
I20250901 01:52:36.525142 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f2b6635f577e4ea5b4acf42537076abf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.526851 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b3f68e5437e94768b22a46c660a6a8ed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.528792 1486 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 9074543dea9949e7a085717b40913348
I20250901 01:52:36.529347 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_6b146dcb2755472598af5692f3b8a630"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.530362 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_681de5a046bb40c88c92935d0cbd5222"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.531788 1490 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: default.Table_1adc70cccd2f4cb3a79df96861cb4181 [id=24fffc7ebf5a47bdb37211910301acf7] alter complete (version 1)
I20250901 01:52:36.532181 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a4b23cc5bcec4f7aaa185625f9859f99"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.537493 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ed2d2ebbc345479286d6286cb4273920"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:36.537659 992 leader_election.cc:343] T 88ceba7f48b04d7885cc0164668700a5 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 88ceba7f48b04d7885cc0164668700a5
I20250901 01:52:36.539649 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_97920be9f67d428bae5d9f0d9af1f83e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.541009 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_571e0178f0a540bea128e8c68fa2e8d4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.542028 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f5f74694a1c542e5ac35205c39399e4c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.543462 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_405210f5fb084fd386cb81e64fabb00d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.544162 2954 tablet.cc:1717] T 45f6e8ba99b74898a2922b36768c6c36 P fd27ab6665304aaea53adbd102dd4f58: Skipping requested alter to schema version 3, tablet already version 3
I20250901 01:52:36.544384 2113 ts_tablet_manager.cc:1918] T 75a703c8e20a4a2fa0ab6ec58d41ef92 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:36.544523 2113 log.cc:1199] T 75a703c8e20a4a2fa0ab6ec58d41ef92 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/75a703c8e20a4a2fa0ab6ec58d41ef92
I20250901 01:52:36.544833 2113 ts_tablet_manager.cc:1939] T 75a703c8e20a4a2fa0ab6ec58d41ef92 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:36.544947 1484 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 8e10a01e77d14cf28c2c5e0205e595b9
W20250901 01:52:36.545217 992 consensus_peers.cc:489] T ad45c90ba0da43c9860745e534664a98 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250901 01:52:36.545325 992 leader_election.cc:336] T 88ceba7f48b04d7885cc0164668700a5 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:36.545426 992 leader_election.cc:304] T 88ceba7f48b04d7885cc0164668700a5 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:36.546000 2878 raft_consensus.cc:2747] T 88ceba7f48b04d7885cc0164668700a5 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:36.546768 2956 tablet.cc:1717] T 45f6e8ba99b74898a2922b36768c6c36 P 00c282f779c1477d920f24dc0dc16628: Skipping requested alter to schema version 3, tablet already version 3
I20250901 01:52:36.550375 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_de0d20b4cfa44340a50706051cfeb6c9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.550803 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_221d768e4048408f81c2961d470dff93"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.552073 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2613580731ae4e7d9cc59ffa66f156a9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.555608 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_eb5f61018d48431eb21b736c828a05fe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.556196 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f7a6b87348b040ad8e61b71aa166d8d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.556955 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e0fbd8bf6ed34be9abf908561d450efa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.557686 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_adb2f8661b7b4a1a9f42c36ca86fbd4f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.557826 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 418) took 3680 ms. Trace:
I20250901 01:52:36.557929 2113 rpcz_store.cc:276] 0901 01:52:32.877509 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:32.878042 (+ 533us) service_pool.cc:225] Handling call
0901 01:52:36.445715 (+3567673us) ts_tablet_manager.cc:1145] Deleting tablet 75a703c8e20a4a2fa0ab6ec58d41ef92
0901 01:52:36.445723 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:36.453385 (+ 7662us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:36.544262 (+ 90877us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:36.557805 (+ 13543us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3566743}
I20250901 01:52:36.558225 2113 tablet_replica.cc:331] T 617cc33d0d23439cb5f1e0c31bf11281 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:36.558290 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a40e01d01a804a9d8d44f965797015ed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.558409 2113 raft_consensus.cc:2241] T 617cc33d0d23439cb5f1e0c31bf11281 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:36.558528 2113 raft_consensus.cc:2270] T 617cc33d0d23439cb5f1e0c31bf11281 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:36.558482 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3c42ff5839184bf4a7d2bcfe6ac35436"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.559005 2113 ts_tablet_manager.cc:1905] T 617cc33d0d23439cb5f1e0c31bf11281 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:36.559002 1491 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: default.Table_a2e84eb190f440afbb9dff19c87fda4c [id=9ce26dcf307149b595a43b6ac4562fba] alter complete (version 3)
I20250901 01:52:36.559741 1490 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_66b111271c15410aab23682f8f6081ac" } new_table_name: "default.Table_f9cac02fe78f4902a8c730f2706f7139" modify_external_catalogs: true
I20250901 01:52:36.560324 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 75a703c8e20a4a2fa0ab6ec58d41ef92 (table default.Table_77728af9eee44f29a1e2131d5b75b7ca [id=a90d1f10531042c3bb7a196b8be61c5f]) successfully deleted
I20250901 01:52:36.560374 2878 raft_consensus.cc:491] T 0cba5329ab31440999faec599d704633 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:36.560523 2878 raft_consensus.cc:513] T 0cba5329ab31440999faec599d704633 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:36.560644 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9485e591c2674f58ac2c5ca005dafb53"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.561122 2878 leader_election.cc:290] T 0cba5329ab31440999faec599d704633 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:36.561275 2954 tablet.cc:1722] T 615b2306fca84003b264da9d5f95ddab P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 7 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 8
W20250901 01:52:36.563064 858 consensus_peers.cc:489] T 6c0abbfc8d904dce989ce51961e4237d P 00c282f779c1477d920f24dc0dc16628 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
E20250901 01:52:36.563640 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 494b6488aa0d4355b4caa46004267465, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7c21b68ef2e74136a807c1dd90e17f4e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.563781 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 01deaae6abb141789f0cd240f3058c73, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7baa92b46d98454a891222be49a6ca48'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.563889 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 77911bcb98cf4d8890118eca9136f86e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_db61e055dbd7450898e82c4c0c4763be'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.563979 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae4844183a3c4a04a775e6fd878391a3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_4bd66ab99a454974aa0f7f4290edfda2'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.564079 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9745582cebfe47b58c21d0191ab4cb9b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77728af9eee44f29a1e2131d5b75b7ca'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.564165 1696 catalog_manager.cc:6205] Error selecting replicas for tablet e2ed32b3292141278af28026528e96c2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_3ee9ea283e004e2197db0e093db416bb'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.564261 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9f62a9d97f514747b940b488907369f3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d53a456dbca64daf92aeb5fb4517b417'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.564353 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5a7fa68d29644ae8afd2711e39dde59a, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d24e3a18ef4843fdb7b74ce23d47cf9e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.564478 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ff2bd017417c41b5b8225b8f00283e63, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0c7355975d6b44dbbe3a77df942b4d43'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.564575 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2821bf7750ba4e38b25ccba2e2bc0eab, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ce2f3b515c8f4c1bb557c5ed84b48dba'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.564672 1696 catalog_manager.cc:6205] Error selecting replicas for tablet dcc60bb000f440f1aca9133928b5ce20, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2fbca8463d6546c9b8f5b018fc60251d'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.564760 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 465c02a130c54e09b5a74fb4dd2ef762, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_bfddafe8e3204ce8bc9dd5934343ec59'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.564862 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 75883d9fd00e429aa8853903a9fdd7e4, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_a9f6ac8609554a38ab632b2a3f7113d8'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.564952 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 8f4f5563df6b4d0db890747ba032ef9f, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_eaa0b271683343998d39a8ca0f151587'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.565052 1696 catalog_manager.cc:6205] Error selecting replicas for tablet b1ba0e19dc0b462e80c537bf2d183f5b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_66b111271c15410aab23682f8f6081ac'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.565145 1696 catalog_manager.cc:6205] Error selecting replicas for tablet fd2ce82a4ee24d0faa44980a3967e187, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_63d0f2ead9d946d39ce40162ebd769d6'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.565253 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 086aa948f93644da8e5835b0fac87ddd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b4fbc34da806475c912052d0fab3fc53'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.565342 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 082db24bb3fd4089950504145a20a075, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ebba044988d5435aba124b1f971356d1'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.565467 1696 catalog_manager.cc:6205] Error selecting replicas for tablet fefbf5ae030446b292d8068e95577534, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_3dde393d3a564e84b8ef08c0d25265ce'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.565577 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 93aa22afc0ad4d7fa24807ea3c5d5c21, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0f4b4094d2974b7591875375eda6bed0'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.565680 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 97418fffe6614dfba4ebd13a6ed57e67, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_360f4423e7f341339188b71cc82a1149'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.565768 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 3a62034e688348ed9e2ac9de009ed3a2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_8fda82fcfd3d4e079289989b3bffb99a'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.565881 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 18498c9cb09141d08c1c5c0854b63ebd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_e44ed688d0654ad5996005dc0b4b4b16'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.565976 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f5e3316adeae47cd848833dd84b972bb, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_fc962c8bc653498e85cc0bb41639e9ad'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.566078 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 6ce9521b925b4f97bb4a810ce77e147e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_bd74ae506bbe40489259a7be70c3d10b'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.566169 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2f61edb50d7a42198bfd70639c7bf4de, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_cd56686217424ab8b95e6bb776d4177a'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:36.565037 2956 tablet.cc:1722] T 615b2306fca84003b264da9d5f95ddab P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 7 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 8
I20250901 01:52:36.567894 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 297) took 4826 ms. Trace:
I20250901 01:52:36.568869 1491 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_3c2306df46ec4c14b64b5fbd9d21eb98" } modify_external_catalogs: true
I20250901 01:52:36.569056 1491 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_3c2306df46ec4c14b64b5fbd9d21eb98" } modify_external_catalogs: true
I20250901 01:52:36.572161 1482 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_c7bfa05d564246a6be270afb7a3f0bde" } modify_external_catalogs: true
I20250901 01:52:36.572355 1482 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_c7bfa05d564246a6be270afb7a3f0bde" } modify_external_catalogs: true
W20250901 01:52:36.573177 992 leader_election.cc:336] T 0cba5329ab31440999faec599d704633 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:36.573961 992 leader_election.cc:343] T 0cba5329ab31440999faec599d704633 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 0cba5329ab31440999faec599d704633
I20250901 01:52:36.574055 992 leader_election.cc:304] T 0cba5329ab31440999faec599d704633 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:36.574400 1484 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_ee3fa2db164c4209b58ffaf8102700ba" } new_table_name: "default.Table_6df4c021a4f44a0bbcfe1c5758999915" modify_external_catalogs: true
I20250901 01:52:36.574460 2911 raft_consensus.cc:2747] T 0cba5329ab31440999faec599d704633 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:36.580183 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 008d23f236c348b6be9aee718f8698af (table default.Table_66b111271c15410aab23682f8f6081ac [id=0d9b514272444aa1914c4ff033b8f611]) successfully deleted
I20250901 01:52:36.584486 1491 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 2973b59bb1df41d18d9a5b6a29ae336f
W20250901 01:52:36.585265 1491 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 2973b59bb1df41d18d9a5b6a29ae336f: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:36.586159 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 2973b59bb1df41d18d9a5b6a29ae336f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:36 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:36.586171 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e16cec22faac450ba55cc75d4bff94fd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.586932 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 2973b59bb1df41d18d9a5b6a29ae336f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:36 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:36.587625 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b8a5aa2b24184feea55d4aba423edd05"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.588568 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7187e55689f44a53ace43ddb2192b5a7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.589499 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_fe3b04786ce74526a136a3d50a9fa457"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.590415 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c1a24cb4821f499eb574c0fe46e5ac31"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.590687 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1235cb915b3f4689ad5f53eaafc9ddba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.591353 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_502bef44f51d48aea9aa26d2a23e1836"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.592252 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a83cc2c7a3984b3abd47f751fe8298b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.593191 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_80c77c388db44940815562ced81932cd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.594177 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e753376714f54c668e8390f73ae8ceff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.596625 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_04361d17f852422da11c62943c675ce0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.567989 2118 rpcz_store.cc:276] 0901 01:52:31.741280 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:31.741328 (+ 48us) service_pool.cc:225] Handling call
0901 01:52:36.567780 (+4826452us) ts_tablet_manager.cc:1145] Deleting tablet 500e3c33598b45a4afa983da7cb0845a
0901 01:52:36.567792 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:36.567875 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4826278}
I20250901 01:52:36.603860 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_52ceb3d758024b0badea9fcdd5e7313c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.604383 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 298) took 4862 ms. Trace:
I20250901 01:52:36.604523 2118 rpcz_store.cc:276] 0901 01:52:31.742088 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:31.742132 (+ 44us) service_pool.cc:225] Handling call
0901 01:52:36.604279 (+4862147us) ts_tablet_manager.cc:1145] Deleting tablet 7442c6ff699141e1aff208ebdc9d192a
0901 01:52:36.604289 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:36.604365 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4862004}
I20250901 01:52:36.604884 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 299) took 4402 ms. Trace:
I20250901 01:52:36.605023 2118 rpcz_store.cc:276] 0901 01:52:32.202551 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:32.203008 (+ 457us) service_pool.cc:225] Handling call
0901 01:52:36.604789 (+4401781us) ts_tablet_manager.cc:1145] Deleting tablet 219fe3882d884ee49b08671b547cac85
0901 01:52:36.604797 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:36.604867 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4401565}
I20250901 01:52:36.605587 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f460d41b621d4468a523d5c69784e944"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.606078 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f7631ff3f55c43b7bf2625017989e847"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.606393 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a3c420477574448380be7b0964785472"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.606791 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_974f2b1e683845d1a58bb9f919adfe34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.607113 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_28291515359e4cb29b7f7ac27e8543b1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.607468 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_fc122068874b4d529df6db3213e9e5e3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:36.607612 992 consensus_peers.cc:489] T 33f637f6a8134215bb1995ec38440073 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250901 01:52:36.607913 2118 tablet_replica.cc:331] T 6ba8241644984394ada95b561e76467e P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:36.607853 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3e27208abba7478eaa4fa29e0abf39e1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.608134 2118 raft_consensus.cc:2241] T 6ba8241644984394ada95b561e76467e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:36.608260 2118 raft_consensus.cc:2270] T 6ba8241644984394ada95b561e76467e P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:36.608295 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_33e3403e36f14f0188d7ed695663f7ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.608843 2118 ts_tablet_manager.cc:1905] T 6ba8241644984394ada95b561e76467e P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:36.609112 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_fb1825895d634d49892b47a7760eb5b5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:36.611112 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 500e3c33598b45a4afa983da7cb0845a because the tablet was not found. No further retry: Not found: Tablet not found: 500e3c33598b45a4afa983da7cb0845a
W20250901 01:52:36.611397 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 7442c6ff699141e1aff208ebdc9d192a because the tablet was not found. No further retry: Not found: Tablet not found: 7442c6ff699141e1aff208ebdc9d192a
W20250901 01:52:36.611670 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 219fe3882d884ee49b08671b547cac85 because the tablet was not found. No further retry: Not found: Tablet not found: 219fe3882d884ee49b08671b547cac85
W20250901 01:52:36.613150 1490 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b1ba0e19dc0b462e80c537bf2d183f5b has no consensus state
I20250901 01:52:36.613684 1482 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet ced4813805004bba9c4667a79ac201fc
W20250901 01:52:36.613889 1482 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet ced4813805004bba9c4667a79ac201fc: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:36.615239 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet ced4813805004bba9c4667a79ac201fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:36 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:36.616000 1482 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_8fda82fcfd3d4e079289989b3bffb99a" } new_table_name: "default.Table_90623cc9a13044b5934c922f14b85470" modify_external_catalogs: true
I20250901 01:52:36.617480 893 tablet_service.cc:1515] Processing DeleteTablet for tablet ced4813805004bba9c4667a79ac201fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:36 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:36.617885 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f0e33eb8c2c84a1db67855f3af83babe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.619391 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_11a263c80e6e43ec978f6856a6acc53c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.621167 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9117f48363d84ef6a478d0b476a9256e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.622859 2113 ts_tablet_manager.cc:1918] T 617cc33d0d23439cb5f1e0c31bf11281 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:36.623026 2113 log.cc:1199] T 617cc33d0d23439cb5f1e0c31bf11281 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/617cc33d0d23439cb5f1e0c31bf11281
I20250901 01:52:36.623363 2113 ts_tablet_manager.cc:1939] T 617cc33d0d23439cb5f1e0c31bf11281 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:36.624106 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 426) took 3739 ms. Trace:
I20250901 01:52:36.624234 2113 rpcz_store.cc:276] 0901 01:52:32.885033 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:32.885090 (+ 57us) service_pool.cc:225] Handling call
0901 01:52:36.558180 (+3673090us) ts_tablet_manager.cc:1145] Deleting tablet 617cc33d0d23439cb5f1e0c31bf11281
0901 01:52:36.558189 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:36.592568 (+ 34379us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:36.622753 (+ 30185us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:36.624089 (+ 1336us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3672893}
I20250901 01:52:36.625125 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 617cc33d0d23439cb5f1e0c31bf11281 (table default.Table_ce1fd26ac65a4ed085d023e10a849e5c [id=fb088d31bba64f04871fbfd355d6263b]) successfully deleted
I20250901 01:52:36.625629 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 427) took 3739 ms. Trace:
I20250901 01:52:36.626340 2113 rpcz_store.cc:276] 0901 01:52:32.885902 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:32.885952 (+ 50us) service_pool.cc:225] Handling call
0901 01:52:36.625519 (+3739567us) ts_tablet_manager.cc:1145] Deleting tablet ad1a6be9fcc045d3bb4e71a2f6bd5ccd
0901 01:52:36.625529 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:36.625612 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3739382}
W20250901 01:52:36.627111 1482 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3a62034e688348ed9e2ac9de009ed3a2 has no consensus state
I20250901 01:52:36.627338 2113 tablet_replica.cc:331] T 36e10328c34242ebb4dbd8f294162696 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:36.627538 2113 raft_consensus.cc:2241] T 36e10328c34242ebb4dbd8f294162696 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:36.627683 2113 raft_consensus.cc:2270] T 36e10328c34242ebb4dbd8f294162696 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:36.632359 1151 tablet_bootstrap.cc:492] T 5dd0c6b806574caf9126dad94833fcf7 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:36.633319 1489 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: default.Table_19fb556a9c4941709db9cd00c545c0eb [id=cecdd8d6ca6d445c9fd82decd2a95366] alter complete (version 8)
I20250901 01:52:36.633651 1151 ts_tablet_manager.cc:1397] T 5dd0c6b806574caf9126dad94833fcf7 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.210s user 0.004s sys 0.011s
I20250901 01:52:36.634331 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_fbf82554187a4556888cf326359c1a77"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.635533 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3f53864f300f443281b60866a513f074"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.636240 1151 raft_consensus.cc:357] T 5dd0c6b806574caf9126dad94833fcf7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:36.636590 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_650ae8b34d0a48ec885c2ef201e3d8e1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.637655 1151 raft_consensus.cc:383] T 5dd0c6b806574caf9126dad94833fcf7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:36.638007 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_cff3bdf46ad746499574bfd48ddc42d9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.638332 1151 raft_consensus.cc:738] T 5dd0c6b806574caf9126dad94833fcf7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:36.638474 2118 ts_tablet_manager.cc:1918] T 6ba8241644984394ada95b561e76467e P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:36.639252 1151 consensus_queue.cc:260] T 5dd0c6b806574caf9126dad94833fcf7 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:36.639243 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7da47639c4d94e2eb195863ed65a4c80"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.641209 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_210bcc5adcad466b8729e787c6a15e70"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.641949 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_07d4bf570ac2458b9eab017ea1a1509a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.640643 2753 raft_consensus.cc:491] T 5d68ea97c71942c7b55b967ba870e0d7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:36.642274 2753 raft_consensus.cc:513] T 5d68ea97c71942c7b55b967ba870e0d7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:36.642797 2753 leader_election.cc:290] T 5d68ea97c71942c7b55b967ba870e0d7 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
W20250901 01:52:36.646230 858 leader_election.cc:336] T 5d68ea97c71942c7b55b967ba870e0d7 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:36.646518 2118 log.cc:1199] T 6ba8241644984394ada95b561e76467e P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/6ba8241644984394ada95b561e76467e
I20250901 01:52:36.647505 2118 ts_tablet_manager.cc:1939] T 6ba8241644984394ada95b561e76467e P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:36.648432 1055 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d68ea97c71942c7b55b967ba870e0d7" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
W20250901 01:52:36.650897 992 consensus_peers.cc:489] T a9b7f2e6548043a7865e25387cdd06c9 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250901 01:52:36.650620 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b45e62ae69a74b6981e2f261a53e0527"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.652890 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5e0c07a7463748b188c3c6bbda296e6d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:36.650319 859 leader_election.cc:343] T 5d68ea97c71942c7b55b967ba870e0d7 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:36.653851 859 leader_election.cc:304] T 5d68ea97c71942c7b55b967ba870e0d7 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:36.654305 2753 raft_consensus.cc:2747] T 5d68ea97c71942c7b55b967ba870e0d7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:36.654994 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_bbe9c48f5c6a40c2a6e65fb2e7d230d6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.655289 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 300) took 4003 ms. Trace:
I20250901 01:52:36.655413 2118 rpcz_store.cc:276] 0901 01:52:32.652315 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:32.652376 (+ 61us) service_pool.cc:225] Handling call
0901 01:52:36.607856 (+3955480us) ts_tablet_manager.cc:1145] Deleting tablet 6ba8241644984394ada95b561e76467e
0901 01:52:36.607867 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:36.626427 (+ 18560us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:36.638355 (+ 11928us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:36.655274 (+ 16919us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3955236}
I20250901 01:52:36.655799 2118 tablet_replica.cc:331] T 44af4764d4f64e44aec84ccb84be6f5f P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:36.655961 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4e671070d7cc4e73872b72fb3b70ed9f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.656376 2118 raft_consensus.cc:2241] T 44af4764d4f64e44aec84ccb84be6f5f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:36.656601 2118 raft_consensus.cc:2270] T 44af4764d4f64e44aec84ccb84be6f5f P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:36.656128 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 6ba8241644984394ada95b561e76467e (table default.Table_a96f01d4ce3c4694a635575ffaa4f02d [id=5dc1ab36f2974f07866b14fedbdabd57]) successfully deleted
I20250901 01:52:36.657197 2118 ts_tablet_manager.cc:1905] T 44af4764d4f64e44aec84ccb84be6f5f P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:36.657487 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a8a248839d4048398246d78f04114885"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.658686 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_13222e75938549f999d894eb5057e907"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.659648 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f1403e1ea1e94d14b9062ff6ba96fbaf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.661264 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_0be5c10dcb6d4592b22cc30c339d90c2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.662566 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_41270085a808416a9b08acb83994feb2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.662940 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2b201e539a734019b50152585ebd40de"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.677014 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1fc536c1eeea488fb2a5552e7ca58c3f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.677966 1151 ts_tablet_manager.cc:1428] T 5dd0c6b806574caf9126dad94833fcf7 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.044s user 0.000s sys 0.001s
I20250901 01:52:36.677912 2956 tablet.cc:1722] T 14b466b2a5a249aa841f1b8b06301edc P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250901 01:52:36.628189 2113 ts_tablet_manager.cc:1905] T 36e10328c34242ebb4dbd8f294162696 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
E20250901 01:52:36.679880 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 494b6488aa0d4355b4caa46004267465, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7c21b68ef2e74136a807c1dd90e17f4e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.680006 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 01deaae6abb141789f0cd240f3058c73, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7baa92b46d98454a891222be49a6ca48'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.680074 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 77911bcb98cf4d8890118eca9136f86e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_db61e055dbd7450898e82c4c0c4763be'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.680135 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae4844183a3c4a04a775e6fd878391a3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_4bd66ab99a454974aa0f7f4290edfda2'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.680212 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9745582cebfe47b58c21d0191ab4cb9b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77728af9eee44f29a1e2131d5b75b7ca'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.680280 1696 catalog_manager.cc:6205] Error selecting replicas for tablet e2ed32b3292141278af28026528e96c2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_3ee9ea283e004e2197db0e093db416bb'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.680367 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9f62a9d97f514747b940b488907369f3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d53a456dbca64daf92aeb5fb4517b417'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.680464 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5a7fa68d29644ae8afd2711e39dde59a, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d24e3a18ef4843fdb7b74ce23d47cf9e'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:36.680757 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_429a154675444a2bbda55802e7e998ca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.683250 1484 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_a525265208584713b1b3465d63130cd9" } modify_external_catalogs: true
I20250901 01:52:36.683470 1484 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_a525265208584713b1b3465d63130cd9" } modify_external_catalogs: true
I20250901 01:52:36.684448 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_eaa6528917e64b97b33425ae2c6b1121"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.685320 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e39aa671d95245a893ddafefa3bad973"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.685773 2954 tablet.cc:1722] T 14b466b2a5a249aa841f1b8b06301edc P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
E20250901 01:52:36.686146 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ff2bd017417c41b5b8225b8f00283e63, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0c7355975d6b44dbbe3a77df942b4d43'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:36.686931 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_add63ebeaaa04c12ac6137aa0bb0a309"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:36.686705 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2821bf7750ba4e38b25ccba2e2bc0eab, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ce2f3b515c8f4c1bb557c5ed84b48dba'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:36.690397 2118 ts_tablet_manager.cc:1918] T 44af4764d4f64e44aec84ccb84be6f5f P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:36.690555 2118 log.cc:1199] T 44af4764d4f64e44aec84ccb84be6f5f P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/44af4764d4f64e44aec84ccb84be6f5f
I20250901 01:52:36.690867 2118 ts_tablet_manager.cc:1939] T 44af4764d4f64e44aec84ccb84be6f5f P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:36.691176 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 301) took 4029 ms. Trace:
I20250901 01:52:36.691321 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_642c2a01bf224dd19e3d162a9d0d8c88"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:36.692028 1696 catalog_manager.cc:6205] Error selecting replicas for tablet dcc60bb000f440f1aca9133928b5ce20, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2fbca8463d6546c9b8f5b018fc60251d'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:36.693239 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_bb8c06b09318408e9d66cfd817d97673"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:36.693011 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 465c02a130c54e09b5a74fb4dd2ef762, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_bfddafe8e3204ce8bc9dd5934343ec59'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.695091 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 75883d9fd00e429aa8853903a9fdd7e4, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_a9f6ac8609554a38ab632b2a3f7113d8'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:36.697028 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2012bef08c5b436a9f7a85f0db36d22a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.697454 2118 rpcz_store.cc:276] 0901 01:52:32.661274 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:32.661325 (+ 51us) service_pool.cc:225] Handling call
0901 01:52:36.655757 (+3994432us) ts_tablet_manager.cc:1145] Deleting tablet 44af4764d4f64e44aec84ccb84be6f5f
0901 01:52:36.655766 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:36.660263 (+ 4497us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:36.690306 (+ 30043us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:36.691162 (+ 856us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3994241}
I20250901 01:52:36.697470 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d8fe211551904410bcaac381990b850a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.697868 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 44af4764d4f64e44aec84ccb84be6f5f (table default.Table_9896b980158948339a0b91f5882ed56b [id=a541b55be0dc420a8353ef4793b925b5]) successfully deleted
I20250901 01:52:36.698166 2118 tablet_replica.cc:331] T 61ddcf86c9f34726939cc116434f0822 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
E20250901 01:52:36.696612 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 8f4f5563df6b4d0db890747ba032ef9f, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_eaa0b271683343998d39a8ca0f151587'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:36.698359 2118 raft_consensus.cc:2241] T 61ddcf86c9f34726939cc116434f0822 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:36.698536 2118 raft_consensus.cc:2270] T 61ddcf86c9f34726939cc116434f0822 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
E20250901 01:52:36.698484 1696 catalog_manager.cc:6205] Error selecting replicas for tablet b1ba0e19dc0b462e80c537bf2d183f5b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_f9cac02fe78f4902a8c730f2706f7139'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.699170 1696 catalog_manager.cc:6205] Error selecting replicas for tablet fd2ce82a4ee24d0faa44980a3967e187, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_63d0f2ead9d946d39ce40162ebd769d6'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:36.699265 2118 ts_tablet_manager.cc:1905] T 61ddcf86c9f34726939cc116434f0822 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
E20250901 01:52:36.699321 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 086aa948f93644da8e5835b0fac87ddd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b4fbc34da806475c912052d0fab3fc53'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.699478 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 082db24bb3fd4089950504145a20a075, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ebba044988d5435aba124b1f971356d1'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.699627 1696 catalog_manager.cc:6205] Error selecting replicas for tablet fefbf5ae030446b292d8068e95577534, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_3dde393d3a564e84b8ef08c0d25265ce'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.699760 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 93aa22afc0ad4d7fa24807ea3c5d5c21, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0f4b4094d2974b7591875375eda6bed0'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.699900 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 97418fffe6614dfba4ebd13a6ed57e67, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_360f4423e7f341339188b71cc82a1149'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.700034 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 3a62034e688348ed9e2ac9de009ed3a2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_90623cc9a13044b5934c922f14b85470'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.700169 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 18498c9cb09141d08c1c5c0854b63ebd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_e44ed688d0654ad5996005dc0b4b4b16'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.700312 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f5e3316adeae47cd848833dd84b972bb, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_fc962c8bc653498e85cc0bb41639e9ad'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.700457 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 6ce9521b925b4f97bb4a810ce77e147e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_bd74ae506bbe40489259a7be70c3d10b'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.700625 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2f61edb50d7a42198bfd70639c7bf4de, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_cd56686217424ab8b95e6bb776d4177a'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:36.705819 2118 ts_tablet_manager.cc:1918] T 61ddcf86c9f34726939cc116434f0822 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:36.706000 2118 log.cc:1199] T 61ddcf86c9f34726939cc116434f0822 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/61ddcf86c9f34726939cc116434f0822
I20250901 01:52:36.706389 2118 ts_tablet_manager.cc:1939] T 61ddcf86c9f34726939cc116434f0822 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:36.706741 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 303) took 3826 ms. Trace:
I20250901 01:52:36.706887 2118 rpcz_store.cc:276] 0901 01:52:32.880573 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:32.880623 (+ 50us) service_pool.cc:225] Handling call
0901 01:52:36.698119 (+3817496us) ts_tablet_manager.cc:1145] Deleting tablet 61ddcf86c9f34726939cc116434f0822
0901 01:52:36.698130 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:36.702842 (+ 4712us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:36.705737 (+ 2895us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:36.706726 (+ 989us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3817265}
I20250901 01:52:36.707396 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 61ddcf86c9f34726939cc116434f0822 (table default.Table_63d0f2ead9d946d39ce40162ebd769d6 [id=f9dcf4937b1143248bfa9d88053fce17]) successfully deleted
I20250901 01:52:36.707762 2118 tablet_replica.cc:331] T 4cde5f4f02044b7bbd7d43b49aac4ae8 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:36.707988 2118 raft_consensus.cc:2241] T 4cde5f4f02044b7bbd7d43b49aac4ae8 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:36.708207 2118 raft_consensus.cc:2270] T 4cde5f4f02044b7bbd7d43b49aac4ae8 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:36.708813 2118 ts_tablet_manager.cc:1905] T 4cde5f4f02044b7bbd7d43b49aac4ae8 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:36.713420 992 consensus_peers.cc:489] T 3239e58531d445fa80165cf678bc277c P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250901 01:52:36.714864 1484 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet c83ce95fd6fa41478de0e2168be201b1
W20250901 01:52:36.715844 1484 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet c83ce95fd6fa41478de0e2168be201b1: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:36.718848 2118 ts_tablet_manager.cc:1918] T 4cde5f4f02044b7bbd7d43b49aac4ae8 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 01:52:36.719098 2118 log.cc:1199] T 4cde5f4f02044b7bbd7d43b49aac4ae8 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/4cde5f4f02044b7bbd7d43b49aac4ae8
I20250901 01:52:36.719483 2118 ts_tablet_manager.cc:1939] T 4cde5f4f02044b7bbd7d43b49aac4ae8 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
W20250901 01:52:36.720098 1473 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet ae21b9ae2e3b472eb67fb5f96388a50f on TS 0cf599f74322478d91475df234e58965 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:36.720256 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 304) took 3838 ms. Trace:
I20250901 01:52:36.720382 2118 rpcz_store.cc:276] 0901 01:52:32.882145 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:32.882202 (+ 57us) service_pool.cc:225] Handling call
0901 01:52:36.707705 (+3825503us) ts_tablet_manager.cc:1145] Deleting tablet 4cde5f4f02044b7bbd7d43b49aac4ae8
0901 01:52:36.707714 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:36.715757 (+ 8043us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:36.718755 (+ 2998us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:36.720237 (+ 1482us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3825348}
I20250901 01:52:36.720866 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 4cde5f4f02044b7bbd7d43b49aac4ae8 (table default.Table_d53a456dbca64daf92aeb5fb4517b417 [id=77ae3376281349e087a3b3cee41f2a09]) successfully deleted
I20250901 01:52:36.721215 2118 tablet_replica.cc:331] T 3239e58531d445fa80165cf678bc277c P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:36.721477 2118 raft_consensus.cc:2241] T 3239e58531d445fa80165cf678bc277c P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:36.721680 2118 raft_consensus.cc:2270] T 3239e58531d445fa80165cf678bc277c P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:36.722337 2118 ts_tablet_manager.cc:1905] T 3239e58531d445fa80165cf678bc277c P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:36.726290 2911 consensus_queue.cc:1035] T ae21b9ae2e3b472eb67fb5f96388a50f P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:36.734594 1151 tablet_bootstrap.cc:492] T 5d68ea97c71942c7b55b967ba870e0d7 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:36.736413 1151 tablet_bootstrap.cc:654] T 5d68ea97c71942c7b55b967ba870e0d7 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:36.737828 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_0d2e3f3493ed4759a214286cdfe782fe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.738950 1485 master_service.cc:948] ReplaceTablet: received request to replace tablet aa424ac6ef1441e6b1cc2867b1c48186 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:36.740425 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a84f32c98b914aeaba8c3e9b78ea79fd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.740829 1483 master_service.cc:948] ReplaceTablet: received request to replace tablet b164b4a67a9143ceaff69b0ce49d63fe from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:36.741091 1486 master_service.cc:948] ReplaceTablet: received request to replace tablet b164b4a67a9143ceaff69b0ce49d63fe from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:36.741451 1488 master_service.cc:948] ReplaceTablet: received request to replace tablet 61ddcf86c9f34726939cc116434f0822 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:36.741856 1488 master_service.cc:948] ReplaceTablet: received request to replace tablet 3eba54e5ddd14c518f4cbd539268ae37 from {username='slave'} at 127.0.0.1:33478
E20250901 01:52:36.742151 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 494b6488aa0d4355b4caa46004267465, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7c21b68ef2e74136a807c1dd90e17f4e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.742250 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 01deaae6abb141789f0cd240f3058c73, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7baa92b46d98454a891222be49a6ca48'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.742329 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 77911bcb98cf4d8890118eca9136f86e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_db61e055dbd7450898e82c4c0c4763be'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:36.742290 1488 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_3ee9ea283e004e2197db0e093db416bb" } new_table_name: "default.Table_6866a2f15cff496783cee46fc2895cb3" modify_external_catalogs: true
E20250901 01:52:36.742398 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae4844183a3c4a04a775e6fd878391a3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_4bd66ab99a454974aa0f7f4290edfda2'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.742468 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9745582cebfe47b58c21d0191ab4cb9b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77728af9eee44f29a1e2131d5b75b7ca'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:36.745397 893 tablet_service.cc:1515] Processing DeleteTablet for tablet c83ce95fd6fa41478de0e2168be201b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:36 UTC) from {username='slave'} at 127.0.0.1:34472
E20250901 01:52:36.751145 1696 catalog_manager.cc:6205] Error selecting replicas for tablet e2ed32b3292141278af28026528e96c2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_3ee9ea283e004e2197db0e093db416bb'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.751559 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9f62a9d97f514747b940b488907369f3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d53a456dbca64daf92aeb5fb4517b417'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:36.752430 1485 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet aa424ac6ef1441e6b1cc2867b1c48186
W20250901 01:52:36.752611 1485 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet aa424ac6ef1441e6b1cc2867b1c48186: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
E20250901 01:52:36.752733 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5a7fa68d29644ae8afd2711e39dde59a, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d24e3a18ef4843fdb7b74ce23d47cf9e'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:36.752921 1485 catalog_manager.cc:6652] ReplaceTablet: tablet aa424ac6ef1441e6b1cc2867b1c48186 deleted and replaced by tablet 9f2416b2cc144054a8d2effc64157615
E20250901 01:52:36.753192 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ff2bd017417c41b5b8225b8f00283e63, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0c7355975d6b44dbbe3a77df942b4d43'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.753465 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2821bf7750ba4e38b25ccba2e2bc0eab, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ce2f3b515c8f4c1bb557c5ed84b48dba'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:36.753458 893 tablet_service.cc:1515] Processing DeleteTablet for tablet aa424ac6ef1441e6b1cc2867b1c48186 with delete_type TABLET_DATA_DELETED (replaced by tablet 9f2416b2cc144054a8d2effc64157615) from {username='slave'} at 127.0.0.1:34472
E20250901 01:52:36.753731 1696 catalog_manager.cc:6205] Error selecting replicas for tablet dcc60bb000f440f1aca9133928b5ce20, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2fbca8463d6546c9b8f5b018fc60251d'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.753990 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 465c02a130c54e09b5a74fb4dd2ef762, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_bfddafe8e3204ce8bc9dd5934343ec59'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:36.754053 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet c83ce95fd6fa41478de0e2168be201b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:36 UTC) from {username='slave'} at 127.0.0.1:35946
E20250901 01:52:36.754272 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 75883d9fd00e429aa8853903a9fdd7e4, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_a9f6ac8609554a38ab632b2a3f7113d8'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.754506 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 8f4f5563df6b4d0db890747ba032ef9f, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_eaa0b271683343998d39a8ca0f151587'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.754710 1696 catalog_manager.cc:6205] Error selecting replicas for tablet b1ba0e19dc0b462e80c537bf2d183f5b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_f9cac02fe78f4902a8c730f2706f7139'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.754941 1696 catalog_manager.cc:6205] Error selecting replicas for tablet fd2ce82a4ee24d0faa44980a3967e187, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_63d0f2ead9d946d39ce40162ebd769d6'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:36.754061 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet aa424ac6ef1441e6b1cc2867b1c48186 with delete_type TABLET_DATA_DELETED (replaced by tablet 9f2416b2cc144054a8d2effc64157615) from {username='slave'} at 127.0.0.1:35946
E20250901 01:52:36.755219 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 086aa948f93644da8e5835b0fac87ddd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b4fbc34da806475c912052d0fab3fc53'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.755529 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 082db24bb3fd4089950504145a20a075, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ebba044988d5435aba124b1f971356d1'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.756001 1696 catalog_manager.cc:6205] Error selecting replicas for tablet fefbf5ae030446b292d8068e95577534, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_3dde393d3a564e84b8ef08c0d25265ce'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.756247 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 93aa22afc0ad4d7fa24807ea3c5d5c21, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0f4b4094d2974b7591875375eda6bed0'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.756738 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 97418fffe6614dfba4ebd13a6ed57e67, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_360f4423e7f341339188b71cc82a1149'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.758205 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 3a62034e688348ed9e2ac9de009ed3a2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_90623cc9a13044b5934c922f14b85470'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.758474 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 18498c9cb09141d08c1c5c0854b63ebd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_e44ed688d0654ad5996005dc0b4b4b16'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.758671 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f5e3316adeae47cd848833dd84b972bb, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_fc962c8bc653498e85cc0bb41639e9ad'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.758859 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 6ce9521b925b4f97bb4a810ce77e147e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_bd74ae506bbe40489259a7be70c3d10b'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.759044 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2f61edb50d7a42198bfd70639c7bf4de, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_cd56686217424ab8b95e6bb776d4177a'. Need at least 3 replicas, but only 2 tablet servers are available
W20250901 01:52:36.761730 992 consensus_peers.cc:489] T ae21b9ae2e3b472eb67fb5f96388a50f P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:36.763046 1485 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_bd74ae506bbe40489259a7be70c3d10b" } new_table_name: "default.Table_666a4667dfa84c5ca860560cef32d55c" modify_external_catalogs: true
W20250901 01:52:36.763125 992 consensus_peers.cc:489] T ed1b41340ce742a88c3e4fd85cd25ca6 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250901 01:52:36.763934 2956 tablet.cc:1722] T 24ddc13d1bb04db4a7850acbff091577 P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6
I20250901 01:52:36.765997 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ace6ae7393754758a38954190533ea49"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.767304 1484 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_655ebd0a95a745c2a952006ada10fc75" } modify_external_catalogs: true
I20250901 01:52:36.767464 1484 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_655ebd0a95a745c2a952006ada10fc75" } modify_external_catalogs: true
I20250901 01:52:36.768926 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_356473f0079246b3bbde3df64b49fe52"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.769026 1489 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_3dde393d3a564e84b8ef08c0d25265ce" } modify_external_catalogs: true
I20250901 01:52:36.769156 1489 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_3dde393d3a564e84b8ef08c0d25265ce" } modify_external_catalogs: true
I20250901 01:52:36.769703 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_6dff5bc56c724626ae60411d01617fb3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.770148 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_80e454aa85b542599b37ae5d9237f064"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.770401 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_31a72b7e128243419bee719fa4748590"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.770777 2954 tablet.cc:1722] T 24ddc13d1bb04db4a7850acbff091577 P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6
I20250901 01:52:36.770838 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e0a88be21887463ba1ca1d6592f86cd0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.771004 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_0055c862cdd5402fae5458568ea2b496"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.771482 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e52b03df708a46ad94dd5a47fd0c363a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.771580 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8ec03adc4d744fe3821ef03ad72a7691"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.772217 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f6358bb05e174e7fb6ae14c3401b9824"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.772382 1483 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet b164b4a67a9143ceaff69b0ce49d63fe
W20250901 01:52:36.772572 1483 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet b164b4a67a9143ceaff69b0ce49d63fe: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:36.773684 893 tablet_service.cc:1515] Processing DeleteTablet for tablet b164b4a67a9143ceaff69b0ce49d63fe with delete_type TABLET_DATA_DELETED (replaced by tablet a5609f111ab34c0ea9fabdbce06c81f9) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:36.773852 1483 catalog_manager.cc:6652] ReplaceTablet: tablet b164b4a67a9143ceaff69b0ce49d63fe deleted and replaced by tablet a5609f111ab34c0ea9fabdbce06c81f9
I20250901 01:52:36.773948 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet b164b4a67a9143ceaff69b0ce49d63fe with delete_type TABLET_DATA_DELETED (replaced by tablet a5609f111ab34c0ea9fabdbce06c81f9) from {username='slave'} at 127.0.0.1:35946
W20250901 01:52:36.777120 1488 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e2ed32b3292141278af28026528e96c2 has no consensus state
I20250901 01:52:36.777276 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9dba83a95fba4fbebf08e192d4f65133"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.777580 1482 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: default.Table_6df4c021a4f44a0bbcfe1c5758999915 [id=78cd71321e9a480ca2d7d88327f157ee] alter complete (version 5)
I20250901 01:52:36.779197 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f443822708c54317af82e0de0f76e6d7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.780694 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f128c20c05674ba88b81f459b2c398e8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.781630 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_aadf2a91256c4d9999c0a88127e1446b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.782567 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a2ea4442f08c4f4da945d9b4e6cf42db"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:36.784220 1485 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6ce9521b925b4f97bb4a810ce77e147e has no consensus state
I20250901 01:52:36.787982 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_278edccbd40b48f09d77cb491996841e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.788143 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4ea815e412784074a84c573737b7bec2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.788676 2113 ts_tablet_manager.cc:1918] T 36e10328c34242ebb4dbd8f294162696 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:36.788798 2113 log.cc:1199] T 36e10328c34242ebb4dbd8f294162696 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/36e10328c34242ebb4dbd8f294162696
I20250901 01:52:36.788827 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c899622898b54df1b8b78e8a9dde4f0a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.789126 2113 ts_tablet_manager.cc:1939] T 36e10328c34242ebb4dbd8f294162696 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:36.789690 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3e8be3d21b0c4961998011f2b022c384"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.791342 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8568565bcb1d4b999ca879de63354358"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.795333 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a75ce04ad251412e82b020e6dc4ee8cd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.795506 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8d4d5a50454740c790be3699416d879e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.795661 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 428) took 3909 ms. Trace:
I20250901 01:52:36.795642 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ed7fa1703b564f528f10ef2b83beed6e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.795740 2113 rpcz_store.cc:276] 0901 01:52:32.886006 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:32.886175 (+ 169us) service_pool.cc:225] Handling call
0901 01:52:36.627291 (+3741116us) ts_tablet_manager.cc:1145] Deleting tablet 36e10328c34242ebb4dbd8f294162696
0901 01:52:36.627300 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:36.748903 (+121603us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:36.762899 (+ 13996us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:36.795634 (+ 32735us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3740986}
I20250901 01:52:36.796087 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 36e10328c34242ebb4dbd8f294162696 (table default.Table_8445fcd774874cadaf4acc3581eee417 [id=958e51a6bbee47638e504e1a44094737]) successfully deleted
I20250901 01:52:36.796118 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_782179e6cbdb4f9492249825feea878a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.796591 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4f5167c413f84b36adc3266027b22cb8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.796847 2113 tablet_replica.cc:331] T d4c9aa723aee4dc4b6237cb5c3722b52 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:36.797024 2113 raft_consensus.cc:2241] T d4c9aa723aee4dc4b6237cb5c3722b52 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:36.782563 1489 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet fefbf5ae030446b292d8068e95577534
I20250901 01:52:36.797248 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3a640af84b654fc4b2410a3908ede14a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.797490 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1896d91cede945378058e6a126b0732d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.797585 2113 raft_consensus.cc:2270] T d4c9aa723aee4dc4b6237cb5c3722b52 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:36.798122 2113 ts_tablet_manager.cc:1905] T d4c9aa723aee4dc4b6237cb5c3722b52 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:36.798321 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7ce6273f8990483ba7e47c6b55eaf85a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.799189 2118 ts_tablet_manager.cc:1918] T 3239e58531d445fa80165cf678bc277c P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:36.799331 2118 log.cc:1199] T 3239e58531d445fa80165cf678bc277c P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/3239e58531d445fa80165cf678bc277c
I20250901 01:52:36.799357 1490 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_1adc70cccd2f4cb3a79df96861cb4181" } modify_external_catalogs: true
I20250901 01:52:36.799682 2118 ts_tablet_manager.cc:1939] T 3239e58531d445fa80165cf678bc277c P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:36.799626 1490 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_1adc70cccd2f4cb3a79df96861cb4181" } modify_external_catalogs: true
I20250901 01:52:36.800429 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 305) took 3583 ms. Trace:
I20250901 01:52:36.800515 2118 rpcz_store.cc:276] 0901 01:52:33.217160 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:33.217206 (+ 46us) service_pool.cc:225] Handling call
0901 01:52:36.721165 (+3503959us) ts_tablet_manager.cc:1145] Deleting tablet 3239e58531d445fa80165cf678bc277c
0901 01:52:36.721175 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:36.731725 (+ 10550us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:36.799089 (+ 67364us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:36.800388 (+ 1299us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3503791}
I20250901 01:52:36.800995 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 3239e58531d445fa80165cf678bc277c (table default.Table_9163155cb611406a8ce0d5ae600a15c8 [id=aa9c7768ab244d60bedd5ea94cc5e94e]) successfully deleted
I20250901 01:52:36.801510 2118 tablet_replica.cc:331] T b009cc95c819431ea9827a3e453b8da3 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:36.801728 2118 raft_consensus.cc:2241] T b009cc95c819431ea9827a3e453b8da3 P 00c282f779c1477d920f24dc0dc16628 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:36.802085 2118 raft_consensus.cc:2270] T b009cc95c819431ea9827a3e453b8da3 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:36.802387 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_0c55d22af5a14345ae4b03b1b50dd266"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.803486 2118 ts_tablet_manager.cc:1905] T b009cc95c819431ea9827a3e453b8da3 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:36.804226 992 consensus_peers.cc:489] T 3dae00aa38204ee3ac77325eb334be42 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Couldn't send request to peer 00c282f779c1477d920f24dc0dc16628. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3dae00aa38204ee3ac77325eb334be42. This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:36.805089 1151 tablet_bootstrap.cc:492] T 5d68ea97c71942c7b55b967ba870e0d7 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:36.805272 1151 ts_tablet_manager.cc:1397] T 5d68ea97c71942c7b55b967ba870e0d7 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.071s user 0.003s sys 0.002s
I20250901 01:52:36.806142 1151 raft_consensus.cc:357] T 5d68ea97c71942c7b55b967ba870e0d7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:36.806180 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_fb49fdce57f446bfbf3f6be927810956"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.806408 2911 raft_consensus.cc:491] T b8ca4e66c6a74f9d8d90a69be5d6a58f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:36.806627 2911 raft_consensus.cc:513] T b8ca4e66c6a74f9d8d90a69be5d6a58f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:36.807147 2911 leader_election.cc:290] T b8ca4e66c6a74f9d8d90a69be5d6a58f P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:36.807338 1151 raft_consensus.cc:383] T 5d68ea97c71942c7b55b967ba870e0d7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:36.807452 1151 raft_consensus.cc:738] T 5d68ea97c71942c7b55b967ba870e0d7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
W20250901 01:52:36.809381 992 leader_election.cc:336] T b8ca4e66c6a74f9d8d90a69be5d6a58f P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:36.809506 1151 consensus_queue.cc:260] T 5d68ea97c71942c7b55b967ba870e0d7 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:36.809840 992 leader_election.cc:343] T b8ca4e66c6a74f9d8d90a69be5d6a58f P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: b8ca4e66c6a74f9d8d90a69be5d6a58f
I20250901 01:52:36.809931 992 leader_election.cc:304] T b8ca4e66c6a74f9d8d90a69be5d6a58f P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:36.811017 2911 raft_consensus.cc:2747] T b8ca4e66c6a74f9d8d90a69be5d6a58f P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:36.812296 1151 ts_tablet_manager.cc:1428] T 5d68ea97c71942c7b55b967ba870e0d7 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20250901 01:52:36.812618 1151 tablet_bootstrap.cc:492] T 3697821cd29446bb908b020599df0db9 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:36.813949 1151 tablet_bootstrap.cc:654] T 3697821cd29446bb908b020599df0db9 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:36.817451 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_105411e30a914380a3cf69dcf7580402"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.818383 1484 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 5d68ea97c71942c7b55b967ba870e0d7
W20250901 01:52:36.818640 1484 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 5d68ea97c71942c7b55b967ba870e0d7: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:36.818846 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f2ac3008f6cf4345af41150c6d316a23"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.819382 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 5d68ea97c71942c7b55b967ba870e0d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:36 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:36.820024 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 5d68ea97c71942c7b55b967ba870e0d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:36 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:36.820832 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e1c7624548104427947b8c3a3cc8c252"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.821492 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_925ea78261544ed1a550f0003af5f828"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.821792 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_796b8bb111b2408b93f77d7700551f02"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.823936 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_cfdd2eb7b5ac44b6a11872482da65f0c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.824069 2118 ts_tablet_manager.cc:1918] T b009cc95c819431ea9827a3e453b8da3 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20250901 01:52:36.824157 2118 log.cc:1199] T b009cc95c819431ea9827a3e453b8da3 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/b009cc95c819431ea9827a3e453b8da3
I20250901 01:52:36.824563 2118 ts_tablet_manager.cc:1939] T b009cc95c819431ea9827a3e453b8da3 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
E20250901 01:52:36.824599 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 494b6488aa0d4355b4caa46004267465, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7c21b68ef2e74136a807c1dd90e17f4e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.824803 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 01deaae6abb141789f0cd240f3058c73, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7baa92b46d98454a891222be49a6ca48'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.824980 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 77911bcb98cf4d8890118eca9136f86e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_db61e055dbd7450898e82c4c0c4763be'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:36.824959 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b7447fdf535f4685a8af9d8d310226c0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:36.825330 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae4844183a3c4a04a775e6fd878391a3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_4bd66ab99a454974aa0f7f4290edfda2'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:36.825320 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 306) took 3607 ms. Trace:
E20250901 01:52:36.825531 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9745582cebfe47b58c21d0191ab4cb9b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77728af9eee44f29a1e2131d5b75b7ca'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:36.825395 2118 rpcz_store.cc:276] 0901 01:52:33.217614 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:33.217657 (+ 43us) service_pool.cc:225] Handling call
0901 01:52:36.801461 (+3583804us) ts_tablet_manager.cc:1145] Deleting tablet b009cc95c819431ea9827a3e453b8da3
0901 01:52:36.801471 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:36.817527 (+ 16056us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:36.823957 (+ 6430us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:36.825301 (+ 1344us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3583654}
I20250901 01:52:36.825850 2118 tablet_replica.cc:331] T d27c8d99528a42ed8dd1c5a7f373bbae P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:36.825850 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet b009cc95c819431ea9827a3e453b8da3 (table default.Table_e53182d58e5244a9b7ba33cb48c939ba [id=cc26c89db2484a6fad919231116285f2]) successfully deleted
I20250901 01:52:36.826074 2118 raft_consensus.cc:2241] T d27c8d99528a42ed8dd1c5a7f373bbae P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:36.826203 2118 raft_consensus.cc:2270] T d27c8d99528a42ed8dd1c5a7f373bbae P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:36.826478 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_97805673151e4b87aa4a0c37db93da55"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.826951 2118 ts_tablet_manager.cc:1905] T d27c8d99528a42ed8dd1c5a7f373bbae P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:36.827392 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3048ec0580384b5793f4e41d8fc63157"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.829135 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_236b65f6e2e546c5ade5553d5fa202a7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.832302 1490 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 0e4009963cfe42d6ba164fc828897f09
W20250901 01:52:36.832527 1490 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 0e4009963cfe42d6ba164fc828897f09: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:36.833760 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 0e4009963cfe42d6ba164fc828897f09 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:36 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:36.834360 1491 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: default.Table_297125e9cb7f407588dcb89fcafdfb1e [id=18aecbf3fe4643c4b589ee47706161bd] alter complete (version 6)
I20250901 01:52:36.835023 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 0e4009963cfe42d6ba164fc828897f09 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:36 UTC) from {username='slave'} at 127.0.0.1:35946
E20250901 01:52:36.838783 1696 catalog_manager.cc:6205] Error selecting replicas for tablet e2ed32b3292141278af28026528e96c2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_6866a2f15cff496783cee46fc2895cb3'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.838917 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9f62a9d97f514747b940b488907369f3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d53a456dbca64daf92aeb5fb4517b417'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.839010 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5a7fa68d29644ae8afd2711e39dde59a, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d24e3a18ef4843fdb7b74ce23d47cf9e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.839099 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ff2bd017417c41b5b8225b8f00283e63, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0c7355975d6b44dbbe3a77df942b4d43'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.839195 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2821bf7750ba4e38b25ccba2e2bc0eab, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ce2f3b515c8f4c1bb557c5ed84b48dba'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.839277 1696 catalog_manager.cc:6205] Error selecting replicas for tablet dcc60bb000f440f1aca9133928b5ce20, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2fbca8463d6546c9b8f5b018fc60251d'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.839355 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 465c02a130c54e09b5a74fb4dd2ef762, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_bfddafe8e3204ce8bc9dd5934343ec59'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.839439 1696 catalog_manager.cc:6205] Error selecting replicas for tablet a5609f111ab34c0ea9fabdbce06c81f9, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_4505d77e4269443995012fef50c7e7b5'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.839512 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9f2416b2cc144054a8d2effc64157615, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_69575b1a9efd4573a2ee01d3f432800b'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.839597 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 75883d9fd00e429aa8853903a9fdd7e4, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_a9f6ac8609554a38ab632b2a3f7113d8'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.839692 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 8f4f5563df6b4d0db890747ba032ef9f, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_eaa0b271683343998d39a8ca0f151587'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.839776 1696 catalog_manager.cc:6205] Error selecting replicas for tablet b1ba0e19dc0b462e80c537bf2d183f5b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_f9cac02fe78f4902a8c730f2706f7139'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.839859 1696 catalog_manager.cc:6205] Error selecting replicas for tablet fd2ce82a4ee24d0faa44980a3967e187, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_63d0f2ead9d946d39ce40162ebd769d6'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.839944 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 086aa948f93644da8e5835b0fac87ddd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b4fbc34da806475c912052d0fab3fc53'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.840018 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 082db24bb3fd4089950504145a20a075, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ebba044988d5435aba124b1f971356d1'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.840083 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 93aa22afc0ad4d7fa24807ea3c5d5c21, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0f4b4094d2974b7591875375eda6bed0'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.840152 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 97418fffe6614dfba4ebd13a6ed57e67, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_360f4423e7f341339188b71cc82a1149'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.840219 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 3a62034e688348ed9e2ac9de009ed3a2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_90623cc9a13044b5934c922f14b85470'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.840284 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 18498c9cb09141d08c1c5c0854b63ebd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_e44ed688d0654ad5996005dc0b4b4b16'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.840348 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f5e3316adeae47cd848833dd84b972bb, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_fc962c8bc653498e85cc0bb41639e9ad'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.840435 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 6ce9521b925b4f97bb4a810ce77e147e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_666a4667dfa84c5ca860560cef32d55c'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.840507 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2f61edb50d7a42198bfd70639c7bf4de, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_cd56686217424ab8b95e6bb776d4177a'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:36.842058 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_bb5c9b6c0a7148e8b2e800fed7cca252"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.842832 2753 raft_consensus.cc:491] T d0463a8d655d4db4800b4e1b221c8315 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:36.842957 2753 raft_consensus.cc:513] T d0463a8d655d4db4800b4e1b221c8315 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:36.843104 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e53205a045ab4c06aa729c2a4409f92b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.843341 1151 tablet_bootstrap.cc:492] T 3697821cd29446bb908b020599df0db9 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:36.843461 1151 ts_tablet_manager.cc:1397] T 3697821cd29446bb908b020599df0db9 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.000s
I20250901 01:52:36.843525 2753 leader_election.cc:290] T d0463a8d655d4db4800b4e1b221c8315 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:36.843971 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1d063eff23984811828b69741857a84e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.844173 1151 raft_consensus.cc:357] T 3697821cd29446bb908b020599df0db9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:36.844457 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_73cf0965e3b04f0d965895959ab66842"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.844542 1055 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0463a8d655d4db4800b4e1b221c8315" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:36.844352 1151 raft_consensus.cc:383] T 3697821cd29446bb908b020599df0db9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:36.844817 1151 raft_consensus.cc:738] T 3697821cd29446bb908b020599df0db9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:36.844928 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7eebf3080a7842eba38d0dcfe35309cb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.845021 1151 consensus_queue.cc:260] T 3697821cd29446bb908b020599df0db9 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:36.845389 859 leader_election.cc:343] T d0463a8d655d4db4800b4e1b221c8315 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:36.845705 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ef150f3ed9bc4ffbb702f8b853110cf4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.846714 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_17a497975ef84b068c3a377920bd07ee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:36.846818 858 leader_election.cc:336] T d0463a8d655d4db4800b4e1b221c8315 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:36.846920 858 leader_election.cc:304] T d0463a8d655d4db4800b4e1b221c8315 P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:36.847400 2753 raft_consensus.cc:2747] T d0463a8d655d4db4800b4e1b221c8315 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:36.847445 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_71674c820cfc4011b39e51906165d229"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.847985 1151 ts_tablet_manager.cc:1428] T 3697821cd29446bb908b020599df0db9 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250901 01:52:36.848291 1151 tablet_bootstrap.cc:492] T d0463a8d655d4db4800b4e1b221c8315 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:36.848621 1490 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_da8cc5e7d1e24421b170ecca2fc2147b" } new_table_name: "default.Table_335c0676987d4bf3a4d64b239be8067e" modify_external_catalogs: true
I20250901 01:52:36.848794 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_769e9e7e50a048c6aacd7c79f4ec9738"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.849639 1151 tablet_bootstrap.cc:654] T d0463a8d655d4db4800b4e1b221c8315 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:36.850217 1489 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_b8f8ee74648c406eaf589d3ab80934db" } modify_external_catalogs: true
I20250901 01:52:36.850342 1489 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_b8f8ee74648c406eaf589d3ab80934db" } modify_external_catalogs: true
I20250901 01:52:36.851490 2118 ts_tablet_manager.cc:1918] T d27c8d99528a42ed8dd1c5a7f373bbae P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:36.851639 2118 log.cc:1199] T d27c8d99528a42ed8dd1c5a7f373bbae P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/d27c8d99528a42ed8dd1c5a7f373bbae
I20250901 01:52:36.851660 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_eb6bdc437a114aa88ef8eab1dc1b7ba4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.851976 2118 ts_tablet_manager.cc:1939] T d27c8d99528a42ed8dd1c5a7f373bbae P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:36.852734 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 307) took 3634 ms. Trace:
I20250901 01:52:36.852864 2118 rpcz_store.cc:276] 0901 01:52:33.217969 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:33.218009 (+ 40us) service_pool.cc:225] Handling call
0901 01:52:36.825796 (+3607787us) ts_tablet_manager.cc:1145] Deleting tablet d27c8d99528a42ed8dd1c5a7f373bbae
0901 01:52:36.825806 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:36.837034 (+ 11228us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:36.851399 (+ 14365us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:36.852716 (+ 1317us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3607638}
I20250901 01:52:36.853312 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet d27c8d99528a42ed8dd1c5a7f373bbae (table default.Table_d6b752d82c2b4be88b762e67bac0d5a1 [id=ce82be9787f14aa6bcdcb070228abc51]) successfully deleted
I20250901 01:52:36.853689 2113 ts_tablet_manager.cc:1918] T d4c9aa723aee4dc4b6237cb5c3722b52 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:36.853844 2113 log.cc:1199] T d4c9aa723aee4dc4b6237cb5c3722b52 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/d4c9aa723aee4dc4b6237cb5c3722b52
I20250901 01:52:36.854180 2113 ts_tablet_manager.cc:1939] T d4c9aa723aee4dc4b6237cb5c3722b52 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:36.854905 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 429) took 3968 ms. Trace:
I20250901 01:52:36.856045 2113 rpcz_store.cc:276] 0901 01:52:32.886092 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:32.886332 (+ 240us) service_pool.cc:225] Handling call
0901 01:52:36.796795 (+3910463us) ts_tablet_manager.cc:1145] Deleting tablet d4c9aa723aee4dc4b6237cb5c3722b52
0901 01:52:36.796805 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:36.828009 (+ 31204us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:36.850883 (+ 22874us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:36.854887 (+ 4004us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3910330}
I20250901 01:52:36.856771 2113 tablet_replica.cc:331] T 88ceba7f48b04d7885cc0164668700a5 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:36.857007 2113 raft_consensus.cc:2241] T 88ceba7f48b04d7885cc0164668700a5 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:36.857120 2753 raft_consensus.cc:491] T ced4813805004bba9c4667a79ac201fc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:36.857234 2753 raft_consensus.cc:513] T ced4813805004bba9c4667a79ac201fc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:36.857771 2753 leader_election.cc:290] T ced4813805004bba9c4667a79ac201fc P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447)
I20250901 01:52:36.858824 1055 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ced4813805004bba9c4667a79ac201fc" candidate_uuid: "00c282f779c1477d920f24dc0dc16628" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd27ab6665304aaea53adbd102dd4f58" is_pre_election: true
I20250901 01:52:36.859567 2911 raft_consensus.cc:491] T 5dd0c6b806574caf9126dad94833fcf7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:36.859691 2911 raft_consensus.cc:513] T 5dd0c6b806574caf9126dad94833fcf7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:36.860200 2911 leader_election.cc:290] T 5dd0c6b806574caf9126dad94833fcf7 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:36.860188 859 leader_election.cc:343] T ced4813805004bba9c4667a79ac201fc P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): Illegal state: must be running to vote when last-logged opid is not known
I20250901 01:52:36.860714 2113 raft_consensus.cc:2270] T 88ceba7f48b04d7885cc0164668700a5 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250901 01:52:36.861306 858 leader_election.cc:336] T ced4813805004bba9c4667a79ac201fc P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:36.861337 918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5dd0c6b806574caf9126dad94833fcf7" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:36.861406 858 leader_election.cc:304] T ced4813805004bba9c4667a79ac201fc P 00c282f779c1477d920f24dc0dc16628 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628; no voters: 0cf599f74322478d91475df234e58965, fd27ab6665304aaea53adbd102dd4f58
I20250901 01:52:36.861483 918 raft_consensus.cc:2466] T 5dd0c6b806574caf9126dad94833fcf7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:36.861960 2113 ts_tablet_manager.cc:1905] T 88ceba7f48b04d7885cc0164668700a5 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:36.861984 2753 raft_consensus.cc:2747] T ced4813805004bba9c4667a79ac201fc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:36.862416 992 leader_election.cc:304] T 5dd0c6b806574caf9126dad94833fcf7 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:36.863178 2911 raft_consensus.cc:2802] T 5dd0c6b806574caf9126dad94833fcf7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:36.863245 2911 raft_consensus.cc:491] T 5dd0c6b806574caf9126dad94833fcf7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:36.863296 2911 raft_consensus.cc:3058] T 5dd0c6b806574caf9126dad94833fcf7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
W20250901 01:52:36.864532 992 leader_election.cc:336] T 5dd0c6b806574caf9126dad94833fcf7 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:36.867722 2911 raft_consensus.cc:513] T 5dd0c6b806574caf9126dad94833fcf7 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:36.868793 2613 raft_consensus.cc:491] T da5f69c5b0ca4257ad2ff7918ee2af2e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:36.868906 2613 raft_consensus.cc:513] T da5f69c5b0ca4257ad2ff7918ee2af2e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:36.869113 2911 leader_election.cc:290] T 5dd0c6b806574caf9126dad94833fcf7 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
E20250901 01:52:36.869146 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 494b6488aa0d4355b4caa46004267465, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7c21b68ef2e74136a807c1dd90e17f4e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.869277 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 01deaae6abb141789f0cd240f3058c73, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7baa92b46d98454a891222be49a6ca48'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.869357 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 77911bcb98cf4d8890118eca9136f86e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_db61e055dbd7450898e82c4c0c4763be'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.869426 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ae4844183a3c4a04a775e6fd878391a3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_4bd66ab99a454974aa0f7f4290edfda2'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.869493 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9745582cebfe47b58c21d0191ab4cb9b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_77728af9eee44f29a1e2131d5b75b7ca'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.869588 1696 catalog_manager.cc:6205] Error selecting replicas for tablet e2ed32b3292141278af28026528e96c2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_6866a2f15cff496783cee46fc2895cb3'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.869657 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9f62a9d97f514747b940b488907369f3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d53a456dbca64daf92aeb5fb4517b417'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.869728 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5a7fa68d29644ae8afd2711e39dde59a, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d24e3a18ef4843fdb7b74ce23d47cf9e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.869789 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ff2bd017417c41b5b8225b8f00283e63, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0c7355975d6b44dbbe3a77df942b4d43'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:36.869757 918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5dd0c6b806574caf9126dad94833fcf7" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
E20250901 01:52:36.869851 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2821bf7750ba4e38b25ccba2e2bc0eab, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ce2f3b515c8f4c1bb557c5ed84b48dba'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.869922 1696 catalog_manager.cc:6205] Error selecting replicas for tablet dcc60bb000f440f1aca9133928b5ce20, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2fbca8463d6546c9b8f5b018fc60251d'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:36.869911 918 raft_consensus.cc:3058] T 5dd0c6b806574caf9126dad94833fcf7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
E20250901 01:52:36.869982 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 465c02a130c54e09b5a74fb4dd2ef762, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_bfddafe8e3204ce8bc9dd5934343ec59'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.870038 1696 catalog_manager.cc:6205] Error selecting replicas for tablet a5609f111ab34c0ea9fabdbce06c81f9, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_4505d77e4269443995012fef50c7e7b5'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.870096 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9f2416b2cc144054a8d2effc64157615, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_69575b1a9efd4573a2ee01d3f432800b'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.870164 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 75883d9fd00e429aa8853903a9fdd7e4, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_a9f6ac8609554a38ab632b2a3f7113d8'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.870241 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 8f4f5563df6b4d0db890747ba032ef9f, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_eaa0b271683343998d39a8ca0f151587'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.870337 1696 catalog_manager.cc:6205] Error selecting replicas for tablet b1ba0e19dc0b462e80c537bf2d183f5b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_f9cac02fe78f4902a8c730f2706f7139'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.870435 1696 catalog_manager.cc:6205] Error selecting replicas for tablet fd2ce82a4ee24d0faa44980a3967e187, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_63d0f2ead9d946d39ce40162ebd769d6'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.870556 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 086aa948f93644da8e5835b0fac87ddd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b4fbc34da806475c912052d0fab3fc53'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.870654 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 082db24bb3fd4089950504145a20a075, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ebba044988d5435aba124b1f971356d1'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.870757 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 93aa22afc0ad4d7fa24807ea3c5d5c21, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0f4b4094d2974b7591875375eda6bed0'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.870857 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 97418fffe6614dfba4ebd13a6ed57e67, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_360f4423e7f341339188b71cc82a1149'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.870939 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 3a62034e688348ed9e2ac9de009ed3a2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_90623cc9a13044b5934c922f14b85470'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.871016 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 18498c9cb09141d08c1c5c0854b63ebd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_e44ed688d0654ad5996005dc0b4b4b16'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:36.869467 2613 leader_election.cc:290] T da5f69c5b0ca4257ad2ff7918ee2af2e P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:36.870946 992 leader_election.cc:336] T 5dd0c6b806574caf9126dad94833fcf7 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
E20250901 01:52:36.871124 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f5e3316adeae47cd848833dd84b972bb, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_fc962c8bc653498e85cc0bb41639e9ad'. Need at least 3 replicas, but only 2 tablet servers are available
W20250901 01:52:36.871380 992 leader_election.cc:343] T da5f69c5b0ca4257ad2ff7918ee2af2e P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: da5f69c5b0ca4257ad2ff7918ee2af2e
I20250901 01:52:36.871680 2118 tablet_replica.cc:331] T 49aaaec611784e6bac131efd58dbca51 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:36.871903 2118 raft_consensus.cc:2241] T 49aaaec611784e6bac131efd58dbca51 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:36.872125 2118 raft_consensus.cc:2270] T 49aaaec611784e6bac131efd58dbca51 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:36.872817 918 raft_consensus.cc:2466] T 5dd0c6b806574caf9126dad94833fcf7 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:36.873808 1151 tablet_bootstrap.cc:492] T d0463a8d655d4db4800b4e1b221c8315 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
W20250901 01:52:36.873867 992 leader_election.cc:336] T da5f69c5b0ca4257ad2ff7918ee2af2e P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:36.874030 2118 ts_tablet_manager.cc:1905] T 49aaaec611784e6bac131efd58dbca51 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:36.874037 992 leader_election.cc:304] T da5f69c5b0ca4257ad2ff7918ee2af2e P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:36.874217 2911 raft_consensus.cc:2747] T da5f69c5b0ca4257ad2ff7918ee2af2e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:36.874414 992 leader_election.cc:304] T 5dd0c6b806574caf9126dad94833fcf7 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:36.874589 2911 raft_consensus.cc:2802] T 5dd0c6b806574caf9126dad94833fcf7 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:36.874831 1151 ts_tablet_manager.cc:1397] T d0463a8d655d4db4800b4e1b221c8315 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.000s
I20250901 01:52:36.875631 1151 raft_consensus.cc:357] T d0463a8d655d4db4800b4e1b221c8315 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:36.875824 1151 raft_consensus.cc:383] T d0463a8d655d4db4800b4e1b221c8315 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:36.875911 1151 raft_consensus.cc:738] T d0463a8d655d4db4800b4e1b221c8315 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:36.875910 2911 raft_consensus.cc:695] T 5dd0c6b806574caf9126dad94833fcf7 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:36.876187 1151 consensus_queue.cc:260] T d0463a8d655d4db4800b4e1b221c8315 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:36.876220 2911 consensus_queue.cc:237] T 5dd0c6b806574caf9126dad94833fcf7 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:36.876804 1151 ts_tablet_manager.cc:1428] T d0463a8d655d4db4800b4e1b221c8315 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250901 01:52:36.880148 1486 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_eaa0b271683343998d39a8ca0f151587" } new_table_name: "default.Table_5335fd8c942e40d68994187618d43eac" modify_external_catalogs: true
I20250901 01:52:36.882381 1490 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_ce2f3b515c8f4c1bb557c5ed84b48dba" } new_table_name: "default.Table_3e7f2097fe1e49b4b93c44b8f921fa47" modify_external_catalogs: true
I20250901 01:52:36.888144 1151 tablet_bootstrap.cc:492] T 315c15cf0afc44d4a79cb29294281d60 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:36.889942 1489 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 93bcac1fd95d44dc927abfca81692921
I20250901 01:52:36.890250 1151 tablet_bootstrap.cc:654] T 315c15cf0afc44d4a79cb29294281d60 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:36.890379 1489 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 93bcac1fd95d44dc927abfca81692921: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:36.891310 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 93bcac1fd95d44dc927abfca81692921 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:36 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:36.892501 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 93bcac1fd95d44dc927abfca81692921 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:36 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:36.894011 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f5a7f390ba8e4abba2046c496cdbc2dd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.895152 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8f046b8143cd4e8fa7cc4b11b8f0b603"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.896025 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e513885cdf8d40c995a0659ca93b74b2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.896885 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_be92438b2a9f453fb1fc57668d6f62ef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.896895 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3619b86d92b94247849383e12069faaa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.897924 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_668ef095ed334e8f89bf62335f9b3760"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.897989 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8749e20e60214b90adcfa747e693ab14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.898931 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_55d283f1468c4e5fabde61f25681152f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.899729 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_899440309cc446de84032ae98b78a6d4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.900523 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_6e02683da6ab47ed83704e0e861eabbb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.901355 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_6aa79e3adc1b4929b7d2042894b1cda7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:36.904055 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 6ce9521b925b4f97bb4a810ce77e147e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_666a4667dfa84c5ca860560cef32d55c'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:36.904235 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2f61edb50d7a42198bfd70639c7bf4de, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_cd56686217424ab8b95e6bb776d4177a'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:36.906750 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e4bd5f4b11994fcca0d3ccb95e39fd9e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.909535 2613 raft_consensus.cc:491] T c83ce95fd6fa41478de0e2168be201b1 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:36.909670 2613 raft_consensus.cc:513] T c83ce95fd6fa41478de0e2168be201b1 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:36.910187 2613 leader_election.cc:290] T c83ce95fd6fa41478de0e2168be201b1 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:36.911077 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_eb6d97e3de864105978e4e2ed1cc5541"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.912222 911 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c83ce95fd6fa41478de0e2168be201b1" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:36.912176 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_91946f55f0fa4a759fd6773791c18749"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.912429 911 raft_consensus.cc:2466] T c83ce95fd6fa41478de0e2168be201b1 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:36.913126 992 leader_election.cc:304] T c83ce95fd6fa41478de0e2168be201b1 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:36.913338 2948 raft_consensus.cc:2802] T c83ce95fd6fa41478de0e2168be201b1 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:36.913406 2948 raft_consensus.cc:491] T c83ce95fd6fa41478de0e2168be201b1 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:36.913465 2948 raft_consensus.cc:3058] T c83ce95fd6fa41478de0e2168be201b1 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:36.915068 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8baceb1b0fcc4b76aa72ca38f31b00f3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.915211 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2564ee374d0e41f195d8002af77ba640"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.915841 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_91d5a673d6754a28bb2b7270d5023a6e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.915870 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f860833ceec84191a7265cfbf31e4fb4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.915961 2948 raft_consensus.cc:513] T c83ce95fd6fa41478de0e2168be201b1 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:36.916615 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3d1ea1e67f3744ff9699361589140a26"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:36.918895 992 leader_election.cc:336] T c83ce95fd6fa41478de0e2168be201b1 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:36.919183 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7d28c3a23d364798b5a6e90833c658fa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.919607 2948 leader_election.cc:290] T c83ce95fd6fa41478de0e2168be201b1 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:36.920116 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_08f5991a256645a5a59d53fb35f919bd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:36.920221 992 leader_election.cc:336] T c83ce95fd6fa41478de0e2168be201b1 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:36.920509 911 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c83ce95fd6fa41478de0e2168be201b1" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:36.920735 911 raft_consensus.cc:3058] T c83ce95fd6fa41478de0e2168be201b1 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:36.920806 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_06047746849d4d7d8d3e7419d817317d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.921447 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b9e2422d09a74ff584efb7c703c1256d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.921777 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_85894ce5a8f14a11a2e5dd6270533880"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.922113 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_6fd06434f9bc4aaf84a23b52ae8fe913"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.922581 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4830c4ee98c4413396ede2c1ee5534a9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.923545 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_33377e8de6e24d68b7c3ed7a78f6b896"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.924108 911 raft_consensus.cc:2466] T c83ce95fd6fa41478de0e2168be201b1 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:36.924444 992 leader_election.cc:304] T c83ce95fd6fa41478de0e2168be201b1 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:36.924540 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c71f9b5ff9194eebb095aa89fb03beba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.925365 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7f565c4c4de246cb85fb752557b99a0f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.926241 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c14c8ba8d80142569c2b8aba7b03de98"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.926709 2948 raft_consensus.cc:2802] T c83ce95fd6fa41478de0e2168be201b1 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:36.926836 2948 raft_consensus.cc:695] T c83ce95fd6fa41478de0e2168be201b1 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:36.927073 2948 consensus_queue.cc:237] T c83ce95fd6fa41478de0e2168be201b1 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:36.929209 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f4062a87e31947de99740018e840a04d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.929934 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_79486f12cd46437dafcfe2719d1fda98"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.930589 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_737842b2f642410aba16cbd64a99145e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:36.931700 1486 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8f4f5563df6b4d0db890747ba032ef9f has no consensus state
I20250901 01:52:36.936003 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_759fe9a8e8ce4157bc967fbcffd4ef09"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.936441 2118 ts_tablet_manager.cc:1918] T 49aaaec611784e6bac131efd58dbca51 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 01:52:36.936597 2118 log.cc:1199] T 49aaaec611784e6bac131efd58dbca51 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/49aaaec611784e6bac131efd58dbca51
I20250901 01:52:36.937553 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_aa445a24e8dc44ff941c8f5c9041f7ae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.938696 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_08727739b7234d89af62a5c7ef0c7121"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.939594 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3795e76c55d247af8bb3e4e027f88ba7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.940541 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8d369919f7ff47578d9ec5e6466a30e3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.941690 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_50fed126bcec48d9847f13134f8fd004"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.946171 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_0a0e4e0a3d274b18acacdff392c950d1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.946930 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ff87df109e11495ba0c598461d7561b9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.947019 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_67fa54dda9734bb0abd26d0e82019a68"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.947376 2118 ts_tablet_manager.cc:1939] T 49aaaec611784e6bac131efd58dbca51 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:36.947796 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_779ea43f5b5948cebe004a9948297f5d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.948900 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_28b44b3a599c4365ba4f10c0e4876186"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.949206 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_cad77cce853346c6ab1801964cd2ef4e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.948886 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e517babbdf854c49b5c209e92544110f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.949940 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_44df7b5a9903421ba8057c3ba8879cac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.953451 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 308) took 3735 ms. Trace:
I20250901 01:52:36.952286 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_afa5d17cffd34954969779ca97130178"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.953564 2118 rpcz_store.cc:276] 0901 01:52:33.218348 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:33.218387 (+ 39us) service_pool.cc:225] Handling call
0901 01:52:36.871630 (+3653243us) ts_tablet_manager.cc:1145] Deleting tablet 49aaaec611784e6bac131efd58dbca51
0901 01:52:36.871641 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:36.890795 (+ 19154us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:36.936299 (+ 45504us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:36.953431 (+ 17132us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3635051}
I20250901 01:52:36.950884 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ae7b9669fb1c41c79557c6655cbb2d7e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.953908 2118 tablet_replica.cc:331] T 268439ff8d3a4f7a89f3d6395f2079c5 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:36.954080 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 49aaaec611784e6bac131efd58dbca51 (table default.Table_95e1338a49a04f22b679e363e253262e [id=e24b91b4ac904ef0a8c4725f70bced2f]) successfully deleted
I20250901 01:52:36.954186 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_20446fd811934092a28585e2ecacb111"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.955312 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f16e015261fb447c9c5e2b6979abc274"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.956041 1485 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_4bd66ab99a454974aa0f7f4290edfda2" } modify_external_catalogs: true
I20250901 01:52:36.956139 2113 ts_tablet_manager.cc:1918] T 88ceba7f48b04d7885cc0164668700a5 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:36.956205 1485 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_4bd66ab99a454974aa0f7f4290edfda2" } modify_external_catalogs: true
I20250901 01:52:36.956270 2113 log.cc:1199] T 88ceba7f48b04d7885cc0164668700a5 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/88ceba7f48b04d7885cc0164668700a5
I20250901 01:52:36.956629 2113 ts_tablet_manager.cc:1939] T 88ceba7f48b04d7885cc0164668700a5 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:36.956560 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_959559a7a46d431dabff4d7d8d783e9e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.957370 2954 tablet.cc:1722] T 9127f9c5239e445ba86012951b7c75d9 P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:36.957511 2118 raft_consensus.cc:2241] T 268439ff8d3a4f7a89f3d6395f2079c5 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:36.957676 2118 raft_consensus.cc:2270] T 268439ff8d3a4f7a89f3d6395f2079c5 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:36.957684 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_6256d2ac309547eead507a11467ba352"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.958232 2118 ts_tablet_manager.cc:1905] T 268439ff8d3a4f7a89f3d6395f2079c5 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:36.958442 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a83c763834024da6bb21e120cac37dc9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.959159 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7950b261e8b842f78cf85c1a43380a37"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.959882 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_6990835287114127b7a72e2b73652f11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.960611 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_0d82a3e1ac5241ecba2b4f55f213f213"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.951810 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e218364340c946fcb4adf320dfd4cc43"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.961432 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ebb34bc616d14bc3bbb0596440900627"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.961470 2956 tablet.cc:1722] T 9127f9c5239e445ba86012951b7c75d9 P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:36.961562 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 431) took 4075 ms. Trace:
I20250901 01:52:36.961634 2113 rpcz_store.cc:276] 0901 01:52:32.886272 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:32.886489 (+ 217us) service_pool.cc:225] Handling call
0901 01:52:36.856721 (+3970232us) ts_tablet_manager.cc:1145] Deleting tablet 88ceba7f48b04d7885cc0164668700a5
0901 01:52:36.856731 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:36.904145 (+ 47414us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:36.956018 (+ 51873us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:36.961546 (+ 5528us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3970108}
I20250901 01:52:36.961997 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 430) took 4075 ms. Trace:
I20250901 01:52:36.963436 2113 rpcz_store.cc:276] 0901 01:52:32.886170 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:32.886631 (+ 461us) service_pool.cc:225] Handling call
0901 01:52:36.961901 (+4075270us) ts_tablet_manager.cc:1145] Deleting tablet 7a607fac57994001b2304e5d81c9decc
0901 01:52:36.961911 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:36.961980 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4075153}
I20250901 01:52:36.963750 2113 tablet_replica.cc:331] T eee34044c39d4013baf78d3705489626 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
W20250901 01:52:36.964165 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 7a607fac57994001b2304e5d81c9decc because the tablet was not found. No further retry: Not found: Tablet not found: 7a607fac57994001b2304e5d81c9decc
W20250901 01:52:36.964430 992 consensus_peers.cc:489] T 615b2306fca84003b264da9d5f95ddab P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250901 01:52:36.965101 2113 raft_consensus.cc:2241] T eee34044c39d4013baf78d3705489626 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:36.965523 2113 raft_consensus.cc:2270] T eee34044c39d4013baf78d3705489626 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:36.965476 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b2442c86b55d43aba0994b78ccaa6d4a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.966665 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3bd5c9f9cd3645959dcf4cf8bd09a791"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.967803 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f589931279ed445597c301079d53780d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.966596 2113 ts_tablet_manager.cc:1905] T eee34044c39d4013baf78d3705489626 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:36.969473 1490 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2821bf7750ba4e38b25ccba2e2bc0eab has no consensus state
I20250901 01:52:36.969792 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_cb66ee3c76e4465eb32533e3eaa646f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.970520 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_402878e486ec42f3840c01baf06d0e7b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.971230 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_0073c7362c5f4361ac00c3c1ac560f6b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.974749 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b3ae508ad364451187c48193d63ff863"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.974888 2118 ts_tablet_manager.cc:1918] T 268439ff8d3a4f7a89f3d6395f2079c5 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:36.974996 2118 log.cc:1199] T 268439ff8d3a4f7a89f3d6395f2079c5 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/268439ff8d3a4f7a89f3d6395f2079c5
I20250901 01:52:36.975339 2118 ts_tablet_manager.cc:1939] T 268439ff8d3a4f7a89f3d6395f2079c5 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:36.975517 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2393ddbdfd994b6e93ed2253ab458415"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.975651 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 309) took 3541 ms. Trace:
I20250901 01:52:36.975723 2118 rpcz_store.cc:276] 0901 01:52:33.433765 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:33.433815 (+ 50us) service_pool.cc:225] Handling call
0901 01:52:36.953862 (+3520047us) ts_tablet_manager.cc:1145] Deleting tablet 268439ff8d3a4f7a89f3d6395f2079c5
0901 01:52:36.953871 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:36.968215 (+ 14344us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:36.974769 (+ 6554us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:36.975633 (+ 864us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3519860}
I20250901 01:52:36.976148 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 268439ff8d3a4f7a89f3d6395f2079c5 (table default.Table_b4fbc34da806475c912052d0fab3fc53 [id=3f1b378fb3444fd08dc947740ac0fa01]) successfully deleted
I20250901 01:52:36.978678 1485 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet ae4844183a3c4a04a775e6fd878391a3
I20250901 01:52:36.980244 1486 master_service.cc:948] ReplaceTablet: received request to replace tablet d0463a8d655d4db4800b4e1b221c8315 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:36.980244 1485 master_service.cc:948] ReplaceTablet: received request to replace tablet e9f1f892658b4736859ff507536920a1 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:36.984489 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_88509829c2e4464997843905ac3ee6ee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.984889 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_dda0e4a5ea044a2f97e9e5804117f715"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.984958 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d7fdad037e0f4e4c874088a7478cb446"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.985291 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_49ffaee33e434a428b4bbee4b30d30ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.986446 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a0644030a28e44e2a7305d54425df315"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.985263 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_0c4c96724d92467c867cb5ff272b5dff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.985666 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_29ccdf1c1f774c908ea24e972bff9fbb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.989317 2118 tablet_replica.cc:331] T d0463a8d655d4db4800b4e1b221c8315 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:36.990867 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a6dcc87b3c314d458d167fc9f5c1bc7c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.991081 2113 ts_tablet_manager.cc:1918] T eee34044c39d4013baf78d3705489626 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:36.991395 2118 raft_consensus.cc:2241] T d0463a8d655d4db4800b4e1b221c8315 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:36.991775 2118 raft_consensus.cc:2270] T d0463a8d655d4db4800b4e1b221c8315 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:36.942613 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_942945d76a8a4992b3f974d69db63a4f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.992427 2118 ts_tablet_manager.cc:1905] T d0463a8d655d4db4800b4e1b221c8315 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:36.995262 1483 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_5335fd8c942e40d68994187618d43eac" } new_table_name: "default.Table_2883135b3ab649a585f1158e8e527c13" modify_external_catalogs: true
I20250901 01:52:36.995886 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a5caef7cd883439fbe32a71047acf456"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.991272 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e0343ffd68e34e5dacef9b2da073355e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.995698 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5644238c56b94156b087c640b1ea2394"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.000063 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_31245b2bdeff48ad9b2528b19ecc8999"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.000772 2113 log.cc:1199] T eee34044c39d4013baf78d3705489626 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/eee34044c39d4013baf78d3705489626
I20250901 01:52:37.001148 2113 ts_tablet_manager.cc:1939] T eee34044c39d4013baf78d3705489626 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:36.991559 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_6b5548c4803e4162bc2e6f098be6688c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.001814 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b9dcbae1f00e46beb62108524ca70b4e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.002168 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5a402e523dd445c5a1f4aa36c99fcef2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.002511 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_6b53ba0b27d246dfacbe8085ca695752"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.002822 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_10d0a5eb29004df482d6d9b1af92ac40"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.003151 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a3a43068320e4d549e24a3d14c4534bc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:36.991441 1696 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet ae4844183a3c4a04a775e6fd878391a3
I20250901 01:52:36.995425 1491 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_e44ed688d0654ad5996005dc0b4b4b16" } new_table_name: "default.Table_df16f2a0280645a0949bc071a2af532f" modify_external_catalogs: true
I20250901 01:52:37.000607 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5c9adf27024f4f519ba1784658560f80"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.000942 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2e62123a04a448f48f9da5fa8a136ade"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.007897 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 432) took 4121 ms. Trace:
I20250901 01:52:37.007995 2113 rpcz_store.cc:276] 0901 01:52:32.886913 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:32.886955 (+ 42us) service_pool.cc:225] Handling call
0901 01:52:36.963706 (+4076751us) ts_tablet_manager.cc:1145] Deleting tablet eee34044c39d4013baf78d3705489626
0901 01:52:36.963716 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:36.984346 (+ 20630us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:36.990968 (+ 6622us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.007872 (+ 16904us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4076605}
I20250901 01:52:37.008338 2113 tablet_replica.cc:331] T da5f69c5b0ca4257ad2ff7918ee2af2e P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:37.008559 2113 raft_consensus.cc:2241] T da5f69c5b0ca4257ad2ff7918ee2af2e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:37.008704 2113 raft_consensus.cc:2270] T da5f69c5b0ca4257ad2ff7918ee2af2e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:37.009222 2113 ts_tablet_manager.cc:1905] T da5f69c5b0ca4257ad2ff7918ee2af2e P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:36.991168 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4c4aed6c88024713b526f9580a0b176e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.008608 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet eee34044c39d4013baf78d3705489626 (table default.Table_99afdac28a4e4ac9ae93b4fbbe704a4d [id=4639e91c14d8427d86c89d3c2b0918ad]) successfully deleted
I20250901 01:52:37.009516 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_256b3f349210446fa9b59be95f37bd7b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.013713 1486 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: default.Table_335c0676987d4bf3a4d64b239be8067e [id=ab670bb02c6e4ed491e7831ad73cdfc6] alter complete (version 1)
I20250901 01:52:37.015902 1151 tablet_bootstrap.cc:492] T 315c15cf0afc44d4a79cb29294281d60 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:37.016067 1151 ts_tablet_manager.cc:1397] T 315c15cf0afc44d4a79cb29294281d60 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.128s user 0.008s sys 0.004s
I20250901 01:52:37.016916 1151 raft_consensus.cc:357] T 315c15cf0afc44d4a79cb29294281d60 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:37.017120 1151 raft_consensus.cc:383] T 315c15cf0afc44d4a79cb29294281d60 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:37.017186 1151 raft_consensus.cc:738] T 315c15cf0afc44d4a79cb29294281d60 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
W20250901 01:52:37.017306 1491 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 18498c9cb09141d08c1c5c0854b63ebd has no consensus state
I20250901 01:52:37.017396 1151 consensus_queue.cc:260] T 315c15cf0afc44d4a79cb29294281d60 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:37.018301 1151 ts_tablet_manager.cc:1428] T 315c15cf0afc44d4a79cb29294281d60 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250901 01:52:37.018757 1151 tablet_bootstrap.cc:492] T c422881b40f54971a68530abfbd1945b P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:37.021065 1151 tablet_bootstrap.cc:654] T c422881b40f54971a68530abfbd1945b P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:37.021432 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_0c4f0c9b3b1b40a7a7ae402909c06e49"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.022922 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_70f17fedb8c241db99c3dfda057cdf4b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.024081 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7f12bb5ed2384626801f87e5ee454efb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.031523 2118 ts_tablet_manager.cc:1918] T d0463a8d655d4db4800b4e1b221c8315 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:37.035480 2118 log.cc:1199] T d0463a8d655d4db4800b4e1b221c8315 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/d0463a8d655d4db4800b4e1b221c8315
I20250901 01:52:37.037876 2118 ts_tablet_manager.cc:1939] T d0463a8d655d4db4800b4e1b221c8315 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:37.038761 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 310) took 3604 ms. Trace:
I20250901 01:52:37.038918 2118 rpcz_store.cc:276] 0901 01:52:33.434288 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:33.434333 (+ 45us) service_pool.cc:225] Handling call
0901 01:52:36.989262 (+3554929us) ts_tablet_manager.cc:1145] Deleting tablet d0463a8d655d4db4800b4e1b221c8315
0901 01:52:36.989271 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.015482 (+ 26211us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.031419 (+ 15937us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.038740 (+ 7321us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3554767}
W20250901 01:52:37.039234 1483 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8f4f5563df6b4d0db890747ba032ef9f has no consensus state
I20250901 01:52:37.039741 2118 tablet_replica.cc:331] T f65ec84958eb488d8b0f812b59b5b3d5 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:37.039994 2118 raft_consensus.cc:2241] T f65ec84958eb488d8b0f812b59b5b3d5 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:37.040220 2118 raft_consensus.cc:2270] T f65ec84958eb488d8b0f812b59b5b3d5 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:37.040824 2118 ts_tablet_manager.cc:1905] T f65ec84958eb488d8b0f812b59b5b3d5 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:37.044759 1484 master_service.cc:948] ReplaceTablet: received request to replace tablet ea846f88b3534beebd0001b7f17b5ca9 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:37.045120 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3ed1012f264241758dd0cc74ec9c8e82"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.045316 1486 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_db61e055dbd7450898e82c4c0c4763be" } modify_external_catalogs: true
I20250901 01:52:37.045475 1486 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_db61e055dbd7450898e82c4c0c4763be" } modify_external_catalogs: true
I20250901 01:52:37.045532 1492 master_service.cc:948] ReplaceTablet: received request to replace tablet b164b4a67a9143ceaff69b0ce49d63fe from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:37.046089 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_609722b715604b9a8fd689705b9af600"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.046613 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_84bde13532ad45798a75b6ef3e758a0c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.046648 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_aa6d67c58a034e8991ca743f3793785b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.046829 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d5884005e9664858a68285309d330b0e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.047410 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c5998a573fbf4b8995a9f4038c511eba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.048044 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_6f1e2dfff7514e49ab7592f8954ef2b6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.049232 1483 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_77728af9eee44f29a1e2131d5b75b7ca" } modify_external_catalogs: true
I20250901 01:52:37.049383 1483 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_77728af9eee44f29a1e2131d5b75b7ca" } modify_external_catalogs: true
I20250901 01:52:37.050045 2118 ts_tablet_manager.cc:1918] T f65ec84958eb488d8b0f812b59b5b3d5 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250901 01:52:37.050143 2118 log.cc:1199] T f65ec84958eb488d8b0f812b59b5b3d5 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/f65ec84958eb488d8b0f812b59b5b3d5
I20250901 01:52:37.051031 2118 ts_tablet_manager.cc:1939] T f65ec84958eb488d8b0f812b59b5b3d5 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:37.051766 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 311) took 3607 ms. Trace:
I20250901 01:52:37.051859 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_53e2dcb107f444c898c96986958b8a52"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.051919 2118 rpcz_store.cc:276] 0901 01:52:33.443892 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:33.443945 (+ 53us) service_pool.cc:225] Handling call
0901 01:52:37.039694 (+3595749us) ts_tablet_manager.cc:1145] Deleting tablet f65ec84958eb488d8b0f812b59b5b3d5
0901 01:52:37.039704 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.044555 (+ 4851us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.049954 (+ 5399us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.051750 (+ 1796us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3595583}
W20250901 01:52:37.052320 992 consensus_peers.cc:489] T aa424ac6ef1441e6b1cc2867b1c48186 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250901 01:52:37.041033 2113 ts_tablet_manager.cc:1918] T da5f69c5b0ca4257ad2ff7918ee2af2e P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:37.052631 2113 log.cc:1199] T da5f69c5b0ca4257ad2ff7918ee2af2e P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/da5f69c5b0ca4257ad2ff7918ee2af2e
I20250901 01:52:37.052663 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet f65ec84958eb488d8b0f812b59b5b3d5 (table default.Table_85b90ce84a904990b15be24c0acea21a [id=d1cd5288671540c0a95f7a037363014e]) successfully deleted
I20250901 01:52:37.052951 2113 ts_tablet_manager.cc:1939] T da5f69c5b0ca4257ad2ff7918ee2af2e P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:37.053375 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4d3a32f45d35403083c28f2c67e1326a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.053659 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 433) took 4166 ms. Trace:
I20250901 01:52:37.053733 2113 rpcz_store.cc:276] 0901 01:52:32.887014 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:32.887134 (+ 120us) service_pool.cc:225] Handling call
0901 01:52:37.008285 (+4121151us) ts_tablet_manager.cc:1145] Deleting tablet da5f69c5b0ca4257ad2ff7918ee2af2e
0901 01:52:37.008296 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.031726 (+ 23430us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.034675 (+ 2949us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.053642 (+ 18967us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4121011}
I20250901 01:52:37.054064 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 434) took 4166 ms. Trace:
I20250901 01:52:37.054131 2113 rpcz_store.cc:276] 0901 01:52:32.887159 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:32.887295 (+ 136us) service_pool.cc:225] Handling call
0901 01:52:37.053974 (+4166679us) ts_tablet_manager.cc:1145] Deleting tablet ddbc15685fca4c708329420109ad0c57
0901 01:52:37.053983 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.054050 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4166553}
I20250901 01:52:37.054407 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 435) took 4167 ms. Trace:
I20250901 01:52:37.054469 2113 rpcz_store.cc:276] 0901 01:52:32.887296 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:32.887446 (+ 150us) service_pool.cc:225] Handling call
0901 01:52:37.054336 (+4166890us) ts_tablet_manager.cc:1145] Deleting tablet 0dbdcb76eeba4fefa56e7835c23c0316
0901 01:52:37.054343 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.054394 (+ 51us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4166769}
I20250901 01:52:37.054754 2113 tablet_replica.cc:331] T 3363fd67b1ad44849ecb7b4090b61a72 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:37.054935 2113 raft_consensus.cc:2241] T 3363fd67b1ad44849ecb7b4090b61a72 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:37.055063 2113 raft_consensus.cc:2270] T 3363fd67b1ad44849ecb7b4090b61a72 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:37.055742 1486 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 77911bcb98cf4d8890118eca9136f86e
I20250901 01:52:37.056152 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet da5f69c5b0ca4257ad2ff7918ee2af2e (table default.Table_ecb9e8f638a043cc87c38afbebce250f [id=61f1e8a842844ed899e5913c76f362ff]) successfully deleted
W20250901 01:52:37.056346 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet ddbc15685fca4c708329420109ad0c57 because the tablet was not found. No further retry: Not found: Tablet not found: ddbc15685fca4c708329420109ad0c57
W20250901 01:52:37.056547 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 0dbdcb76eeba4fefa56e7835c23c0316 because the tablet was not found. No further retry: Not found: Tablet not found: 0dbdcb76eeba4fefa56e7835c23c0316
I20250901 01:52:37.055513 2113 ts_tablet_manager.cc:1905] T 3363fd67b1ad44849ecb7b4090b61a72 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:37.059401 1483 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 9745582cebfe47b58c21d0191ab4cb9b
I20250901 01:52:37.060079 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_10f49db26d9a4556b6f08a5353a061da"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.060367 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_22633728f63d42c68ea6a4677ba45636"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.060703 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2e4dde7c5e53497fa6faa46768da7f63"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.061142 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3ca936d279394f52bae8ad39763fa4c6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.061622 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_cdc3f33a83c1451b9c648bd6bff19dce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.061890 1488 master_service.cc:948] ReplaceTablet: received request to replace tablet 92f0ad59fbbb435ea872c669a52f854a from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:37.046118 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a53c3c9efc2d4b8eb21322e22eefa011"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.062479 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e06cc96016eb47798471e3777a450df1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.064474 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9f54bee670754ee6b55183d336555721"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.065819 2113 ts_tablet_manager.cc:1918] T 3363fd67b1ad44849ecb7b4090b61a72 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:37.066599 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9279a2f5fb454b55b5fb7cc864216c50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:37.067224 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 494b6488aa0d4355b4caa46004267465, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7c21b68ef2e74136a807c1dd90e17f4e'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.067566 2118 tablet_replica.cc:331] T 6ba88af177874544b764cc32420e7ac9 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:37.067843 2118 raft_consensus.cc:2241] T 6ba88af177874544b764cc32420e7ac9 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
E20250901 01:52:37.068029 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 01deaae6abb141789f0cd240f3058c73, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7baa92b46d98454a891222be49a6ca48'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.068255 2118 raft_consensus.cc:2270] T 6ba88af177874544b764cc32420e7ac9 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
E20250901 01:52:37.068382 1696 catalog_manager.cc:6205] Error selecting replicas for tablet e2ed32b3292141278af28026528e96c2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_6866a2f15cff496783cee46fc2895cb3'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.068536 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9f62a9d97f514747b940b488907369f3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d53a456dbca64daf92aeb5fb4517b417'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.069089 2118 ts_tablet_manager.cc:1905] T 6ba88af177874544b764cc32420e7ac9 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
E20250901 01:52:37.069234 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5a7fa68d29644ae8afd2711e39dde59a, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d24e3a18ef4843fdb7b74ce23d47cf9e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.069344 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ff2bd017417c41b5b8225b8f00283e63, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0c7355975d6b44dbbe3a77df942b4d43'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.069453 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2821bf7750ba4e38b25ccba2e2bc0eab, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_3e7f2097fe1e49b4b93c44b8f921fa47'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.069555 1696 catalog_manager.cc:6205] Error selecting replicas for tablet dcc60bb000f440f1aca9133928b5ce20, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2fbca8463d6546c9b8f5b018fc60251d'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.069658 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 465c02a130c54e09b5a74fb4dd2ef762, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_bfddafe8e3204ce8bc9dd5934343ec59'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.069762 1696 catalog_manager.cc:6205] Error selecting replicas for tablet a5609f111ab34c0ea9fabdbce06c81f9, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_4505d77e4269443995012fef50c7e7b5'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.069886 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9f2416b2cc144054a8d2effc64157615, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_69575b1a9efd4573a2ee01d3f432800b'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.069993 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 75883d9fd00e429aa8853903a9fdd7e4, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_a9f6ac8609554a38ab632b2a3f7113d8'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.073887 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 8f4f5563df6b4d0db890747ba032ef9f, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2883135b3ab649a585f1158e8e527c13'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.074065 1696 catalog_manager.cc:6205] Error selecting replicas for tablet b1ba0e19dc0b462e80c537bf2d183f5b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_f9cac02fe78f4902a8c730f2706f7139'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.074209 2113 log.cc:1199] T 3363fd67b1ad44849ecb7b4090b61a72 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/3363fd67b1ad44849ecb7b4090b61a72
E20250901 01:52:37.074326 1696 catalog_manager.cc:6205] Error selecting replicas for tablet fd2ce82a4ee24d0faa44980a3967e187, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_63d0f2ead9d946d39ce40162ebd769d6'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.073987 1151 tablet_bootstrap.cc:492] T c422881b40f54971a68530abfbd1945b P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:37.075192 2113 ts_tablet_manager.cc:1939] T 3363fd67b1ad44849ecb7b4090b61a72 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:37.075160 1151 ts_tablet_manager.cc:1397] T c422881b40f54971a68530abfbd1945b P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.057s user 0.004s sys 0.000s
I20250901 01:52:37.075913 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 436) took 4188 ms. Trace:
I20250901 01:52:37.076004 2113 rpcz_store.cc:276] 0901 01:52:32.887462 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:32.887592 (+ 130us) service_pool.cc:225] Handling call
0901 01:52:37.054702 (+4167110us) ts_tablet_manager.cc:1145] Deleting tablet 3363fd67b1ad44849ecb7b4090b61a72
0901 01:52:37.054712 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.062890 (+ 8178us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.065729 (+ 2839us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.075890 (+ 10161us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4166983}
E20250901 01:52:37.076550 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 086aa948f93644da8e5835b0fac87ddd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b4fbc34da806475c912052d0fab3fc53'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.076424 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 3363fd67b1ad44849ecb7b4090b61a72 (table default.Table_09ffc2a7b4b14b25adbc5471363c02af [id=eaa8142db4844b908cf36991898aca3c]) successfully deleted
I20250901 01:52:37.066570 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_13bf70eefc3549dbb3a15468c4c53bc4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.076561 1151 raft_consensus.cc:357] T c422881b40f54971a68530abfbd1945b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:37.077030 1151 raft_consensus.cc:383] T c422881b40f54971a68530abfbd1945b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:37.077162 1151 raft_consensus.cc:738] T c422881b40f54971a68530abfbd1945b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
E20250901 01:52:37.077450 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 082db24bb3fd4089950504145a20a075, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ebba044988d5435aba124b1f971356d1'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.077580 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 93aa22afc0ad4d7fa24807ea3c5d5c21, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0f4b4094d2974b7591875375eda6bed0'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.077606 1151 consensus_queue.cc:260] T c422881b40f54971a68530abfbd1945b P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
E20250901 01:52:37.077849 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 97418fffe6614dfba4ebd13a6ed57e67, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_360f4423e7f341339188b71cc82a1149'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.077945 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 3a62034e688348ed9e2ac9de009ed3a2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_90623cc9a13044b5934c922f14b85470'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.078042 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 18498c9cb09141d08c1c5c0854b63ebd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_df16f2a0280645a0949bc071a2af532f'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.078128 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f5e3316adeae47cd848833dd84b972bb, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_fc962c8bc653498e85cc0bb41639e9ad'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.078207 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 6ce9521b925b4f97bb4a810ce77e147e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_666a4667dfa84c5ca860560cef32d55c'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.078904 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2f61edb50d7a42198bfd70639c7bf4de, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_cd56686217424ab8b95e6bb776d4177a'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.080677 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 437) took 4192 ms. Trace:
I20250901 01:52:37.080775 2113 rpcz_store.cc:276] 0901 01:52:32.888405 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:32.888461 (+ 56us) service_pool.cc:225] Handling call
0901 01:52:37.080570 (+4192109us) ts_tablet_manager.cc:1145] Deleting tablet f62d96c37cca4143a4b098b7c69979e6
0901 01:52:37.080581 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.080660 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4191938}
W20250901 01:52:37.081211 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet f62d96c37cca4143a4b098b7c69979e6 because the tablet was not found. No further retry: Not found: Tablet not found: f62d96c37cca4143a4b098b7c69979e6
I20250901 01:52:37.081580 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 438) took 4193 ms. Trace:
I20250901 01:52:37.081679 2113 rpcz_store.cc:276] 0901 01:52:32.888529 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:32.888669 (+ 140us) service_pool.cc:225] Handling call
0901 01:52:37.081499 (+4192830us) ts_tablet_manager.cc:1145] Deleting tablet 7cc1efcb6bff4aba8015d2d3b7f4c923
0901 01:52:37.081508 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.081565 (+ 57us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4192691}
W20250901 01:52:37.082046 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 7cc1efcb6bff4aba8015d2d3b7f4c923 because the tablet was not found. No further retry: Not found: Tablet not found: 7cc1efcb6bff4aba8015d2d3b7f4c923
I20250901 01:52:37.082329 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 439) took 4193 ms. Trace:
I20250901 01:52:37.083079 2113 rpcz_store.cc:276] 0901 01:52:32.888678 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:32.888819 (+ 141us) service_pool.cc:225] Handling call
0901 01:52:37.082255 (+4193436us) ts_tablet_manager.cc:1145] Deleting tablet 500e3c33598b45a4afa983da7cb0845a
0901 01:52:37.082263 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.082314 (+ 51us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4193331}
W20250901 01:52:37.083443 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 500e3c33598b45a4afa983da7cb0845a because the tablet was not found. No further retry: Not found: Tablet not found: 500e3c33598b45a4afa983da7cb0845a
I20250901 01:52:37.083736 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 440) took 4194 ms. Trace:
I20250901 01:52:37.083824 2113 rpcz_store.cc:276] 0901 01:52:32.888818 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:32.888937 (+ 119us) service_pool.cc:225] Handling call
0901 01:52:37.083668 (+4194731us) ts_tablet_manager.cc:1145] Deleting tablet 831e42de95854541a49ae6b3e039d3e9
0901 01:52:37.083676 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.083723 (+ 47us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4194635}
W20250901 01:52:37.084282 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 831e42de95854541a49ae6b3e039d3e9 because the tablet was not found. No further retry: Not found: Tablet not found: 831e42de95854541a49ae6b3e039d3e9
I20250901 01:52:37.085299 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 441) took 4196 ms. Trace:
I20250901 01:52:37.085932 2113 rpcz_store.cc:276] 0901 01:52:32.888970 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:32.889046 (+ 76us) service_pool.cc:225] Handling call
0901 01:52:37.085223 (+4196177us) ts_tablet_manager.cc:1145] Deleting tablet 72ba689f9230411995dc96e2b45601df
0901 01:52:37.085232 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.085282 (+ 50us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4196081}
W20250901 01:52:37.086324 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 72ba689f9230411995dc96e2b45601df because the tablet was not found. No further retry: Not found: Tablet not found: 72ba689f9230411995dc96e2b45601df
I20250901 01:52:37.088043 1151 ts_tablet_manager.cc:1428] T c422881b40f54971a68530abfbd1945b P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20250901 01:52:37.091872 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5b4b26b9ed0a42569f07726bcb65f5d5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.092170 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b92964f0a5dc44eba1012e10deb21728"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.093004 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_63cb3e50fe304ee9916ce9a85ae1d198"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.093439 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet c422881b40f54971a68530abfbd1945b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:28 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:37.093724 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_187ea073e34f4fb18cde4a287e306c64"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:37.098655 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 494b6488aa0d4355b4caa46004267465, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7c21b68ef2e74136a807c1dd90e17f4e'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.086820 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 442) took 4197 ms. Trace:
I20250901 01:52:37.098860 2113 rpcz_store.cc:276] 0901 01:52:32.889161 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:32.889194 (+ 33us) service_pool.cc:225] Handling call
0901 01:52:37.086470 (+4197276us) ts_tablet_manager.cc:1145] Deleting tablet c754129977904f978f87fd89d7092e36
0901 01:52:37.086478 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.086805 (+ 327us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4197174}
I20250901 01:52:37.099273 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 443) took 4210 ms. Trace:
I20250901 01:52:37.105741 2113 rpcz_store.cc:276] 0901 01:52:32.889330 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:32.889362 (+ 32us) service_pool.cc:225] Handling call
0901 01:52:37.099181 (+4209819us) ts_tablet_manager.cc:1145] Deleting tablet eaa521c52a2f4408b59770bd2a05907c
0901 01:52:37.099190 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.099255 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4209717}
W20250901 01:52:37.105659 992 consensus_peers.cc:489] T 49aaaec611784e6bac131efd58dbca51 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Couldn't send request to peer 00c282f779c1477d920f24dc0dc16628. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 49aaaec611784e6bac131efd58dbca51. This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:37.106829 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 444) took 4217 ms. Trace:
I20250901 01:52:37.106978 2113 rpcz_store.cc:276] 0901 01:52:32.889493 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:32.889524 (+ 31us) service_pool.cc:225] Handling call
0901 01:52:37.106744 (+4217220us) ts_tablet_manager.cc:1145] Deleting tablet 2599f98abfdc4871b573edfc2ad97d22
0901 01:52:37.106755 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.106813 (+ 58us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4217085}
I20250901 01:52:37.102662 1151 tablet_bootstrap.cc:492] T 98c56eb62ad84f79b04f17a43d44c479 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:37.107447 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 445) took 4217 ms. Trace:
I20250901 01:52:37.107569 2113 rpcz_store.cc:276] 0901 01:52:32.889684 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:32.889715 (+ 31us) service_pool.cc:225] Handling call
0901 01:52:37.107377 (+4217662us) ts_tablet_manager.cc:1145] Deleting tablet 8ed6616ce54548ba83ba299c4ecf3c38
0901 01:52:37.107386 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.107435 (+ 49us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4217546}
I20250901 01:52:37.107992 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 446) took 4218 ms. Trace:
I20250901 01:52:37.108103 2113 rpcz_store.cc:276] 0901 01:52:32.889887 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:32.889920 (+ 33us) service_pool.cc:225] Handling call
0901 01:52:37.107928 (+4218008us) ts_tablet_manager.cc:1145] Deleting tablet c58edaa76ebf47dfb1d9d5fab6de7dfa
0901 01:52:37.107937 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.107980 (+ 43us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4217881}
I20250901 01:52:37.108521 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 447) took 4218 ms. Trace:
W20250901 01:52:37.099362 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet c754129977904f978f87fd89d7092e36 because the tablet was not found. No further retry: Not found: Tablet not found: c754129977904f978f87fd89d7092e36
E20250901 01:52:37.108717 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 01deaae6abb141789f0cd240f3058c73, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7baa92b46d98454a891222be49a6ca48'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.108799 1696 catalog_manager.cc:6205] Error selecting replicas for tablet e2ed32b3292141278af28026528e96c2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_6866a2f15cff496783cee46fc2895cb3'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.108860 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9f62a9d97f514747b940b488907369f3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d53a456dbca64daf92aeb5fb4517b417'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.108929 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5a7fa68d29644ae8afd2711e39dde59a, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d24e3a18ef4843fdb7b74ce23d47cf9e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.108989 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ff2bd017417c41b5b8225b8f00283e63, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0c7355975d6b44dbbe3a77df942b4d43'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.109072 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2821bf7750ba4e38b25ccba2e2bc0eab, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_3e7f2097fe1e49b4b93c44b8f921fa47'. Need at least 3 replicas, but only 2 tablet servers are available
W20250901 01:52:37.109145 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet eaa521c52a2f4408b59770bd2a05907c because the tablet was not found. No further retry: Not found: Tablet not found: eaa521c52a2f4408b59770bd2a05907c
W20250901 01:52:37.109339 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 2599f98abfdc4871b573edfc2ad97d22 because the tablet was not found. No further retry: Not found: Tablet not found: 2599f98abfdc4871b573edfc2ad97d22
W20250901 01:52:37.109495 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 8ed6616ce54548ba83ba299c4ecf3c38 because the tablet was not found. No further retry: Not found: Tablet not found: 8ed6616ce54548ba83ba299c4ecf3c38
I20250901 01:52:37.109575 2118 ts_tablet_manager.cc:1918] T 6ba88af177874544b764cc32420e7ac9 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250901 01:52:37.109656 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet c58edaa76ebf47dfb1d9d5fab6de7dfa because the tablet was not found. No further retry: Not found: Tablet not found: c58edaa76ebf47dfb1d9d5fab6de7dfa
I20250901 01:52:37.109696 2118 log.cc:1199] T 6ba88af177874544b764cc32420e7ac9 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/6ba88af177874544b764cc32420e7ac9
I20250901 01:52:37.103022 1483 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_ea8e4576448a476b8bf399421a378e2c" } modify_external_catalogs: true
I20250901 01:52:37.110064 2118 ts_tablet_manager.cc:1939] T 6ba88af177874544b764cc32420e7ac9 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:37.110167 2113 rpcz_store.cc:276] 0901 01:52:32.890276 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:32.890312 (+ 36us) service_pool.cc:225] Handling call
0901 01:52:37.108457 (+4218145us) ts_tablet_manager.cc:1145] Deleting tablet e249474dcbf642189560c91ec3a99fdf
0901 01:52:37.108466 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.108509 (+ 43us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4217976}
I20250901 01:52:37.109956 1151 tablet_bootstrap.cc:654] T 98c56eb62ad84f79b04f17a43d44c479 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:37.103592 1492 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_50ff85e8f1314f4499e76299441bc110" } new_table_name: "default.Table_1888ad759a544e5a9988201f9ae538ee" modify_external_catalogs: true
I20250901 01:52:37.103739 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1a79b76f94e042eea114de0217d59fb6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.111491 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 449) took 4214 ms. Trace:
I20250901 01:52:37.111560 2113 rpcz_store.cc:276] 0901 01:52:32.896895 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:32.896948 (+ 53us) service_pool.cc:225] Handling call
0901 01:52:37.111422 (+4214474us) ts_tablet_manager.cc:1145] Deleting tablet 5e2472ee5d6e4d3bbf1f24027ba531ca
0901 01:52:37.111431 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.111478 (+ 47us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4214308}
E20250901 01:52:37.109149 1696 catalog_manager.cc:6205] Error selecting replicas for tablet dcc60bb000f440f1aca9133928b5ce20, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2fbca8463d6546c9b8f5b018fc60251d'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.111910 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 465c02a130c54e09b5a74fb4dd2ef762, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_bfddafe8e3204ce8bc9dd5934343ec59'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.111984 1696 catalog_manager.cc:6205] Error selecting replicas for tablet a5609f111ab34c0ea9fabdbce06c81f9, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_4505d77e4269443995012fef50c7e7b5'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.112042 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9f2416b2cc144054a8d2effc64157615, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_69575b1a9efd4573a2ee01d3f432800b'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.111944 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_95e02c7f408646c59cc80743f0f03159"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:37.112098 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 75883d9fd00e429aa8853903a9fdd7e4, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_a9f6ac8609554a38ab632b2a3f7113d8'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.112154 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 8f4f5563df6b4d0db890747ba032ef9f, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2883135b3ab649a585f1158e8e527c13'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.112210 1696 catalog_manager.cc:6205] Error selecting replicas for tablet b1ba0e19dc0b462e80c537bf2d183f5b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_f9cac02fe78f4902a8c730f2706f7139'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.112267 1696 catalog_manager.cc:6205] Error selecting replicas for tablet fd2ce82a4ee24d0faa44980a3967e187, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_63d0f2ead9d946d39ce40162ebd769d6'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.112329 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 086aa948f93644da8e5835b0fac87ddd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b4fbc34da806475c912052d0fab3fc53'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.112386 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 082db24bb3fd4089950504145a20a075, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ebba044988d5435aba124b1f971356d1'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.112475 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 93aa22afc0ad4d7fa24807ea3c5d5c21, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0f4b4094d2974b7591875375eda6bed0'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.104080 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_cd2fdb5182ee447394137b9b09a81bdf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:37.112545 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 97418fffe6614dfba4ebd13a6ed57e67, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_360f4423e7f341339188b71cc82a1149'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.112630 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 3a62034e688348ed9e2ac9de009ed3a2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_90623cc9a13044b5934c922f14b85470'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.112695 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 18498c9cb09141d08c1c5c0854b63ebd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_df16f2a0280645a0949bc071a2af532f'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.112658 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_42edd20995554a8d9f12486f1884a264"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:37.112751 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f5e3316adeae47cd848833dd84b972bb, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_fc962c8bc653498e85cc0bb41639e9ad'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.112840 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 6ce9521b925b4f97bb4a810ce77e147e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_666a4667dfa84c5ca860560cef32d55c'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.112898 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2f61edb50d7a42198bfd70639c7bf4de, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_cd56686217424ab8b95e6bb776d4177a'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.113135 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9cdbcccc1f274cfa9712fce9726ff156"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.113780 2113 tablet_replica.cc:331] T 3239e58531d445fa80165cf678bc277c P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:37.112526 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_70b8ff96738640c6904af8ec7a46a2b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.113998 2113 raft_consensus.cc:2241] T 3239e58531d445fa80165cf678bc277c P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:37.114395 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d3c8ef06f66c499dbcf9462ff897e062"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.114892 2113 raft_consensus.cc:2270] T 3239e58531d445fa80165cf678bc277c P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:37.104274 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_51a6f15c19794e04b1e3fc415914c63e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.104378 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7dc5a9bbb4ef4cf1bd9eea2ecf470014"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.115450 2113 ts_tablet_manager.cc:1905] T 3239e58531d445fa80165cf678bc277c P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:37.106129 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a2855011fc5e4538b652e424a88f0ccc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.106221 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_05943f33c3784a289835bc01b44fa7d3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.106366 1484 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_3e7f2097fe1e49b4b93c44b8f921fa47" } modify_external_catalogs: true
I20250901 01:52:37.110033 1483 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_ea8e4576448a476b8bf399421a378e2c" } modify_external_catalogs: true
I20250901 01:52:37.116747 1484 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_3e7f2097fe1e49b4b93c44b8f921fa47" } modify_external_catalogs: true
W20250901 01:52:37.111085 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet e249474dcbf642189560c91ec3a99fdf because the tablet was not found. No further retry: Not found: Tablet not found: e249474dcbf642189560c91ec3a99fdf
I20250901 01:52:37.118587 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_32ebd259383a4126bac143cc2846ecf1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.118772 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 312) took 3651 ms. Trace:
I20250901 01:52:37.118854 2118 rpcz_store.cc:276] 0901 01:52:33.467598 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:33.467656 (+ 58us) service_pool.cc:225] Handling call
0901 01:52:37.067505 (+3599849us) ts_tablet_manager.cc:1145] Deleting tablet 6ba88af177874544b764cc32420e7ac9
0901 01:52:37.067518 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.072967 (+ 5449us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.109455 (+ 36488us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.118682 (+ 9227us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3599660}
I20250901 01:52:37.119450 2118 tablet_replica.cc:331] T b13afacd37474781877b9f2cd20b313a P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:37.119410 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c70df8a561d44243a06db22da54a0267"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.116530 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_625a15b6954b41cdab3c82c920f1621d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.120142 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3587066d2b0045bab64dd9b0c06e1938"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.120386 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2efaac1688c345dcbb7993a391aeaecd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.121039 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3240f597142a4270a11a2d63afa4d42a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.121865 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_25db77d477334b3fafe79991211c4152"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.123756 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_feae2485af0c47e8a93eabbc3d2b153b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.124042 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_03c8e6f6fec24b7dbebb95f97c759b36"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.124445 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8f26a77052c04c0299b5315aa89093f1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.124855 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_fcbdc06ded8445d4861c605c04bbffe5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.125563 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4dea782c9cfe453780c9aa2248e86518"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.126001 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7d30ee2cf87f4d4491a73245965ae571"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.126909 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7f1b8d77822c45fbbb850604cdca3e7d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.127737 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2f6107e1150d42eca26f6347660a6c4a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.128064 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_639de45ff732497b84a1d32b74075c10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.128850 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_59471f5c394a4e2488147eb03bf045f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.129422 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d74438e809a94da6858a128643f32fcd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.130453 1484 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 2821bf7750ba4e38b25ccba2e2bc0eab
I20250901 01:52:37.130777 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ae8c85b94c154a8cbb453f3e56f3b55a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:37.131317 1492 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 98c56eb62ad84f79b04f17a43d44c479 consensus state has no leader
I20250901 01:52:37.131870 1696 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 2821bf7750ba4e38b25ccba2e2bc0eab
I20250901 01:52:37.131897 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3c75461179c8480c982678103290257b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.133066 2118 raft_consensus.cc:2241] T b13afacd37474781877b9f2cd20b313a P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:37.133272 2118 raft_consensus.cc:2270] T b13afacd37474781877b9f2cd20b313a P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:37.133289 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a53d34eb03ef4347934ece8e9fc2231b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.135408 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_40966e3416ce487397279a38f9fdd08f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.136229 2118 ts_tablet_manager.cc:1905] T b13afacd37474781877b9f2cd20b313a P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:37.139226 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a0c9643981ee4fb3979cf890fd0b63ee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.140116 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f0f9c749f0054caa9981d20f6f080ab6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.136073 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_24e016bd9e734cb0ad09bdbc3fc5957f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.141085 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_742d602a780145248e76ec8eaf634711"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.142313 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8d5863d909ed463e9d3476f479d99750"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.143122 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_51a36c1e6e1b4c09b12358d1a56c7868"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.143793 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ab104278819842d28655327c9a5791c2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.142989 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_913dc261a30b4520bbc9052c2d3bed18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.136219 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_12c4dd0ad9be4975b8acd7e50ad1fbf1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:37.145068 992 consensus_peers.cc:489] T e9f1f892658b4736859ff507536920a1 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:37.136626 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d7bd20ea205c4853a9295743914feaf0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.136998 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ab57bdaf125d4961bee7fec0fe2d5dd1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.138310 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_45fb7b39f02c4bf8af4332186c4f97f3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.135761 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_982b0b27d1214701ac2f64769ab16088"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.135872 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_dc5f72b68df44f74b373f711a45ae84f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.147428 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b0d06117b6514264948a527cbda6d1cc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.147594 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_39b2488a9be04b6e87dc011a915d94c6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.148102 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_71b58b6ff76b44a4aaa2fdb81f553ac1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.148382 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e45e1c893b624d62a7442b90063f63d7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.148767 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_74926add65cf41b9ae8d458e06ce7617"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.149109 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_693c82e19b0b4d9b9d3b25670bf22c78"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.150048 1483 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet a4ffb0db51e54066a973d29b92cfbcf0
W20250901 01:52:37.150249 1483 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet a4ffb0db51e54066a973d29b92cfbcf0: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:37.150352 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d285d2c6465f40b69cb95a9c0409a3bd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.145910 1485 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_edd1006860a84e209149771e23bb8834" } new_table_name: "default.Table_488ee444aadb4fe4a20d172bcd85e423" modify_external_catalogs: true
I20250901 01:52:37.150947 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet a4ffb0db51e54066a973d29b92cfbcf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:37 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:37.151117 893 tablet_service.cc:1515] Processing DeleteTablet for tablet a4ffb0db51e54066a973d29b92cfbcf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:37 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:37.149966 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1f6a3236b3b9495fbd3bf93f19fba068"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.152381 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_747575e142a04de18a6569e93bde6307"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.153434 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d002dfd779cf404381df3a092e12a409"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.153474 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d66f6412c4784f899c3b9ec7222a5a65"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.154345 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_253fd959735b446a9f282e61a8212166"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.155004 2948 raft_consensus.cc:491] T 3a2bc5ad1cd243daa3c92fceca8ba414 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:37.155151 2948 raft_consensus.cc:513] T 3a2bc5ad1cd243daa3c92fceca8ba414 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:37.155198 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_54c8df39e7594fc4a95fe3900ea2d2bf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.155788 2948 leader_election.cc:290] T 3a2bc5ad1cd243daa3c92fceca8ba414 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:37.155959 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_0496d7dfe051494ebb34ae37b8bf6df3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.157899 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f376c69fceb143a79e85cc489343de69"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.158105 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9f5785f11f5b4983919be8bb94ae2e71"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.158862 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c1bce9fad1974f5d9e293b9a7e8a9a45"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:37.159358 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 494b6488aa0d4355b4caa46004267465, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7c21b68ef2e74136a807c1dd90e17f4e'. Need at least 3 replicas, but only 2 tablet servers are available
W20250901 01:52:37.160158 992 leader_election.cc:343] T 3a2bc5ad1cd243daa3c92fceca8ba414 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 3a2bc5ad1cd243daa3c92fceca8ba414
W20250901 01:52:37.160872 992 leader_election.cc:336] T 3a2bc5ad1cd243daa3c92fceca8ba414 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:37.160969 992 leader_election.cc:304] T 3a2bc5ad1cd243daa3c92fceca8ba414 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:37.161345 2948 raft_consensus.cc:2747] T 3a2bc5ad1cd243daa3c92fceca8ba414 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:37.161036 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_af618483b663477897936c8e9e5a7a8f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.163388 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b335ecd3017745aa9e5be0bf1b8784de"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.161386 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_724a7f40b8e8415487378c9a194b512b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.164222 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_df85f7ab3d424befabb92e7c6ed857d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.161506 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_490a39200b614cf49785d0558e871b26"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.161752 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a039da1e2278471dbd0fd3b846485b73"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.162109 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_126e88f170014150baf8e05bdb2fb94c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.162420 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_28678f7c3553411286805f0de6f162f1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.162616 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2dd13739ddf147ff9da893f9c3efe9c7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.167438 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_942ffe6da73a4a9286e58efefa5ae820"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.168058 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8c6593f0ad2644dfa596fc70416dab53"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.168306 1151 tablet_bootstrap.cc:492] T 98c56eb62ad84f79b04f17a43d44c479 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:37.168450 1151 ts_tablet_manager.cc:1397] T 98c56eb62ad84f79b04f17a43d44c479 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.066s user 0.006s sys 0.011s
E20250901 01:52:37.168753 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 01deaae6abb141789f0cd240f3058c73, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7baa92b46d98454a891222be49a6ca48'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.168905 1696 catalog_manager.cc:6205] Error selecting replicas for tablet e2ed32b3292141278af28026528e96c2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_6866a2f15cff496783cee46fc2895cb3'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.169031 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9f62a9d97f514747b940b488907369f3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d53a456dbca64daf92aeb5fb4517b417'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.169261 1151 raft_consensus.cc:357] T 98c56eb62ad84f79b04f17a43d44c479 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:37.170065 1151 raft_consensus.cc:383] T 98c56eb62ad84f79b04f17a43d44c479 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:37.169637 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_49b7b919525e4bef957deea916951b10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.170058 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a1b82e06298d41f58b0e579fdeafaf1e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.169965 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7f4b36156c7b42d886e4c29c4510ecf8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.170310 1151 raft_consensus.cc:738] T 98c56eb62ad84f79b04f17a43d44c479 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:37.169991 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a2a4ed48f0bd4506b8331d183f405ba2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.170509 1151 consensus_queue.cc:260] T 98c56eb62ad84f79b04f17a43d44c479 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:37.170987 1151 ts_tablet_manager.cc:1428] T 98c56eb62ad84f79b04f17a43d44c479 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
E20250901 01:52:37.170990 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5a7fa68d29644ae8afd2711e39dde59a, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d24e3a18ef4843fdb7b74ce23d47cf9e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.171094 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ff2bd017417c41b5b8225b8f00283e63, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0c7355975d6b44dbbe3a77df942b4d43'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.171211 1696 catalog_manager.cc:6205] Error selecting replicas for tablet dcc60bb000f440f1aca9133928b5ce20, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2fbca8463d6546c9b8f5b018fc60251d'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.171314 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 465c02a130c54e09b5a74fb4dd2ef762, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_bfddafe8e3204ce8bc9dd5934343ec59'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.171414 1696 catalog_manager.cc:6205] Error selecting replicas for tablet a5609f111ab34c0ea9fabdbce06c81f9, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_4505d77e4269443995012fef50c7e7b5'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.171497 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9f2416b2cc144054a8d2effc64157615, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_69575b1a9efd4573a2ee01d3f432800b'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.171602 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 75883d9fd00e429aa8853903a9fdd7e4, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_a9f6ac8609554a38ab632b2a3f7113d8'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.171701 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 8f4f5563df6b4d0db890747ba032ef9f, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2883135b3ab649a585f1158e8e527c13'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.171798 1696 catalog_manager.cc:6205] Error selecting replicas for tablet b1ba0e19dc0b462e80c537bf2d183f5b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_f9cac02fe78f4902a8c730f2706f7139'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.171892 1696 catalog_manager.cc:6205] Error selecting replicas for tablet fd2ce82a4ee24d0faa44980a3967e187, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_63d0f2ead9d946d39ce40162ebd769d6'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.172537 1151 tablet_bootstrap.cc:492] T 7304937048ca48a7bd153f3cdcfaeff0 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
E20250901 01:52:37.174042 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 086aa948f93644da8e5835b0fac87ddd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b4fbc34da806475c912052d0fab3fc53'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.174157 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 082db24bb3fd4089950504145a20a075, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ebba044988d5435aba124b1f971356d1'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.174256 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 93aa22afc0ad4d7fa24807ea3c5d5c21, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0f4b4094d2974b7591875375eda6bed0'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.174333 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 97418fffe6614dfba4ebd13a6ed57e67, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_360f4423e7f341339188b71cc82a1149'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.174422 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 3a62034e688348ed9e2ac9de009ed3a2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_90623cc9a13044b5934c922f14b85470'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.176978 1151 tablet_bootstrap.cc:654] T 7304937048ca48a7bd153f3cdcfaeff0 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:37.178562 2887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
I20250901 01:52:37.178966 2118 ts_tablet_manager.cc:1918] T b13afacd37474781877b9f2cd20b313a P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250901 01:52:37.179057 2887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
I20250901 01:52:37.179085 2118 log.cc:1199] T b13afacd37474781877b9f2cd20b313a P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/b13afacd37474781877b9f2cd20b313a
W20250901 01:52:37.179175 2887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250901 01:52:37.179412 2118 ts_tablet_manager.cc:1939] T b13afacd37474781877b9f2cd20b313a P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:37.180141 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 315) took 3687 ms. Trace:
I20250901 01:52:37.180233 2118 rpcz_store.cc:276] 0901 01:52:33.492957 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:33.493200 (+ 243us) service_pool.cc:225] Handling call
0901 01:52:37.119405 (+3626205us) ts_tablet_manager.cc:1145] Deleting tablet b13afacd37474781877b9f2cd20b313a
0901 01:52:37.119414 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.172464 (+ 53050us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.178867 (+ 6403us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.180120 (+ 1253us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3625985}
I20250901 01:52:37.175441 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d3da6cced43a4240869c976b55eca4d1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.175503 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_32852165ba1e4672a6e5f123bf5d1f10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.181794 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f0018832d07f4f7a8164c126604e5062"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:37.182037 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 18498c9cb09141d08c1c5c0854b63ebd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_df16f2a0280645a0949bc071a2af532f'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.182166 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f5e3316adeae47cd848833dd84b972bb, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_fc962c8bc653498e85cc0bb41639e9ad'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.182255 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 6ce9521b925b4f97bb4a810ce77e147e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_666a4667dfa84c5ca860560cef32d55c'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.182358 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2f61edb50d7a42198bfd70639c7bf4de, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_cd56686217424ab8b95e6bb776d4177a'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.182648 1489 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_1da54cdf1dfd4c0fb5f2538e26f17ad9" } new_table_name: "default.Table_63498be0da4f4dfb9fd9e73a721eab13" modify_external_catalogs: true
I20250901 01:52:37.183815 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_0c6a1788268b41ada762bb0dd5e47e49"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.184988 1486 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_2883135b3ab649a585f1158e8e527c13" } modify_external_catalogs: true
I20250901 01:52:37.185207 1486 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_2883135b3ab649a585f1158e8e527c13" } modify_external_catalogs: true
I20250901 01:52:37.185678 2113 ts_tablet_manager.cc:1918] T 3239e58531d445fa80165cf678bc277c P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:37.185705 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f284d4fe395a43f68dc1b24fc5b229a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.185848 2113 log.cc:1199] T 3239e58531d445fa80165cf678bc277c P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/3239e58531d445fa80165cf678bc277c
I20250901 01:52:37.186836 2113 ts_tablet_manager.cc:1939] T 3239e58531d445fa80165cf678bc277c P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:37.190110 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 452) took 3982 ms. Trace:
I20250901 01:52:37.190246 2113 rpcz_store.cc:276] 0901 01:52:33.207644 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:33.207706 (+ 62us) service_pool.cc:225] Handling call
0901 01:52:37.113720 (+3906014us) ts_tablet_manager.cc:1145] Deleting tablet 3239e58531d445fa80165cf678bc277c
0901 01:52:37.113731 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.139492 (+ 25761us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.175156 (+ 35664us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.190091 (+ 14935us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3905814}
I20250901 01:52:37.190739 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 3239e58531d445fa80165cf678bc277c (table default.Table_9163155cb611406a8ce0d5ae600a15c8 [id=aa9c7768ab244d60bedd5ea94cc5e94e]) successfully deleted
I20250901 01:52:37.190785 1151 tablet_bootstrap.cc:492] T 7304937048ca48a7bd153f3cdcfaeff0 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:37.190902 1151 ts_tablet_manager.cc:1397] T 7304937048ca48a7bd153f3cdcfaeff0 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.004s
I20250901 01:52:37.191123 2113 tablet_replica.cc:331] T b009cc95c819431ea9827a3e453b8da3 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:37.191356 2113 raft_consensus.cc:2241] T b009cc95c819431ea9827a3e453b8da3 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:37.191561 2113 raft_consensus.cc:2270] T b009cc95c819431ea9827a3e453b8da3 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:37.191959 1151 raft_consensus.cc:357] T 7304937048ca48a7bd153f3cdcfaeff0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:37.175747 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e9abf4a6bd8d48ffa117575132d337ff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.192190 1151 raft_consensus.cc:383] T 7304937048ca48a7bd153f3cdcfaeff0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:37.192265 1151 raft_consensus.cc:738] T 7304937048ca48a7bd153f3cdcfaeff0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:37.192276 2118 tablet_replica.cc:331] T 92f0ad59fbbb435ea872c669a52f854a P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:37.176036 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4725e088c5154b72b9b6d8e54cbcd757"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.192498 2118 raft_consensus.cc:2241] T 92f0ad59fbbb435ea872c669a52f854a P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:37.192513 1151 consensus_queue.cc:260] T 7304937048ca48a7bd153f3cdcfaeff0 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:37.192648 2118 raft_consensus.cc:2270] T 92f0ad59fbbb435ea872c669a52f854a P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:37.176097 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_60eb4609c655468ba61106297d03a2ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.193264 2118 ts_tablet_manager.cc:1905] T 92f0ad59fbbb435ea872c669a52f854a P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:37.176311 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_aad5cf53c85842fdabe2496b387b01aa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.180943 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ce9dd68b892b4a5c8da29d8557336ffb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.194443 1151 ts_tablet_manager.cc:1428] T 7304937048ca48a7bd153f3cdcfaeff0 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250901 01:52:37.194753 1151 tablet_bootstrap.cc:492] T d27c8d99528a42ed8dd1c5a7f373bbae P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:37.192077 2113 ts_tablet_manager.cc:1905] T b009cc95c819431ea9827a3e453b8da3 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:37.200520 2118 ts_tablet_manager.cc:1918] T 92f0ad59fbbb435ea872c669a52f854a P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:37.200593 2948 raft_consensus.cc:491] T 5d68ea97c71942c7b55b967ba870e0d7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:37.200757 2118 log.cc:1199] T 92f0ad59fbbb435ea872c669a52f854a P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/92f0ad59fbbb435ea872c669a52f854a
I20250901 01:52:37.200747 2948 raft_consensus.cc:513] T 5d68ea97c71942c7b55b967ba870e0d7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:37.201211 2118 ts_tablet_manager.cc:1939] T 92f0ad59fbbb435ea872c669a52f854a P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:37.201349 2948 leader_election.cc:290] T 5d68ea97c71942c7b55b967ba870e0d7 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:37.201651 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 316) took 3686 ms. Trace:
I20250901 01:52:37.201803 2118 rpcz_store.cc:276] 0901 01:52:33.515157 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:33.515918 (+ 761us) service_pool.cc:225] Handling call
0901 01:52:37.192224 (+3676306us) ts_tablet_manager.cc:1145] Deleting tablet 92f0ad59fbbb435ea872c669a52f854a
0901 01:52:37.192234 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.196345 (+ 4111us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.200368 (+ 4023us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.201634 (+ 1266us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3676055}
I20250901 01:52:37.202337 1151 tablet_bootstrap.cc:654] T d27c8d99528a42ed8dd1c5a7f373bbae P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:37.202306 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 92f0ad59fbbb435ea872c669a52f854a (table default.Table_4aa01c0e60364804a5e99d552ca3956c [id=59306e9184ca4715b1cd675609807eca]) successfully deleted
I20250901 01:52:37.202821 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 322) took 3594 ms. Trace:
I20250901 01:52:37.202942 2118 rpcz_store.cc:276] 0901 01:52:33.608356 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:33.608584 (+ 228us) service_pool.cc:225] Handling call
0901 01:52:37.202686 (+3594102us) ts_tablet_manager.cc:1145] Deleting tablet 0cba5329ab31440999faec599d704633
0901 01:52:37.202699 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.202800 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3593932}
I20250901 01:52:37.203579 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 319) took 3595 ms. Trace:
I20250901 01:52:37.203713 2118 rpcz_store.cc:276] 0901 01:52:33.608106 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:33.608740 (+ 634us) service_pool.cc:225] Handling call
0901 01:52:37.203480 (+3594740us) ts_tablet_manager.cc:1145] Deleting tablet 9fbd44641d9e43c3b592e388e4adf350
0901 01:52:37.203491 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.203564 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3594551}
I20250901 01:52:37.204128 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 320) took 3595 ms. Trace:
I20250901 01:52:37.204257 2118 rpcz_store.cc:276] 0901 01:52:33.608185 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:33.608982 (+ 797us) service_pool.cc:225] Handling call
0901 01:52:37.204056 (+3595074us) ts_tablet_manager.cc:1145] Deleting tablet cb7d7e1796c0456f8de98fb812b9e374
0901 01:52:37.204064 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.204116 (+ 52us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3594931}
I20250901 01:52:37.204721 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 323) took 3596 ms. Trace:
I20250901 01:52:37.204841 2118 rpcz_store.cc:276] 0901 01:52:33.608475 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:33.609164 (+ 689us) service_pool.cc:225] Handling call
0901 01:52:37.204655 (+3595491us) ts_tablet_manager.cc:1145] Deleting tablet 3697821cd29446bb908b020599df0db9
0901 01:52:37.204662 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.204708 (+ 46us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3595348}
I20250901 01:52:37.205276 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 318) took 3597 ms. Trace:
I20250901 01:52:37.205433 2118 rpcz_store.cc:276] 0901 01:52:33.608002 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:33.608659 (+ 657us) service_pool.cc:225] Handling call
0901 01:52:37.205198 (+3596539us) ts_tablet_manager.cc:1145] Deleting tablet 8151bad707c04ac687c5bddfea573e3f
0901 01:52:37.205207 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.205263 (+ 56us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3595792}
I20250901 01:52:37.205905 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 324) took 3597 ms. Trace:
I20250901 01:52:37.206025 2118 rpcz_store.cc:276] 0901 01:52:33.608600 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:33.608793 (+ 193us) service_pool.cc:225] Handling call
0901 01:52:37.205829 (+3597036us) ts_tablet_manager.cc:1145] Deleting tablet 85322a864d154f299f2d670fdd2a7bfb
0901 01:52:37.205837 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.205892 (+ 55us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3589219}
I20250901 01:52:37.206449 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 321) took 3598 ms. Trace:
I20250901 01:52:37.206569 2118 rpcz_store.cc:276] 0901 01:52:33.608262 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:33.608317 (+ 55us) service_pool.cc:225] Handling call
0901 01:52:37.206371 (+3598054us) ts_tablet_manager.cc:1145] Deleting tablet 7f46e9ddadaf4ff986d8c991178daa0c
0901 01:52:37.206378 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.206434 (+ 56us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8087,"tablet-delete.queue_time_us":3597809}
I20250901 01:52:37.206938 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_02c15d45200443d49ad4019e6d5a05d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.207988 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7fc32dcad10d42ddaad45c256aa1a521"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.209086 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 327) took 3589 ms. Trace:
I20250901 01:52:37.209205 2118 rpcz_store.cc:276] 0901 01:52:33.619153 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:33.619208 (+ 55us) service_pool.cc:225] Handling call
0901 01:52:37.209005 (+3589797us) ts_tablet_manager.cc:1145] Deleting tablet 641a83a3e50d451189c33c7b7b4ca070
0901 01:52:37.209015 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.209071 (+ 56us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3589618}
I20250901 01:52:37.209197 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_aaf32b8a9e0a4f94836e6b8232f503c2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.209736 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 329) took 3590 ms. Trace:
I20250901 01:52:37.209847 2118 rpcz_store.cc:276] 0901 01:52:33.619338 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:33.619505 (+ 167us) service_pool.cc:225] Handling call
0901 01:52:37.209664 (+3590159us) ts_tablet_manager.cc:1145] Deleting tablet f23df6f253a943fa8e146bb8c811ef51
0901 01:52:37.209671 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.209722 (+ 51us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3589972}
I20250901 01:52:37.210311 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 326) took 3591 ms. Trace:
I20250901 01:52:37.210423 2118 rpcz_store.cc:276] 0901 01:52:33.619043 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:33.619623 (+ 580us) service_pool.cc:225] Handling call
0901 01:52:37.210246 (+3590623us) ts_tablet_manager.cc:1145] Deleting tablet d10c193452954c198bba686169b8025b
0901 01:52:37.210255 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.210299 (+ 44us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3590501}
I20250901 01:52:37.210633 2113 ts_tablet_manager.cc:1918] T b009cc95c819431ea9827a3e453b8da3 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20250901 01:52:37.210789 2113 log.cc:1199] T b009cc95c819431ea9827a3e453b8da3 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/b009cc95c819431ea9827a3e453b8da3
I20250901 01:52:37.210999 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 325) took 3592 ms. Trace:
I20250901 01:52:37.211108 2118 rpcz_store.cc:276] 0901 01:52:33.618915 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:33.619780 (+ 865us) service_pool.cc:225] Handling call
0901 01:52:37.210908 (+3591128us) ts_tablet_manager.cc:1145] Deleting tablet ff013ea889d44cba8797834e4611b489
0901 01:52:37.210919 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.210983 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3590989}
I20250901 01:52:37.211550 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 328) took 3592 ms. Trace:
I20250901 01:52:37.211681 2118 rpcz_store.cc:276] 0901 01:52:33.619246 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:33.619821 (+ 575us) service_pool.cc:225] Handling call
0901 01:52:37.211465 (+3591644us) ts_tablet_manager.cc:1145] Deleting tablet b85d64e02cc646e380581ac9f59dee9e
0901 01:52:37.211474 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.211534 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3591535}
I20250901 01:52:37.212069 2113 ts_tablet_manager.cc:1939] T b009cc95c819431ea9827a3e453b8da3 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:37.212216 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 331) took 3592 ms. Trace:
I20250901 01:52:37.212328 2118 rpcz_store.cc:276] 0901 01:52:33.619567 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:33.620045 (+ 478us) service_pool.cc:225] Handling call
0901 01:52:37.212083 (+3592038us) ts_tablet_manager.cc:1145] Deleting tablet 75345ea2a3dc40fc8c9f8090d8839d2d
0901 01:52:37.212091 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.212202 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3591903}
I20250901 01:52:37.212848 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 330) took 3593 ms. Trace:
I20250901 01:52:37.212971 2118 rpcz_store.cc:276] 0901 01:52:33.619433 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:33.620566 (+ 1133us) service_pool.cc:225] Handling call
0901 01:52:37.212780 (+3592214us) ts_tablet_manager.cc:1145] Deleting tablet 088ef4861dc14f81ae470c3aaee5b4e1
0901 01:52:37.212788 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.212835 (+ 47us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3592060}
I20250901 01:52:37.213435 2118 tablet_replica.cc:331] T 8ca5cf4e99014c0d84a3acad4ef713af P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:37.213754 2118 raft_consensus.cc:2241] T 8ca5cf4e99014c0d84a3acad4ef713af P 00c282f779c1477d920f24dc0dc16628 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:37.214175 2118 raft_consensus.cc:2270] T 8ca5cf4e99014c0d84a3acad4ef713af P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:37.213925 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 453) took 4002 ms. Trace:
I20250901 01:52:37.214570 2113 rpcz_store.cc:276] 0901 01:52:33.211426 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:33.211750 (+ 324us) service_pool.cc:225] Handling call
0901 01:52:37.191067 (+3979317us) ts_tablet_manager.cc:1145] Deleting tablet b009cc95c819431ea9827a3e453b8da3
0901 01:52:37.191077 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.204684 (+ 13607us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.207615 (+ 2931us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.213905 (+ 6290us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3979056}
W20250901 01:52:37.215180 2887 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
I20250901 01:52:37.215202 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 454) took 4000 ms. Trace:
W20250901 01:52:37.215293 2887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
I20250901 01:52:37.215265 2113 rpcz_store.cc:276] 0901 01:52:33.214655 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:33.214844 (+ 189us) service_pool.cc:225] Handling call
0901 01:52:37.215091 (+4000247us) ts_tablet_manager.cc:1145] Deleting tablet d27c8d99528a42ed8dd1c5a7f373bbae
0901 01:52:37.215099 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.215186 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4000040}
W20250901 01:52:37.215394 2887 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250901 01:52:37.215472 2887 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250901 01:52:37.215830 2113 tablet_replica.cc:331] T 49aaaec611784e6bac131efd58dbca51 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:37.216043 2113 raft_consensus.cc:2241] T 49aaaec611784e6bac131efd58dbca51 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:37.216213 1151 tablet_bootstrap.cc:492] T d27c8d99528a42ed8dd1c5a7f373bbae P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:37.216367 1151 ts_tablet_manager.cc:1397] T d27c8d99528a42ed8dd1c5a7f373bbae P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.004s
I20250901 01:52:37.216522 2113 raft_consensus.cc:2270] T 49aaaec611784e6bac131efd58dbca51 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:37.217061 2113 ts_tablet_manager.cc:1905] T 49aaaec611784e6bac131efd58dbca51 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:37.217339 1151 raft_consensus.cc:357] T d27c8d99528a42ed8dd1c5a7f373bbae P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:37.217553 1151 raft_consensus.cc:383] T d27c8d99528a42ed8dd1c5a7f373bbae P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:37.217635 1151 raft_consensus.cc:738] T d27c8d99528a42ed8dd1c5a7f373bbae P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:37.217901 1151 consensus_queue.cc:260] T d27c8d99528a42ed8dd1c5a7f373bbae P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:37.218998 1151 ts_tablet_manager.cc:1428] T d27c8d99528a42ed8dd1c5a7f373bbae P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250901 01:52:37.219287 1151 tablet_bootstrap.cc:492] T ced4813805004bba9c4667a79ac201fc P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:37.220623 2118 ts_tablet_manager.cc:1905] T 8ca5cf4e99014c0d84a3acad4ef713af P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:37.221268 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_92d3eba69cae44e1bfaa131c70efe036"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.221316 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4c304eebd2564f3bada10abe5d53d60a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.222049 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2c03fd7337814ad5b0e302b09f303bbb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.222895 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e39776e85e1645758fcc8e2ff768ab33"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.222927 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_fcfdc0f4be09458598f53b5b0ee92827"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.223560 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ebfea87049aa45769d1ff622a6ce63ef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.223588 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8a48cf5831744e6484009b056159dd25"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.224143 1485 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 98c56eb62ad84f79b04f17a43d44c479 (table default.Table_1888ad759a544e5a9988201f9ae538ee [id=00e7f7fe6dc44f2787b9dd8e3cbf4a51]). Expected version 2 got 0
I20250901 01:52:37.224366 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet b009cc95c819431ea9827a3e453b8da3 (table default.Table_e53182d58e5244a9b7ba33cb48c939ba [id=cc26c89db2484a6fad919231116285f2]) successfully deleted
W20250901 01:52:37.221515 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 0cba5329ab31440999faec599d704633 because the tablet was not found. No further retry: Not found: Tablet not found: 0cba5329ab31440999faec599d704633
W20250901 01:52:37.224762 1473 catalog_manager.cc:4908] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet d27c8d99528a42ed8dd1c5a7f373bbae because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d27c8d99528a42ed8dd1c5a7f373bbae already in progress: creating tablet
I20250901 01:52:37.224789 1151 tablet_bootstrap.cc:654] T ced4813805004bba9c4667a79ac201fc P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
W20250901 01:52:37.224903 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 9fbd44641d9e43c3b592e388e4adf350 because the tablet was not found. No further retry: Not found: Tablet not found: 9fbd44641d9e43c3b592e388e4adf350
W20250901 01:52:37.225138 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet cb7d7e1796c0456f8de98fb812b9e374 because the tablet was not found. No further retry: Not found: Tablet not found: cb7d7e1796c0456f8de98fb812b9e374
W20250901 01:52:37.225310 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 3697821cd29446bb908b020599df0db9 because the tablet was not found. No further retry: Not found: Tablet not found: 3697821cd29446bb908b020599df0db9
W20250901 01:52:37.225534 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 8151bad707c04ac687c5bddfea573e3f because the tablet was not found. No further retry: Not found: Tablet not found: 8151bad707c04ac687c5bddfea573e3f
W20250901 01:52:37.225719 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 85322a864d154f299f2d670fdd2a7bfb because the tablet was not found. No further retry: Not found: Tablet not found: 85322a864d154f299f2d670fdd2a7bfb
W20250901 01:52:37.225900 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 7f46e9ddadaf4ff986d8c991178daa0c because the tablet was not found. No further retry: Not found: Tablet not found: 7f46e9ddadaf4ff986d8c991178daa0c
W20250901 01:52:37.226168 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet f23df6f253a943fa8e146bb8c811ef51 because the tablet was not found. No further retry: Not found: Tablet not found: f23df6f253a943fa8e146bb8c811ef51
I20250901 01:52:37.226337 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1f5cd7e8953641eb8a40bd4197d95bda"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:37.226507 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet ff013ea889d44cba8797834e4611b489 because the tablet was not found. No further retry: Not found: Tablet not found: ff013ea889d44cba8797834e4611b489
W20250901 01:52:37.226711 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet b85d64e02cc646e380581ac9f59dee9e because the tablet was not found. No further retry: Not found: Tablet not found: b85d64e02cc646e380581ac9f59dee9e
W20250901 01:52:37.226961 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 75345ea2a3dc40fc8c9f8090d8839d2d because the tablet was not found. No further retry: Not found: Tablet not found: 75345ea2a3dc40fc8c9f8090d8839d2d
I20250901 01:52:37.227046 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_43924849b85e41e9a10e663e1d7c9afd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:37.227162 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 088ef4861dc14f81ae470c3aaee5b4e1 because the tablet was not found. No further retry: Not found: Tablet not found: 088ef4861dc14f81ae470c3aaee5b4e1
I20250901 01:52:37.221640 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b78b972ca6344f8293eb2e1c406a87a2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.221812 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_6f312219469942288b7cbc828731bd4a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.221961 1488 master_service.cc:948] ReplaceTablet: received request to replace tablet ae21b9ae2e3b472eb67fb5f96388a50f from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:37.228456 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_42641e8aa6dd4f768dc4b45722263869"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.230016 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d4703b94b59b48ad9d07bd53fb02075e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.230620 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1df2aaa75dfa417a952c5d7643469ad4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.230751 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_58429499631c4649b36909cfcad6e5fc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.231386 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_73c4692b32714cffb4b405a25e539202"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.231451 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_fbc76d69f68f42d49d332832de951ff3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.232467 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_75b5937ad4ac4cae9a531b58e395a946"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.232764 2118 ts_tablet_manager.cc:1918] T 8ca5cf4e99014c0d84a3acad4ef713af P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 01:52:37.232878 2118 log.cc:1199] T 8ca5cf4e99014c0d84a3acad4ef713af P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/8ca5cf4e99014c0d84a3acad4ef713af
I20250901 01:52:37.234037 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ccd8ac772a674607b075e4805753b8c3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.234478 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5051c7bf9e5d4a318a36543ce16266e0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.234838 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e67a7851a9cc4afc90d2462a54303ed4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:37.235211 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 494b6488aa0d4355b4caa46004267465, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7c21b68ef2e74136a807c1dd90e17f4e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.235353 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 01deaae6abb141789f0cd240f3058c73, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7baa92b46d98454a891222be49a6ca48'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.235469 1696 catalog_manager.cc:6205] Error selecting replicas for tablet e2ed32b3292141278af28026528e96c2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_6866a2f15cff496783cee46fc2895cb3'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.235215 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_0635a44460274c8ab2518f54d51546fb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:37.235582 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9f62a9d97f514747b940b488907369f3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d53a456dbca64daf92aeb5fb4517b417'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.235687 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5a7fa68d29644ae8afd2711e39dde59a, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d24e3a18ef4843fdb7b74ce23d47cf9e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.235807 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ff2bd017417c41b5b8225b8f00283e63, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0c7355975d6b44dbbe3a77df942b4d43'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.235899 1696 catalog_manager.cc:6205] Error selecting replicas for tablet dcc60bb000f440f1aca9133928b5ce20, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2fbca8463d6546c9b8f5b018fc60251d'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.236008 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 465c02a130c54e09b5a74fb4dd2ef762, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_bfddafe8e3204ce8bc9dd5934343ec59'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.236094 1696 catalog_manager.cc:6205] Error selecting replicas for tablet a5609f111ab34c0ea9fabdbce06c81f9, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_4505d77e4269443995012fef50c7e7b5'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.236219 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9f2416b2cc144054a8d2effc64157615, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_69575b1a9efd4573a2ee01d3f432800b'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.236320 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 75883d9fd00e429aa8853903a9fdd7e4, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_a9f6ac8609554a38ab632b2a3f7113d8'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.236258 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_606c1ae3e60844b795b545faedffdf00"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:37.237403 1696 catalog_manager.cc:6205] Error selecting replicas for tablet b1ba0e19dc0b462e80c537bf2d183f5b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_f9cac02fe78f4902a8c730f2706f7139'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.237550 1696 catalog_manager.cc:6205] Error selecting replicas for tablet fd2ce82a4ee24d0faa44980a3967e187, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_63d0f2ead9d946d39ce40162ebd769d6'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.237605 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b57915fa2cf442b98fe49c8414584081"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:37.237771 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 086aa948f93644da8e5835b0fac87ddd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b4fbc34da806475c912052d0fab3fc53'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.237886 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 082db24bb3fd4089950504145a20a075, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ebba044988d5435aba124b1f971356d1'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.238039 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 93aa22afc0ad4d7fa24807ea3c5d5c21, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0f4b4094d2974b7591875375eda6bed0'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.238119 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 97418fffe6614dfba4ebd13a6ed57e67, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_360f4423e7f341339188b71cc82a1149'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.238240 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 3a62034e688348ed9e2ac9de009ed3a2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_90623cc9a13044b5934c922f14b85470'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.238312 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 18498c9cb09141d08c1c5c0854b63ebd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_df16f2a0280645a0949bc071a2af532f'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.237426 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4dbd785c739d4de19565a0ef724981fd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.238404 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_35263319986c4046baea474ac86d0155"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.238976 1486 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 8f4f5563df6b4d0db890747ba032ef9f
E20250901 01:52:37.239131 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f5e3316adeae47cd848833dd84b972bb, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_fc962c8bc653498e85cc0bb41639e9ad'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.239063 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_176f125aa2bb4b1cbf62e7e6f24a26bb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:37.239241 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 6ce9521b925b4f97bb4a810ce77e147e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_666a4667dfa84c5ca860560cef32d55c'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.239360 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2f61edb50d7a42198bfd70639c7bf4de, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_cd56686217424ab8b95e6bb776d4177a'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.240620 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5532ffa7a3e34aa88b26eff1008e7550"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.241384 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_af5c9c449ef941c790f05671847aefa9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.241720 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5edc026f351b4eff8b36a26bbcaacd0f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.242326 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_83de46f120c841c9a0e1d1ac6207706d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.243098 1489 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_0c7355975d6b44dbbe3a77df942b4d43" } new_table_name: "default.Table_8db3eec03fc7427096bb410c9da63608" modify_external_catalogs: true
I20250901 01:52:37.244462 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a7bc0c38f0084a338a003aa5ee13c2ac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:37.245050 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 494b6488aa0d4355b4caa46004267465, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7c21b68ef2e74136a807c1dd90e17f4e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.245155 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 01deaae6abb141789f0cd240f3058c73, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7baa92b46d98454a891222be49a6ca48'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.245227 1696 catalog_manager.cc:6205] Error selecting replicas for tablet e2ed32b3292141278af28026528e96c2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_6866a2f15cff496783cee46fc2895cb3'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.245304 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9f62a9d97f514747b940b488907369f3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d53a456dbca64daf92aeb5fb4517b417'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.245383 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5a7fa68d29644ae8afd2711e39dde59a, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d24e3a18ef4843fdb7b74ce23d47cf9e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.245482 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ff2bd017417c41b5b8225b8f00283e63, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0c7355975d6b44dbbe3a77df942b4d43'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.245569 1696 catalog_manager.cc:6205] Error selecting replicas for tablet dcc60bb000f440f1aca9133928b5ce20, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2fbca8463d6546c9b8f5b018fc60251d'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.245703 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 465c02a130c54e09b5a74fb4dd2ef762, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_bfddafe8e3204ce8bc9dd5934343ec59'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.245939 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_97a0bc950bfd4b619c0acd97511dd87c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.246568 918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d68ea97c71942c7b55b967ba870e0d7" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:37.246762 918 raft_consensus.cc:2466] T 5d68ea97c71942c7b55b967ba870e0d7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
E20250901 01:52:37.245769 1696 catalog_manager.cc:6205] Error selecting replicas for tablet a5609f111ab34c0ea9fabdbce06c81f9, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_4505d77e4269443995012fef50c7e7b5'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.247725 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_18676287f5984afba6ca86fcbaa1f877"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.248100 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2882b292ae96412bb7710645bf8d5f0b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.248682 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b857b0c0b5a34bedbf777b1fbbdbe609"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.248741 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_339127b660e4487c8965a798f0035e69"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.248942 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_bc087e7eca384c0989fea693a58d190d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.249387 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_facbb70e60854beb8e6e44716f3c5369"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.247990 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c874b60101074c6c988ff859ac1fcbbd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.247777 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d5503b78aaa24353a441826bb001dcab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:37.249468 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9f2416b2cc144054a8d2effc64157615, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_69575b1a9efd4573a2ee01d3f432800b'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.250429 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 75883d9fd00e429aa8853903a9fdd7e4, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_a9f6ac8609554a38ab632b2a3f7113d8'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.250512 1696 catalog_manager.cc:6205] Error selecting replicas for tablet b1ba0e19dc0b462e80c537bf2d183f5b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_f9cac02fe78f4902a8c730f2706f7139'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.250602 1696 catalog_manager.cc:6205] Error selecting replicas for tablet fd2ce82a4ee24d0faa44980a3967e187, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_63d0f2ead9d946d39ce40162ebd769d6'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.250664 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 086aa948f93644da8e5835b0fac87ddd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b4fbc34da806475c912052d0fab3fc53'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.250730 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 082db24bb3fd4089950504145a20a075, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ebba044988d5435aba124b1f971356d1'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.250700 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1372023f6a9a4b4dbba4f9532c0ff6d7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:37.250818 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 93aa22afc0ad4d7fa24807ea3c5d5c21, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0f4b4094d2974b7591875375eda6bed0'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.250890 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 97418fffe6614dfba4ebd13a6ed57e67, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_360f4423e7f341339188b71cc82a1149'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.250953 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 3a62034e688348ed9e2ac9de009ed3a2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_90623cc9a13044b5934c922f14b85470'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.251015 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 18498c9cb09141d08c1c5c0854b63ebd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_df16f2a0280645a0949bc071a2af532f'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.251665 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e81a277a746e452cb9d407617ef6a6d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.252012 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8ae37a51786c48cbb1296097a173dd9e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.251678 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3a3bd64d064d468a9ec8477ce9d76797"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.162925 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_0c082589aa1d4877a935ee1fc8e802ba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.252796 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_722ab5672ff442e19e7716355690b821"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.252929 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1caca84be02b46d5ba34470b0c7489a7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.253422 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4f55d53003c04ff78b33f26d293b8811"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.253476 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_927a384fcca44d329b7d1efab963377b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:37.254251 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f5e3316adeae47cd848833dd84b972bb, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_fc962c8bc653498e85cc0bb41639e9ad'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.254658 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 6ce9521b925b4f97bb4a810ce77e147e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_666a4667dfa84c5ca860560cef32d55c'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.254777 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2f61edb50d7a42198bfd70639c7bf4de, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_cd56686217424ab8b95e6bb776d4177a'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.255896 1492 catalog_manager.cc:5674] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447) does not have the latest schema for tablet 98c56eb62ad84f79b04f17a43d44c479 (table default.Table_1888ad759a544e5a9988201f9ae538ee [id=00e7f7fe6dc44f2787b9dd8e3cbf4a51]). Expected version 2 got 0
I20250901 01:52:37.256601 2118 ts_tablet_manager.cc:1939] T 8ca5cf4e99014c0d84a3acad4ef713af P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:37.256939 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 332) took 3513 ms. Trace:
I20250901 01:52:37.257014 2118 rpcz_store.cc:276] 0901 01:52:33.743987 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:33.744042 (+ 55us) service_pool.cc:225] Handling call
0901 01:52:37.213379 (+3469337us) ts_tablet_manager.cc:1145] Deleting tablet 8ca5cf4e99014c0d84a3acad4ef713af
0901 01:52:37.213390 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.226830 (+ 13440us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.232658 (+ 5828us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.256920 (+ 24262us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3469153}
I20250901 01:52:37.258323 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 7304937048ca48a7bd153f3cdcfaeff0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:28 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:37.258525 2118 tablet_replica.cc:331] T ad45c90ba0da43c9860745e534664a98 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:37.258663 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_dc72955a5a694d63b17006485544a466"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.258751 2118 raft_consensus.cc:2241] T ad45c90ba0da43c9860745e534664a98 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:37.258958 2118 raft_consensus.cc:2270] T ad45c90ba0da43c9860745e534664a98 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:37.259526 2118 ts_tablet_manager.cc:1905] T ad45c90ba0da43c9860745e534664a98 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:37.262346 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet d27c8d99528a42ed8dd1c5a7f373bbae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:32 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:37.262657 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3a26d03dd67a4d4e87b1e705a2d6ac80"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.263379 1151 tablet_bootstrap.cc:492] T ced4813805004bba9c4667a79ac201fc P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:37.262617 1492 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_a2e84eb190f440afbb9dff19c87fda4c" } modify_external_catalogs: true
I20250901 01:52:37.263510 1492 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_a2e84eb190f440afbb9dff19c87fda4c" } modify_external_catalogs: true
I20250901 01:52:37.263511 1151 ts_tablet_manager.cc:1397] T ced4813805004bba9c4667a79ac201fc P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.044s user 0.007s sys 0.000s
I20250901 01:52:37.264838 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e2af70c90a0b458c97ca6cf1ff91ebd2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.264922 1151 raft_consensus.cc:357] T ced4813805004bba9c4667a79ac201fc P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:37.262681 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 8ca5cf4e99014c0d84a3acad4ef713af (table default.Table_bfddafe8e3204ce8bc9dd5934343ec59 [id=303e76c918bc4c39aa47a4d24849a358]) successfully deleted
I20250901 01:52:37.265084 1151 raft_consensus.cc:383] T ced4813805004bba9c4667a79ac201fc P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:37.265667 1151 raft_consensus.cc:738] T ced4813805004bba9c4667a79ac201fc P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:37.265872 1151 consensus_queue.cc:260] T ced4813805004bba9c4667a79ac201fc P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:37.265937 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c565611f0cd947428a5e65406f8451b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:37.267061 1489 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ff2bd017417c41b5b8225b8f00283e63 has no consensus state
I20250901 01:52:37.267143 1151 ts_tablet_manager.cc:1428] T ced4813805004bba9c4667a79ac201fc P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250901 01:52:37.267411 1151 tablet_bootstrap.cc:492] T 2df2240631844defa433270a3829d65b P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:37.269276 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5ac03427a2f1486abe91e20b1821c678"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.269645 1151 tablet_bootstrap.cc:654] T 2df2240631844defa433270a3829d65b P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:37.271354 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_38f0cc5f72014c32bc393b33b120d81e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.271883 2113 ts_tablet_manager.cc:1918] T 49aaaec611784e6bac131efd58dbca51 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 01:52:37.272003 2113 log.cc:1199] T 49aaaec611784e6bac131efd58dbca51 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/49aaaec611784e6bac131efd58dbca51
I20250901 01:52:37.272382 2113 ts_tablet_manager.cc:1939] T 49aaaec611784e6bac131efd58dbca51 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:37.272889 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9fa6784f7bec4ccca637f542273b5cef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.273625 1482 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_9b9afa82d7de43048ee139b228df93a5" } new_table_name: "default.Table_6b991bcd83b94749bfee161fbfb8d8fa" modify_external_catalogs: true
E20250901 01:52:37.273957 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 494b6488aa0d4355b4caa46004267465, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7c21b68ef2e74136a807c1dd90e17f4e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.274120 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 01deaae6abb141789f0cd240f3058c73, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7baa92b46d98454a891222be49a6ca48'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.274261 1486 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_6866a2f15cff496783cee46fc2895cb3" } modify_external_catalogs: true
I20250901 01:52:37.274396 1486 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_6866a2f15cff496783cee46fc2895cb3" } modify_external_catalogs: true
E20250901 01:52:37.274266 1696 catalog_manager.cc:6205] Error selecting replicas for tablet e2ed32b3292141278af28026528e96c2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_6866a2f15cff496783cee46fc2895cb3'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.276551 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9f62a9d97f514747b940b488907369f3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d53a456dbca64daf92aeb5fb4517b417'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.276685 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5a7fa68d29644ae8afd2711e39dde59a, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d24e3a18ef4843fdb7b74ce23d47cf9e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.276808 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ff2bd017417c41b5b8225b8f00283e63, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_8db3eec03fc7427096bb410c9da63608'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.276985 2118 ts_tablet_manager.cc:1918] T ad45c90ba0da43c9860745e534664a98 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
E20250901 01:52:37.277027 1696 catalog_manager.cc:6205] Error selecting replicas for tablet dcc60bb000f440f1aca9133928b5ce20, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2fbca8463d6546c9b8f5b018fc60251d'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.277099 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 465c02a130c54e09b5a74fb4dd2ef762, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_bfddafe8e3204ce8bc9dd5934343ec59'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.277161 1696 catalog_manager.cc:6205] Error selecting replicas for tablet a5609f111ab34c0ea9fabdbce06c81f9, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_4505d77e4269443995012fef50c7e7b5'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.277293 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9f2416b2cc144054a8d2effc64157615, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_69575b1a9efd4573a2ee01d3f432800b'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.277411 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 75883d9fd00e429aa8853903a9fdd7e4, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_a9f6ac8609554a38ab632b2a3f7113d8'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.277525 1696 catalog_manager.cc:6205] Error selecting replicas for tablet b1ba0e19dc0b462e80c537bf2d183f5b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_f9cac02fe78f4902a8c730f2706f7139'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.277637 1696 catalog_manager.cc:6205] Error selecting replicas for tablet fd2ce82a4ee24d0faa44980a3967e187, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_63d0f2ead9d946d39ce40162ebd769d6'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.278997 1151 tablet_bootstrap.cc:492] T 2df2240631844defa433270a3829d65b P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:37.279141 1151 ts_tablet_manager.cc:1397] T 2df2240631844defa433270a3829d65b P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20250901 01:52:37.279973 1151 raft_consensus.cc:357] T 2df2240631844defa433270a3829d65b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:37.280159 1151 raft_consensus.cc:383] T 2df2240631844defa433270a3829d65b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:37.280215 1151 raft_consensus.cc:738] T 2df2240631844defa433270a3829d65b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:37.280623 1151 consensus_queue.cc:260] T 2df2240631844defa433270a3829d65b P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:37.281435 1151 ts_tablet_manager.cc:1428] T 2df2240631844defa433270a3829d65b P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250901 01:52:37.281529 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_00124878c82245b5a9cfa035102a0d6b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.282582 1151 tablet_bootstrap.cc:492] T 008d23f236c348b6be9aee718f8698af P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
E20250901 01:52:37.283788 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 086aa948f93644da8e5835b0fac87ddd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b4fbc34da806475c912052d0fab3fc53'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.283967 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 082db24bb3fd4089950504145a20a075, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ebba044988d5435aba124b1f971356d1'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.284117 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 93aa22afc0ad4d7fa24807ea3c5d5c21, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0f4b4094d2974b7591875375eda6bed0'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.284240 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 97418fffe6614dfba4ebd13a6ed57e67, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_360f4423e7f341339188b71cc82a1149'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.284355 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 3a62034e688348ed9e2ac9de009ed3a2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_90623cc9a13044b5934c922f14b85470'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.284507 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 18498c9cb09141d08c1c5c0854b63ebd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_df16f2a0280645a0949bc071a2af532f'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.284622 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f5e3316adeae47cd848833dd84b972bb, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_fc962c8bc653498e85cc0bb41639e9ad'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.284731 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 6ce9521b925b4f97bb4a810ce77e147e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_666a4667dfa84c5ca860560cef32d55c'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.284845 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2f61edb50d7a42198bfd70639c7bf4de, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_cd56686217424ab8b95e6bb776d4177a'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.284844 1151 tablet_bootstrap.cc:654] T 008d23f236c348b6be9aee718f8698af P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:37.285751 2118 log.cc:1199] T ad45c90ba0da43c9860745e534664a98 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/ad45c90ba0da43c9860745e534664a98
I20250901 01:52:37.286365 2118 ts_tablet_manager.cc:1939] T ad45c90ba0da43c9860745e534664a98 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:37.287147 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 333) took 3498 ms. Trace:
I20250901 01:52:37.287762 2118 rpcz_store.cc:276] 0901 01:52:33.788692 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:33.788765 (+ 73us) service_pool.cc:225] Handling call
0901 01:52:37.258475 (+3469710us) ts_tablet_manager.cc:1145] Deleting tablet ad45c90ba0da43c9860745e534664a98
0901 01:52:37.258485 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.273146 (+ 14661us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.276886 (+ 3740us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.287129 (+ 10243us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3469488}
I20250901 01:52:37.288348 2118 tablet_replica.cc:331] T d0b7c84ec9394a20ad77add3808e555a P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:37.288646 2118 raft_consensus.cc:2241] T d0b7c84ec9394a20ad77add3808e555a P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:37.288890 2118 raft_consensus.cc:2270] T d0b7c84ec9394a20ad77add3808e555a P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:37.289827 2118 ts_tablet_manager.cc:1905] T d0b7c84ec9394a20ad77add3808e555a P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:37.290995 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f32b08eaf14943a3a771a7c041adb38f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.291145 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_fcf7b9722cbf494b83f9354e3ae38d2c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.291923 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a5d36ad35e6e49f88e20457f3bf282a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.292516 1486 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet e2ed32b3292141278af28026528e96c2
I20250901 01:52:37.292622 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f2996f93bdfd482d8b863c443bff0da9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.293622 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_15a3c71d292e4f5db8764a23f8a84ed0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.294625 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 455) took 4079 ms. Trace:
I20250901 01:52:37.294716 2113 rpcz_store.cc:276] 0901 01:52:33.215661 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:33.215827 (+ 166us) service_pool.cc:225] Handling call
0901 01:52:37.215792 (+3999965us) ts_tablet_manager.cc:1145] Deleting tablet 49aaaec611784e6bac131efd58dbca51
0901 01:52:37.215800 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.221219 (+ 5419us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.271776 (+ 50557us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.294600 (+ 22824us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3999777}
I20250901 01:52:37.295542 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f992f085377c4510a8d889fd8984fb62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.296793 1482 master_service.cc:948] ReplaceTablet: received request to replace tablet d0c3c363dda444519a17dca63ad9caa7 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:37.298084 1482 master_service.cc:948] ReplaceTablet: received request to replace tablet c6f7c525f59b49938b697de98d4b5835 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:37.298127 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9959b9cd6743487c97625095f34dcf26"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.298821 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 49aaaec611784e6bac131efd58dbca51 (table default.Table_95e1338a49a04f22b679e363e253262e [id=e24b91b4ac904ef0a8c4725f70bced2f]) successfully deleted
I20250901 01:52:37.299002 2113 tablet_replica.cc:331] T 268439ff8d3a4f7a89f3d6395f2079c5 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:37.299217 2113 raft_consensus.cc:2241] T 268439ff8d3a4f7a89f3d6395f2079c5 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:37.299497 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_14d7f842e8284b219ff600e5dd8874ed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:37.299518 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 494b6488aa0d4355b4caa46004267465, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7c21b68ef2e74136a807c1dd90e17f4e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.300598 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 01deaae6abb141789f0cd240f3058c73, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7baa92b46d98454a891222be49a6ca48'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.300721 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9f62a9d97f514747b940b488907369f3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d53a456dbca64daf92aeb5fb4517b417'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.300866 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5a7fa68d29644ae8afd2711e39dde59a, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d24e3a18ef4843fdb7b74ce23d47cf9e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.300980 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ff2bd017417c41b5b8225b8f00283e63, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_8db3eec03fc7427096bb410c9da63608'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.301103 1696 catalog_manager.cc:6205] Error selecting replicas for tablet dcc60bb000f440f1aca9133928b5ce20, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_2fbca8463d6546c9b8f5b018fc60251d'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.301226 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 465c02a130c54e09b5a74fb4dd2ef762, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_bfddafe8e3204ce8bc9dd5934343ec59'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.301367 1696 catalog_manager.cc:6205] Error selecting replicas for tablet a5609f111ab34c0ea9fabdbce06c81f9, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_4505d77e4269443995012fef50c7e7b5'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.301486 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9f2416b2cc144054a8d2effc64157615, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_69575b1a9efd4573a2ee01d3f432800b'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.302165 2113 raft_consensus.cc:2270] T 268439ff8d3a4f7a89f3d6395f2079c5 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
E20250901 01:52:37.302732 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 75883d9fd00e429aa8853903a9fdd7e4, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_a9f6ac8609554a38ab632b2a3f7113d8'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.302882 1696 catalog_manager.cc:6205] Error selecting replicas for tablet b1ba0e19dc0b462e80c537bf2d183f5b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_f9cac02fe78f4902a8c730f2706f7139'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.303004 1696 catalog_manager.cc:6205] Error selecting replicas for tablet fd2ce82a4ee24d0faa44980a3967e187, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_63d0f2ead9d946d39ce40162ebd769d6'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.303126 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 086aa948f93644da8e5835b0fac87ddd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b4fbc34da806475c912052d0fab3fc53'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.303251 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 082db24bb3fd4089950504145a20a075, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ebba044988d5435aba124b1f971356d1'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.303372 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 93aa22afc0ad4d7fa24807ea3c5d5c21, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0f4b4094d2974b7591875375eda6bed0'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.303481 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 97418fffe6614dfba4ebd13a6ed57e67, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_360f4423e7f341339188b71cc82a1149'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.303589 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 3a62034e688348ed9e2ac9de009ed3a2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_90623cc9a13044b5934c922f14b85470'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.303558 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a768beb402f549fe82c9200f924082ae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:37.303778 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 18498c9cb09141d08c1c5c0854b63ebd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_df16f2a0280645a0949bc071a2af532f'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.303900 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f5e3316adeae47cd848833dd84b972bb, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_fc962c8bc653498e85cc0bb41639e9ad'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.304005 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 6ce9521b925b4f97bb4a810ce77e147e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_666a4667dfa84c5ca860560cef32d55c'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.304112 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2f61edb50d7a42198bfd70639c7bf4de, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_cd56686217424ab8b95e6bb776d4177a'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.304664 2118 ts_tablet_manager.cc:1918] T d0b7c84ec9394a20ad77add3808e555a P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:37.305068 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_0876a8d28b8147f786880f63f916058d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.305124 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7ec8a79e6a7b43fea79a115ed0b393b1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.301656 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_bedd564fd2344581b6222a7370aa12df"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.304420 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d5dfd8ca55454142b6f00e1e0fe7b1f2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.298779 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_91321ea6504047dcb83cf39266dd6d0d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.304769 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_6bf4c4f6fe814aa39781934e6e28f21a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.304926 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_906de1ce535d4f33bf96439748650d1c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.306133 2113 ts_tablet_manager.cc:1905] T 268439ff8d3a4f7a89f3d6395f2079c5 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:37.307457 2118 log.cc:1199] T d0b7c84ec9394a20ad77add3808e555a P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/d0b7c84ec9394a20ad77add3808e555a
I20250901 01:52:37.307929 2887 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:46333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/master-1/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:43871,127.0.104.61:42375,127.0.104.60:44231
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:42375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.104.61
--webserver_port=46533
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 06466b25e294b4d7d455e6c56263318dc34a80ef
build type FASTDEBUG
built by None at 01 Sep 2025 01:43:20 UTC on 24a791456cd2
build id 7840
ASAN enabled
I20250901 01:52:37.308684 2118 ts_tablet_manager.cc:1939] T d0b7c84ec9394a20ad77add3808e555a P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:37.308861 1492 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 45f6e8ba99b74898a2922b36768c6c36
W20250901 01:52:37.309047 1492 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 45f6e8ba99b74898a2922b36768c6c36: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:37.309468 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 334) took 3460 ms. Trace:
I20250901 01:52:37.310134 2887 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250901 01:52:37.310235 2911 raft_consensus.cc:491] T f8d9bbf0c5be462a8ddc539238cbdf97 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:37.309906 2118 rpcz_store.cc:276] 0901 01:52:33.848746 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:33.849129 (+ 383us) service_pool.cc:225] Handling call
0901 01:52:37.288293 (+3439164us) ts_tablet_manager.cc:1145] Deleting tablet d0b7c84ec9394a20ad77add3808e555a
0901 01:52:37.288308 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.295090 (+ 6782us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.304556 (+ 9466us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.309452 (+ 4896us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3438970}
I20250901 01:52:37.310400 2911 raft_consensus.cc:513] T f8d9bbf0c5be462a8ddc539238cbdf97 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:37.311045 2118 tablet_replica.cc:331] T 482a9a2cb81f448bb2823892e35920d4 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:37.311019 2911 leader_election.cc:290] T f8d9bbf0c5be462a8ddc539238cbdf97 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:37.311527 2118 raft_consensus.cc:2241] T 482a9a2cb81f448bb2823892e35920d4 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:37.311861 2113 ts_tablet_manager.cc:1918] T 268439ff8d3a4f7a89f3d6395f2079c5 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:37.312011 2118 raft_consensus.cc:2270] T 482a9a2cb81f448bb2823892e35920d4 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:37.312716 2887 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250901 01:52:37.315109 2118 ts_tablet_manager.cc:1905] T 482a9a2cb81f448bb2823892e35920d4 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:37.316013 2113 log.cc:1199] T 268439ff8d3a4f7a89f3d6395f2079c5 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/268439ff8d3a4f7a89f3d6395f2079c5
I20250901 01:52:37.316767 2113 ts_tablet_manager.cc:1939] T 268439ff8d3a4f7a89f3d6395f2079c5 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:37.310935 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet d0b7c84ec9394a20ad77add3808e555a (table default.Table_85b9e2f3f8f7400aa833201daae93dc3 [id=cac94430e07342dfa9b266df6472b6e3]) successfully deleted
I20250901 01:52:37.318017 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 458) took 3893 ms. Trace:
I20250901 01:52:37.313462 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_dc07cec669f143b89e5a9c0b29def527"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.318094 2113 rpcz_store.cc:276] 0901 01:52:33.424945 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:33.425122 (+ 177us) service_pool.cc:225] Handling call
0901 01:52:37.298929 (+3873807us) ts_tablet_manager.cc:1145] Deleting tablet 268439ff8d3a4f7a89f3d6395f2079c5
0901 01:52:37.298944 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.309066 (+ 10122us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.311772 (+ 2706us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.317997 (+ 6225us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3873619}
I20250901 01:52:37.318440 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 268439ff8d3a4f7a89f3d6395f2079c5 (table default.Table_b4fbc34da806475c912052d0fab3fc53 [id=3f1b378fb3444fd08dc947740ac0fa01]) successfully deleted
I20250901 01:52:37.318706 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 45f6e8ba99b74898a2922b36768c6c36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:37 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:37.318866 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a3bb9c1d12034a84a9a4c7be061df7ad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.319001 2113 tablet_replica.cc:331] T d0463a8d655d4db4800b4e1b221c8315 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:37.319172 2113 raft_consensus.cc:2241] T d0463a8d655d4db4800b4e1b221c8315 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:37.319298 2113 raft_consensus.cc:2270] T d0463a8d655d4db4800b4e1b221c8315 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:37.315059 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5f3cd7738d2b4495abb6243058210508"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.315303 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4f51bc930222430ebb519766f4599352"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.315274 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_93bdfab68cf14e5b931628a013f51788"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.315577 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_30973a5431674a2da82251249732679d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.315632 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3161e8b4fb1e416cbf1e637b3fb4e3f1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.315977 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5e505a420d8e4ce1a6b92229cba052b3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.321518 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_bb36760642bf47cd80807b782e96b14c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.316531 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ef0d227d71b84988bcf296521d45628d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.317054 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_49d56703c24d41b8b26480599bb1700a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:37.326365 992 leader_election.cc:336] T 5d68ea97c71942c7b55b967ba870e0d7 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:37.327165 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_50acb42d739544f4abf82f712b6bf518"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.327946 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3d44e7d8dda24d4e9aaea0aeb269f6f0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.327967 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1c4b15ce6c504010bf8e5d24b6b5f7e4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.329560 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f75cc4142f89438c8d9d0909c0da2afa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.329783 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9bc7cad08f594ddf90e726dc0ac192a9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.331148 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_69f822409f334634997de1e68794fe68"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.331455 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 45f6e8ba99b74898a2922b36768c6c36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:37 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:37.332278 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_200374bb0f4e47bdada96e05e69db1d7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.333688 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1b9544f27e5441a9b35793eb5ce9ecef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.340011 2118 ts_tablet_manager.cc:1918] T 482a9a2cb81f448bb2823892e35920d4 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 01:52:37.340430 2113 ts_tablet_manager.cc:1905] T d0463a8d655d4db4800b4e1b221c8315 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:37.342501 2118 log.cc:1199] T 482a9a2cb81f448bb2823892e35920d4 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/482a9a2cb81f448bb2823892e35920d4
I20250901 01:52:37.343099 2118 ts_tablet_manager.cc:1939] T 482a9a2cb81f448bb2823892e35920d4 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:37.345005 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 335) took 3172 ms. Trace:
I20250901 01:52:37.345575 2118 rpcz_store.cc:276] 0901 01:52:34.172438 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:34.172496 (+ 58us) service_pool.cc:225] Handling call
0901 01:52:37.310996 (+3138500us) ts_tablet_manager.cc:1145] Deleting tablet 482a9a2cb81f448bb2823892e35920d4
0901 01:52:37.311008 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.334404 (+ 23396us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.339909 (+ 5505us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.344986 (+ 5077us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3138299}
I20250901 01:52:37.347828 2113 ts_tablet_manager.cc:1918] T d0463a8d655d4db4800b4e1b221c8315 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:37.348623 2113 log.cc:1199] T d0463a8d655d4db4800b4e1b221c8315 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/d0463a8d655d4db4800b4e1b221c8315
I20250901 01:52:37.350109 2113 ts_tablet_manager.cc:1939] T d0463a8d655d4db4800b4e1b221c8315 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:37.335016 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e8163a72f1bb43f199d332fac7d71309"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.351265 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_fab212ad793840228ca2e875172650ec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.352152 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_04b123484d134632ab5941839218c18f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.352825 1489 master_service.cc:948] ReplaceTablet: received request to replace tablet b2b172757e50404aae78e818921a7d9d from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:37.353260 1489 master_service.cc:948] ReplaceTablet: received request to replace tablet b4278a013ae84082bc7a8497b04ac147 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:37.335590 1484 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_1888ad759a544e5a9988201f9ae538ee" } modify_external_catalogs: true
I20250901 01:52:37.353694 1484 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_1888ad759a544e5a9988201f9ae538ee" } modify_external_catalogs: true
I20250901 01:52:37.354926 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 459) took 3929 ms. Trace:
I20250901 01:52:37.355193 2118 tablet_replica.cc:331] T 5adf6b492640417ca071c94702b48cb9 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:37.355089 2113 rpcz_store.cc:276] 0901 01:52:33.425069 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:33.425343 (+ 274us) service_pool.cc:225] Handling call
0901 01:52:37.318703 (+3893360us) ts_tablet_manager.cc:1145] Deleting tablet d0463a8d655d4db4800b4e1b221c8315
0901 01:52:37.318963 (+ 260us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.344283 (+ 25320us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.347720 (+ 3437us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.354902 (+ 7182us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3893199}
I20250901 01:52:37.355827 2113 tablet_replica.cc:331] T f65ec84958eb488d8b0f812b59b5b3d5 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:37.355870 2118 raft_consensus.cc:2241] T 5adf6b492640417ca071c94702b48cb9 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:37.356113 2118 raft_consensus.cc:2270] T 5adf6b492640417ca071c94702b48cb9 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:37.356715 2113 raft_consensus.cc:2241] T f65ec84958eb488d8b0f812b59b5b3d5 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:37.356832 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 482a9a2cb81f448bb2823892e35920d4 (table default.Table_7baa92b46d98454a891222be49a6ca48 [id=b94e5ce799ee4046982550aace3a08d6]) successfully deleted
I20250901 01:52:37.357273 2113 raft_consensus.cc:2270] T f65ec84958eb488d8b0f812b59b5b3d5 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:37.357340 2118 ts_tablet_manager.cc:1905] T 5adf6b492640417ca071c94702b48cb9 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:37.357856 2113 ts_tablet_manager.cc:1905] T f65ec84958eb488d8b0f812b59b5b3d5 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:37.358387 1489 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_488ee444aadb4fe4a20d172bcd85e423" } modify_external_catalogs: true
I20250901 01:52:37.358575 1489 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_488ee444aadb4fe4a20d172bcd85e423" } modify_external_catalogs: true
I20250901 01:52:37.360898 1484 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 98c56eb62ad84f79b04f17a43d44c479
W20250901 01:52:37.361145 1484 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 98c56eb62ad84f79b04f17a43d44c479: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:37.361925 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a7f6d1cdb9cf4e028b9bb4a22862ac1c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:37.361940 3006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 01:52:37.362929 2948 raft_consensus.cc:491] T c422881b40f54971a68530abfbd1945b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:37.363062 2948 raft_consensus.cc:513] T c422881b40f54971a68530abfbd1945b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:37.363849 2948 leader_election.cc:290] T c422881b40f54971a68530abfbd1945b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:37.364437 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 98c56eb62ad84f79b04f17a43d44c479 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:37 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:37.364445 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 98c56eb62ad84f79b04f17a43d44c479 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:37 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:37.365545 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c8c6dbbf623a4babbe7a32bbbae1d140"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.366465 1484 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_2fbca8463d6546c9b8f5b018fc60251d" } new_table_name: "default.Table_5a19f7d6321b49249c4edd7afb1d24ef" modify_external_catalogs: true
I20250901 01:52:37.369405 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e5873ed110e94931b41873c7bb1fa8a9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.373464 2118 ts_tablet_manager.cc:1918] T 5adf6b492640417ca071c94702b48cb9 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250901 01:52:37.375819 2118 log.cc:1199] T 5adf6b492640417ca071c94702b48cb9 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/5adf6b492640417ca071c94702b48cb9
I20250901 01:52:37.376518 2118 ts_tablet_manager.cc:1939] T 5adf6b492640417ca071c94702b48cb9 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:37.377386 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 336) took 3199 ms. Trace:
I20250901 01:52:37.377918 2118 rpcz_store.cc:276] 0901 01:52:34.177480 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:34.177541 (+ 61us) service_pool.cc:225] Handling call
0901 01:52:37.355113 (+3177572us) ts_tablet_manager.cc:1145] Deleting tablet 5adf6b492640417ca071c94702b48cb9
0901 01:52:37.355130 (+ 17us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.365931 (+ 10801us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.373339 (+ 7408us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.377366 (+ 4027us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3177364}
I20250901 01:52:37.378815 2118 tablet_replica.cc:331] T 81ce448b148447ddbc41875d863e0518 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:37.379609 2118 raft_consensus.cc:2241] T 81ce448b148447ddbc41875d863e0518 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:37.380036 2118 raft_consensus.cc:2270] T 81ce448b148447ddbc41875d863e0518 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:37.380905 2118 ts_tablet_manager.cc:1905] T 81ce448b148447ddbc41875d863e0518 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:37.383061 992 consensus_peers.cc:489] T 6ba88af177874544b764cc32420e7ac9 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Couldn't send request to peer 00c282f779c1477d920f24dc0dc16628. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6ba88af177874544b764cc32420e7ac9. This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:37.383268 992 leader_election.cc:304] T 5d68ea97c71942c7b55b967ba870e0d7 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:37.383608 2948 raft_consensus.cc:2802] T 5d68ea97c71942c7b55b967ba870e0d7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:37.383723 2948 raft_consensus.cc:491] T 5d68ea97c71942c7b55b967ba870e0d7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:37.383805 2948 raft_consensus.cc:3058] T 5d68ea97c71942c7b55b967ba870e0d7 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:37.338655 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_84350b05eec14cce860fde5939befa15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.385795 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_845c4caf0ba94835a9e5dc460e570813"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.386507 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d237e439a12c4cf791610e1ad6946cd2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.387176 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f1b5867743d24621a85020cc1b817568"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.338725 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ba7d32db81d04783ac3c2d91fda4ca3c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.338886 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_fff1373767b84722ab30389419be9ba0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.388535 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f8611b24da554070bcfc33221e02b7c9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.339336 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3c98b106a0b74f4db9eda65cc9716a40"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.338711 1491 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_9f7d6f3bd6c6495280e3bd7725bc8f6d" } new_table_name: "default.Table_35f5e3ea21714809a7d4fdd3f4870133" modify_external_catalogs: true
I20250901 01:52:37.339612 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1b6866289835439090b77c1b8c8e3a30"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.338009 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a30e873253dc40eeb457037a26f751b8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.389513 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c48b41e3928e42d0961d180373a79bbf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.369899 1489 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 81c36693199945d98465579b509852e2
W20250901 01:52:37.390115 1489 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 81c36693199945d98465579b509852e2: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:37.378767 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 5adf6b492640417ca071c94702b48cb9 (table default.Table_99f0fb91a6994b2abf5af0c8f701159b [id=5d14706774d345c48ca3575feabb17b0]) successfully deleted
W20250901 01:52:37.378880 1484 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dcc60bb000f440f1aca9133928b5ce20 has no consensus state
I20250901 01:52:37.391026 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 81c36693199945d98465579b509852e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:37 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:37.391526 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 81c36693199945d98465579b509852e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:37 UTC) from {username='slave'} at 127.0.0.1:35946
W20250901 01:52:37.390461 992 consensus_peers.cc:489] T ad45c90ba0da43c9860745e534664a98 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Couldn't send request to peer 00c282f779c1477d920f24dc0dc16628. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20250901 01:52:37.392478 992 consensus_peers.cc:489] T b13afacd37474781877b9f2cd20b313a P fd27ab6665304aaea53adbd102dd4f58 -> Peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Couldn't send request to peer 00c282f779c1477d920f24dc0dc16628. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b13afacd37474781877b9f2cd20b313a. This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:37.393594 2911 consensus_queue.cc:1035] T e9f1f892658b4736859ff507536920a1 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
E20250901 01:52:37.395011 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 494b6488aa0d4355b4caa46004267465, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7c21b68ef2e74136a807c1dd90e17f4e'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.394980 2954 tablet.cc:1722] T 2dacb608897d47d28512df87dc320927 P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
E20250901 01:52:37.395150 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 01deaae6abb141789f0cd240f3058c73, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7baa92b46d98454a891222be49a6ca48'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.395243 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9f62a9d97f514747b940b488907369f3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d53a456dbca64daf92aeb5fb4517b417'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.395330 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5a7fa68d29644ae8afd2711e39dde59a, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d24e3a18ef4843fdb7b74ce23d47cf9e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.395416 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ff2bd017417c41b5b8225b8f00283e63, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_8db3eec03fc7427096bb410c9da63608'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.395502 1696 catalog_manager.cc:6205] Error selecting replicas for tablet dcc60bb000f440f1aca9133928b5ce20, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_5a19f7d6321b49249c4edd7afb1d24ef'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.395602 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 465c02a130c54e09b5a74fb4dd2ef762, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_bfddafe8e3204ce8bc9dd5934343ec59'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.395615 2118 ts_tablet_manager.cc:1918] T 81ce448b148447ddbc41875d863e0518 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
E20250901 01:52:37.395681 1696 catalog_manager.cc:6205] Error selecting replicas for tablet a5609f111ab34c0ea9fabdbce06c81f9, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_4505d77e4269443995012fef50c7e7b5'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.395761 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9f2416b2cc144054a8d2effc64157615, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_69575b1a9efd4573a2ee01d3f432800b'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.395767 2118 log.cc:1199] T 81ce448b148447ddbc41875d863e0518 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/81ce448b148447ddbc41875d863e0518
E20250901 01:52:37.395833 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 75883d9fd00e429aa8853903a9fdd7e4, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_a9f6ac8609554a38ab632b2a3f7113d8'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.395921 1696 catalog_manager.cc:6205] Error selecting replicas for tablet b1ba0e19dc0b462e80c537bf2d183f5b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_f9cac02fe78f4902a8c730f2706f7139'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.395994 1696 catalog_manager.cc:6205] Error selecting replicas for tablet fd2ce82a4ee24d0faa44980a3967e187, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_63d0f2ead9d946d39ce40162ebd769d6'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.396059 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 086aa948f93644da8e5835b0fac87ddd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b4fbc34da806475c912052d0fab3fc53'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.396127 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 082db24bb3fd4089950504145a20a075, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ebba044988d5435aba124b1f971356d1'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.396186 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 93aa22afc0ad4d7fa24807ea3c5d5c21, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0f4b4094d2974b7591875375eda6bed0'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.396245 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 97418fffe6614dfba4ebd13a6ed57e67, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_360f4423e7f341339188b71cc82a1149'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.396291 2118 ts_tablet_manager.cc:1939] T 81ce448b148447ddbc41875d863e0518 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
E20250901 01:52:37.396318 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 3a62034e688348ed9e2ac9de009ed3a2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_90623cc9a13044b5934c922f14b85470'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.398629 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 338) took 3212 ms. Trace:
I20250901 01:52:37.398747 2118 rpcz_store.cc:276] 0901 01:52:34.186273 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:34.186335 (+ 62us) service_pool.cc:225] Handling call
0901 01:52:37.378752 (+3192417us) ts_tablet_manager.cc:1145] Deleting tablet 81ce448b148447ddbc41875d863e0518
0901 01:52:37.378763 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.388507 (+ 9744us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.395500 (+ 6993us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.398602 (+ 3102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3192214}
I20250901 01:52:37.399293 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 81ce448b148447ddbc41875d863e0518 (table default.Table_6e461d9cb0ae4dd095975bbc21533794 [id=49673fa73f8c436e9b8026efc4f3cbbc]) successfully deleted
I20250901 01:52:37.400560 2118 tablet_replica.cc:331] T aec951f2cc84499b9162b9fa24d5440e P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:37.403847 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f7c8f7ad0924472cb44bd1312b7a60cd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.387390 2948 raft_consensus.cc:513] T 5d68ea97c71942c7b55b967ba870e0d7 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:37.404320 2118 raft_consensus.cc:2241] T aec951f2cc84499b9162b9fa24d5440e P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:37.404580 2118 raft_consensus.cc:2270] T aec951f2cc84499b9162b9fa24d5440e P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:37.404480 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f8f660db5b0f448092ea92f099f9de87"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.404582 2948 leader_election.cc:290] T 5d68ea97c71942c7b55b967ba870e0d7 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:37.404702 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a2bbfcc0d4424f52b8fe02cc56518067"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:37.404881 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 18498c9cb09141d08c1c5c0854b63ebd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_df16f2a0280645a0949bc071a2af532f'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.404985 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f5e3316adeae47cd848833dd84b972bb, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_fc962c8bc653498e85cc0bb41639e9ad'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.405052 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 6ce9521b925b4f97bb4a810ce77e147e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_666a4667dfa84c5ca860560cef32d55c'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.405112 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2f61edb50d7a42198bfd70639c7bf4de, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_cd56686217424ab8b95e6bb776d4177a'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.405174 2118 ts_tablet_manager.cc:1905] T aec951f2cc84499b9162b9fa24d5440e P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:37.405444 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e1fca36260d14ac99e5c24beebaca47f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.406370 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2d17b4afd2344041ac67e68590043871"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.407653 2887 server_base.cc:1047] running on GCE node
W20250901 01:52:37.408581 3007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 01:52:37.408983 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_daea5134b00343aaa34381f45dd0f123"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.409410 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_6fdb91cbbde245688e61e43e27abef02"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.410271 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_57550058328841deb8bb63d5e9810a8c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.411340 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b15df6e19eac444c89acf5f10109c50d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.411672 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ac5137b10f3a4644a6f1dcfadb6073f6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.412675 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a51481ac49144b0bb401f7f5aeebef23"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.414852 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_94b95e35fa794a45bfcdfe295750dd37"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.414883 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1f886d3996c847ecb64c7ee460bd5493"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.416594 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_0008ff7fbeef4cad8e660b581148d006"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.416601 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7e903df9079b48fab46bfa81b689eda8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.417375 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b6c2e963ab19482b8b2b37c847c2058f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.417500 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_431c1413df8c475eace63898e7c972cf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.417219 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_0329d8d566714da892931e6ea1bef897"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.418193 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8447979aad784f3f8cc57fe46eb3f9c8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.418880 1491 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_bfddafe8e3204ce8bc9dd5934343ec59" } new_table_name: "default.Table_8e4ca21f21734e049d174baa0a9335a4" modify_external_catalogs: true
I20250901 01:52:37.419458 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d4dab01d9283491e9bfba61c34084e0c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:37.420173 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 494b6488aa0d4355b4caa46004267465, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7c21b68ef2e74136a807c1dd90e17f4e'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.420394 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_365afcb7c507474ba9e898d768b9d6d5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:37.421900 3011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250901 01:52:37.422241 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_edb0aa8a896f4503b6bcda9b1eec0d66"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.421224 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2142b8d90ab94386a13fd1b3fb865594"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.421151 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_db2562e480f3495ab66d3adbe56d2624"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.424989 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c08b37ce5bf442f5aca72c0e8b9fa6eb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.426771 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ebdfac4b320e4edb8b61e7458c6190b2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.426998 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_16592fe5534647d09e5a7a895ac8a398"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.427663 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_89cae87fe7e040989a2682b474f506f4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:37.427855 1491 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 465c02a130c54e09b5a74fb4dd2ef762 has no consensus state
I20250901 01:52:37.429395 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e01dcb4d13304ad598b0489249c987b8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.429446 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7ce8b690d7154b83a14d45fbe088afdc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.429872 2118 ts_tablet_manager.cc:1918] T aec951f2cc84499b9162b9fa24d5440e P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250901 01:52:37.429987 2118 log.cc:1199] T aec951f2cc84499b9162b9fa24d5440e P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/aec951f2cc84499b9162b9fa24d5440e
I20250901 01:52:37.430091 2887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250901 01:52:37.430347 2118 ts_tablet_manager.cc:1939] T aec951f2cc84499b9162b9fa24d5440e P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:37.431311 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d320a0ad4e5a4e8286cdd3bbb853985f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.417177 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_438658ca93b54e4f85c8a8e183722113"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.432453 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 339) took 3171 ms. Trace:
I20250901 01:52:37.432420 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_36c52cd97c364c16aecaef56f66f5b76"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.432761 2113 ts_tablet_manager.cc:1918] T f65ec84958eb488d8b0f812b59b5b3d5 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250901 01:52:37.432983 2113 log.cc:1199] T f65ec84958eb488d8b0f812b59b5b3d5 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/f65ec84958eb488d8b0f812b59b5b3d5
I20250901 01:52:37.433005 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5ef2c301f46c45508263c3c7a5609abd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.433095 2118 rpcz_store.cc:276] 0901 01:52:34.261370 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:34.261616 (+ 246us) service_pool.cc:225] Handling call
0901 01:52:37.400504 (+3138888us) ts_tablet_manager.cc:1145] Deleting tablet aec951f2cc84499b9162b9fa24d5440e
0901 01:52:37.400521 (+ 17us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.421859 (+ 21338us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.429758 (+ 7899us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.432431 (+ 2673us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3138702}
I20250901 01:52:37.433609 2113 ts_tablet_manager.cc:1939] T f65ec84958eb488d8b0f812b59b5b3d5 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:37.434536 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 460) took 3985 ms. Trace:
I20250901 01:52:37.435158 2118 tablet_replica.cc:331] T 71941b815d6840478c5acc1cbb187cc9 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:37.434684 2113 rpcz_store.cc:276] 0901 01:52:33.449235 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:33.449293 (+ 58us) service_pool.cc:225] Handling call
0901 01:52:37.355764 (+3906471us) ts_tablet_manager.cc:1145] Deleting tablet f65ec84958eb488d8b0f812b59b5b3d5
0901 01:52:37.355779 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.376908 (+ 21129us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.423643 (+ 46735us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.434515 (+ 10872us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3906277}
I20250901 01:52:37.433498 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a144dfb07ffa476984fd02a325fe7778"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.435479 2118 raft_consensus.cc:2241] T 71941b815d6840478c5acc1cbb187cc9 P 00c282f779c1477d920f24dc0dc16628 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:37.433884 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet aec951f2cc84499b9162b9fa24d5440e (table default.Table_d8e839ac6396439197b8b727af6a31b7 [id=55c31cd42fac43e6b8eddc048f565d0a]) successfully deleted
I20250901 01:52:37.435776 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet f65ec84958eb488d8b0f812b59b5b3d5 (table default.Table_85b90ce84a904990b15be24c0acea21a [id=d1cd5288671540c0a95f7a037363014e]) successfully deleted
I20250901 01:52:37.435844 2118 raft_consensus.cc:2270] T 71941b815d6840478c5acc1cbb187cc9 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:37.434899 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_633cdf1fb7fb42a5ad680b558ef2d334"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.436515 2118 ts_tablet_manager.cc:1905] T 71941b815d6840478c5acc1cbb187cc9 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:37.436480 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_23b9fd30ed3b4a008faa213595235d28"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.436731 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c2029c8c38994ad4a792058583320080"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.435156 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_91b972c71deb4c3c80ed7a5bd411fc31"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.437606 2113 tablet_replica.cc:331] T 6ba88af177874544b764cc32420e7ac9 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:37.437779 2113 raft_consensus.cc:2241] T 6ba88af177874544b764cc32420e7ac9 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
E20250901 01:52:37.435211 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 01deaae6abb141789f0cd240f3058c73, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7baa92b46d98454a891222be49a6ca48'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.435237 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f0abd15ffaa74fca9dd0cbda3fc5e5f4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.438344 2113 raft_consensus.cc:2270] T 6ba88af177874544b764cc32420e7ac9 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:37.436772 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_274d0b27a41b4054a8a01ce7ca7f9550"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.435040 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e70f775d75434b7f9ba28262d996f73d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:37.439474 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9f62a9d97f514747b940b488907369f3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d53a456dbca64daf92aeb5fb4517b417'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.439669 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5a7fa68d29644ae8afd2711e39dde59a, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d24e3a18ef4843fdb7b74ce23d47cf9e'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.439772 2113 ts_tablet_manager.cc:1905] T 6ba88af177874544b764cc32420e7ac9 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
E20250901 01:52:37.439796 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ff2bd017417c41b5b8225b8f00283e63, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_8db3eec03fc7427096bb410c9da63608'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.439872 1696 catalog_manager.cc:6205] Error selecting replicas for tablet dcc60bb000f440f1aca9133928b5ce20, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_5a19f7d6321b49249c4edd7afb1d24ef'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.439960 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 465c02a130c54e09b5a74fb4dd2ef762, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_8e4ca21f21734e049d174baa0a9335a4'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.440047 1696 catalog_manager.cc:6205] Error selecting replicas for tablet a5609f111ab34c0ea9fabdbce06c81f9, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_4505d77e4269443995012fef50c7e7b5'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.440136 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9f2416b2cc144054a8d2effc64157615, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_69575b1a9efd4573a2ee01d3f432800b'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.440212 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 75883d9fd00e429aa8853903a9fdd7e4, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_a9f6ac8609554a38ab632b2a3f7113d8'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.440272 1696 catalog_manager.cc:6205] Error selecting replicas for tablet b1ba0e19dc0b462e80c537bf2d183f5b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_f9cac02fe78f4902a8c730f2706f7139'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.440328 1696 catalog_manager.cc:6205] Error selecting replicas for tablet fd2ce82a4ee24d0faa44980a3967e187, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_63d0f2ead9d946d39ce40162ebd769d6'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.440379 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 086aa948f93644da8e5835b0fac87ddd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b4fbc34da806475c912052d0fab3fc53'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.441049 2887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250901 01:52:37.443332 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b95178f0877d44958fb2b9c103167c04"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:37.444389 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 082db24bb3fd4089950504145a20a075, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ebba044988d5435aba124b1f971356d1'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.444499 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 93aa22afc0ad4d7fa24807ea3c5d5c21, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0f4b4094d2974b7591875375eda6bed0'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.444561 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 97418fffe6614dfba4ebd13a6ed57e67, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_360f4423e7f341339188b71cc82a1149'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.444628 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 3a62034e688348ed9e2ac9de009ed3a2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_90623cc9a13044b5934c922f14b85470'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.444698 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 18498c9cb09141d08c1c5c0854b63ebd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_df16f2a0280645a0949bc071a2af532f'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.444756 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f5e3316adeae47cd848833dd84b972bb, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_fc962c8bc653498e85cc0bb41639e9ad'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.444811 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 6ce9521b925b4f97bb4a810ce77e147e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_666a4667dfa84c5ca860560cef32d55c'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.444869 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2f61edb50d7a42198bfd70639c7bf4de, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_cd56686217424ab8b95e6bb776d4177a'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.450110 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_894bfb772ced43f69380f3da074a5b85"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.450585 1485 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_63d0f2ead9d946d39ce40162ebd769d6" } modify_external_catalogs: true
I20250901 01:52:37.450774 1485 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_63d0f2ead9d946d39ce40162ebd769d6" } modify_external_catalogs: true
I20250901 01:52:37.451118 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_814fdb3284524a5887c24e7ee64072fd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.451683 2118 ts_tablet_manager.cc:1918] T 71941b815d6840478c5acc1cbb187cc9 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 01:52:37.452114 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5bc444218f7b450e938f17e0ea2dd71f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.452973 2118 log.cc:1199] T 71941b815d6840478c5acc1cbb187cc9 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/71941b815d6840478c5acc1cbb187cc9
I20250901 01:52:37.453289 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f046f42b648a4687bc77a22ff9829d19"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.453526 2118 ts_tablet_manager.cc:1939] T 71941b815d6840478c5acc1cbb187cc9 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:37.454533 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 341) took 3074 ms. Trace:
I20250901 01:52:37.454975 2118 rpcz_store.cc:276] 0901 01:52:34.379754 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:34.379803 (+ 49us) service_pool.cc:225] Handling call
0901 01:52:37.434109 (+3054306us) ts_tablet_manager.cc:1145] Deleting tablet 71941b815d6840478c5acc1cbb187cc9
0901 01:52:37.434123 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.445392 (+ 11269us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.451564 (+ 6172us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.454517 (+ 2953us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3053202}
I20250901 01:52:37.455330 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_40525f40f4d74a1eb8dbb2e7f750c87e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.455581 2118 tablet_replica.cc:331] T 858218c4427c41e980b2b5c969a5af40 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:37.455700 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 71941b815d6840478c5acc1cbb187cc9 (table default.Table_c903c87140a144d587f6fa78e6d9b11b [id=3ac80ef18f1044c0b22018ae09997fd3]) successfully deleted
I20250901 01:52:37.455780 2118 raft_consensus.cc:2241] T 858218c4427c41e980b2b5c969a5af40 P 00c282f779c1477d920f24dc0dc16628 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:37.456115 2118 raft_consensus.cc:2270] T 858218c4427c41e980b2b5c969a5af40 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:37.456527 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_bb37c6fd191a4608b90db6d1b0f09ff5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.457449 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e4ebd518de5f490c97a374a78760d82a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.457783 2118 ts_tablet_manager.cc:1905] T 858218c4427c41e980b2b5c969a5af40 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:37.458910 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f9c1b5e7618645f3935af5505db124cb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.459240 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2d86267d8e3d4717aafb9440ab4ea8cd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:37.451390 992 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:40561, user_credentials={real_user=slave}} blocked reactor thread for 57070us
I20250901 01:52:37.457610 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 2df2240631844defa433270a3829d65b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:30 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:37.461951 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_844571de85e94cf88313824676abed69"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.462739 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_540816142f9f486a8dd2f1cf86e97a0c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:37.464612 992 leader_election.cc:343] T f8d9bbf0c5be462a8ddc539238cbdf97 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: f8d9bbf0c5be462a8ddc539238cbdf97
I20250901 01:52:37.464587 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_fd0e8e9cd180408e98d60206bfb350f0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.464637 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9fd0ed7fae094f868a365fc7c64b982a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.465481 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_832bb33eee314d7c91b8954d59032581"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.466615 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2cb89cc3cba245ce8902783f707e3b41"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:37.467167 992 consensus_peers.cc:489] T 3dae00aa38204ee3ac77325eb334be42 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:37.468086 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_100920e48e054ae2b164b9fe48f9671f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.469195 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5f82d8ae99ee4c7f80786d039215fdaf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.469401 911 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d68ea97c71942c7b55b967ba870e0d7" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:37.469547 911 raft_consensus.cc:3058] T 5d68ea97c71942c7b55b967ba870e0d7 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:37.470211 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_bf1f82719b344a7e8443dbc7cf6cfd89"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.472133 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_bc2042ed7b6a47c291f50bea0f96ba0c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.473191 911 raft_consensus.cc:2466] T 5d68ea97c71942c7b55b967ba870e0d7 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:37.474659 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2e9d79b7839e4fafbe16f72a3fe0dc93"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.475669 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4cf8e624454442c1a2e6c7ff5e7b6e8b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.476550 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_0a7057a065994d0ea3dcafec5458a7d5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.477160 2113 ts_tablet_manager.cc:1918] T 6ba88af177874544b764cc32420e7ac9 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:37.477322 2113 log.cc:1199] T 6ba88af177874544b764cc32420e7ac9 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/6ba88af177874544b764cc32420e7ac9
I20250901 01:52:37.477550 2954 tablet.cc:1722] T 5afae316ff164413be0d25d4bc0f58dc P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:37.477746 2113 ts_tablet_manager.cc:1939] T 6ba88af177874544b764cc32420e7ac9 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:37.478212 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b1ff081ee34244fd8c0d3c3d38f2dcee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.479080 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a7313015d0bf4a7c8d43afa04678d0dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.479993 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_219255cdd2214f4a9fed155d13e34314"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.480823 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_139a6ef382ba4e979b29fdb6afe2bb5f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.484983 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_56ee11fb859048f58294801ec5cdcad4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.485010 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9f0d406b93da47b09e8caf1cd42661e7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.485121 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e8dc3c7e080b4817b17a5f24013c5cf0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:37.487047 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 494b6488aa0d4355b4caa46004267465, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7c21b68ef2e74136a807c1dd90e17f4e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.487238 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 01deaae6abb141789f0cd240f3058c73, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7baa92b46d98454a891222be49a6ca48'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.489039 2118 ts_tablet_manager.cc:1918] T 858218c4427c41e980b2b5c969a5af40 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 01:52:37.487543 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_6f6b8842f4c44d0a8f88936531de2eeb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.489579 2887 hybrid_clock.cc:648] HybridClock initialized: now 1756691557489458 us; error 92 us; skew 500 ppm
I20250901 01:52:37.489775 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e14bc3ee6a7b41e3af1766b79bb1f0b9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.488358 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9d5c9e76014b4d6aba5ed89c87bf9b2f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.488917 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1b5cf6f3d5cb4d76bb189f2790055125"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.490173 2887 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
E20250901 01:52:37.491025 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9f62a9d97f514747b940b488907369f3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d53a456dbca64daf92aeb5fb4517b417'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.491236 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 5a7fa68d29644ae8afd2711e39dde59a, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d24e3a18ef4843fdb7b74ce23d47cf9e'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.491689 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ff2bd017417c41b5b8225b8f00283e63, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_8db3eec03fc7427096bb410c9da63608'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.491849 1696 catalog_manager.cc:6205] Error selecting replicas for tablet dcc60bb000f440f1aca9133928b5ce20, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_5a19f7d6321b49249c4edd7afb1d24ef'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.491991 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 465c02a130c54e09b5a74fb4dd2ef762, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_8e4ca21f21734e049d174baa0a9335a4'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.492142 1696 catalog_manager.cc:6205] Error selecting replicas for tablet a5609f111ab34c0ea9fabdbce06c81f9, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_4505d77e4269443995012fef50c7e7b5'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.492831 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9f2416b2cc144054a8d2effc64157615, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_69575b1a9efd4573a2ee01d3f432800b'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.493539 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 461) took 4024 ms. Trace:
I20250901 01:52:37.488862 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_062fa119ee184239955f30555ab738a5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.489239 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_73830665c4064e819f09ae95b984f9f3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.493680 2113 rpcz_store.cc:276] 0901 01:52:33.469330 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:33.469537 (+ 207us) service_pool.cc:225] Handling call
0901 01:52:37.437559 (+3968022us) ts_tablet_manager.cc:1145] Deleting tablet 6ba88af177874544b764cc32420e7ac9
0901 01:52:37.437568 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.471401 (+ 33833us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.477057 (+ 5656us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.493512 (+ 16455us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3967860}
I20250901 01:52:37.494555 2113 tablet_replica.cc:331] T b13afacd37474781877b9f2cd20b313a P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:37.489363 1491 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_7c21b68ef2e74136a807c1dd90e17f4e" } new_table_name: "default.Table_dc77f0e1986d409dbf72b15b05150586" modify_external_catalogs: true
I20250901 01:52:37.493093 1482 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_9f86b15283e34dd88287f72985404935" } new_table_name: "default.Table_ba32ff5aeabb4d0caa98298e418ab717" modify_external_catalogs: true
I20250901 01:52:37.494957 2113 raft_consensus.cc:2241] T b13afacd37474781877b9f2cd20b313a P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:37.495677 2113 raft_consensus.cc:2270] T b13afacd37474781877b9f2cd20b313a P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:37.500010 2113 ts_tablet_manager.cc:1905] T b13afacd37474781877b9f2cd20b313a P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:37.508868 1151 tablet_bootstrap.cc:492] T 008d23f236c348b6be9aee718f8698af P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:37.509363 1151 ts_tablet_manager.cc:1397] T 008d23f236c348b6be9aee718f8698af P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.228s user 0.026s sys 0.047s
I20250901 01:52:37.510318 1151 raft_consensus.cc:357] T 008d23f236c348b6be9aee718f8698af P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:37.511377 1151 raft_consensus.cc:383] T 008d23f236c348b6be9aee718f8698af P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:37.511520 1151 raft_consensus.cc:738] T 008d23f236c348b6be9aee718f8698af P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:37.511759 1151 consensus_queue.cc:260] T 008d23f236c348b6be9aee718f8698af P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:37.512733 1151 ts_tablet_manager.cc:1428] T 008d23f236c348b6be9aee718f8698af P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
E20250901 01:52:37.514324 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 75883d9fd00e429aa8853903a9fdd7e4, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_a9f6ac8609554a38ab632b2a3f7113d8'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.514647 1151 tablet_bootstrap.cc:492] T 438ae684af234c868a419fbab43b0b98 P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
E20250901 01:52:37.514781 1696 catalog_manager.cc:6205] Error selecting replicas for tablet b1ba0e19dc0b462e80c537bf2d183f5b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_f9cac02fe78f4902a8c730f2706f7139'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.515918 1696 catalog_manager.cc:6205] Error selecting replicas for tablet fd2ce82a4ee24d0faa44980a3967e187, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_63d0f2ead9d946d39ce40162ebd769d6'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.516122 1488 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: default.Table_6b991bcd83b94749bfee161fbfb8d8fa [id=94d314a62cf24de19d6e1fff9b3794ce] alter complete (version 2)
E20250901 01:52:37.516245 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 086aa948f93644da8e5835b0fac87ddd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b4fbc34da806475c912052d0fab3fc53'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.516780 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 082db24bb3fd4089950504145a20a075, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ebba044988d5435aba124b1f971356d1'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.517639 1151 tablet_bootstrap.cc:654] T 438ae684af234c868a419fbab43b0b98 P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:37.519631 2113 ts_tablet_manager.cc:1918] T b13afacd37474781877b9f2cd20b313a P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250901 01:52:37.521350 1491 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 494b6488aa0d4355b4caa46004267465 has no consensus state
I20250901 01:52:37.521534 2113 log.cc:1199] T b13afacd37474781877b9f2cd20b313a P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/b13afacd37474781877b9f2cd20b313a
I20250901 01:52:37.522058 2113 ts_tablet_manager.cc:1939] T b13afacd37474781877b9f2cd20b313a P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
E20250901 01:52:37.522073 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 93aa22afc0ad4d7fa24807ea3c5d5c21, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0f4b4094d2974b7591875375eda6bed0'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.523914 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_0a63136ff0644305901ad0a74fe0c4a2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.524721 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 463) took 4034 ms. Trace:
E20250901 01:52:37.525564 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 97418fffe6614dfba4ebd13a6ed57e67, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_360f4423e7f341339188b71cc82a1149'. Need at least 3 replicas, but only 2 tablet servers are available
W20250901 01:52:37.525936 992 consensus_peers.cc:489] T a8483e7212d94daba7e5401256c33aef P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250901 01:52:37.526109 992 leader_election.cc:336] T f8d9bbf0c5be462a8ddc539238cbdf97 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:37.526227 992 leader_election.cc:304] T f8d9bbf0c5be462a8ddc539238cbdf97 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:37.525879 2113 rpcz_store.cc:276] 0901 01:52:33.490646 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:33.491210 (+ 564us) service_pool.cc:225] Handling call
0901 01:52:37.494508 (+4003298us) ts_tablet_manager.cc:1145] Deleting tablet b13afacd37474781877b9f2cd20b313a
0901 01:52:37.494517 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.506591 (+ 12074us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.519506 (+ 12915us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.524702 (+ 5196us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4003076}
E20250901 01:52:37.526289 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 3a62034e688348ed9e2ac9de009ed3a2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_90623cc9a13044b5934c922f14b85470'. Need at least 3 replicas, but only 2 tablet servers are available
W20250901 01:52:37.526559 992 leader_election.cc:336] T c422881b40f54971a68530abfbd1945b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
E20250901 01:52:37.526612 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 18498c9cb09141d08c1c5c0854b63ebd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_df16f2a0280645a0949bc071a2af532f'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.527027 3009 raft_consensus.cc:2747] T f8d9bbf0c5be462a8ddc539238cbdf97 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:37.527601 2113 tablet_replica.cc:331] T 92f0ad59fbbb435ea872c669a52f854a P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:37.527948 2113 raft_consensus.cc:2241] T 92f0ad59fbbb435ea872c669a52f854a P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:37.528692 2113 raft_consensus.cc:2270] T 92f0ad59fbbb435ea872c669a52f854a P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:37.529397 2113 ts_tablet_manager.cc:1905] T 92f0ad59fbbb435ea872c669a52f854a P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:37.535897 2113 ts_tablet_manager.cc:1918] T 92f0ad59fbbb435ea872c669a52f854a P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:37.541190 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_083781fd2a0c43cda7c2936a3278b249"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.542563 1484 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_35f5e3ea21714809a7d4fdd3f4870133" } modify_external_catalogs: true
I20250901 01:52:37.542780 1484 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_35f5e3ea21714809a7d4fdd3f4870133" } modify_external_catalogs: true
I20250901 01:52:37.548072 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_cfa724a2f09545b7a25da5a71fae4ac3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.552204 2113 log.cc:1199] T 92f0ad59fbbb435ea872c669a52f854a P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/92f0ad59fbbb435ea872c669a52f854a
I20250901 01:52:37.543066 1151 tablet_bootstrap.cc:492] T 438ae684af234c868a419fbab43b0b98 P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:37.552712 1151 ts_tablet_manager.cc:1397] T 438ae684af234c868a419fbab43b0b98 P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.038s user 0.002s sys 0.009s
I20250901 01:52:37.552829 2113 ts_tablet_manager.cc:1939] T 92f0ad59fbbb435ea872c669a52f854a P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:37.553521 1151 raft_consensus.cc:357] T 438ae684af234c868a419fbab43b0b98 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:37.553696 1151 raft_consensus.cc:383] T 438ae684af234c868a419fbab43b0b98 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:37.553757 1151 raft_consensus.cc:738] T 438ae684af234c868a419fbab43b0b98 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:37.553937 1151 consensus_queue.cc:260] T 438ae684af234c868a419fbab43b0b98 P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:37.554565 1151 ts_tablet_manager.cc:1428] T 438ae684af234c868a419fbab43b0b98 P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250901 01:52:37.554862 1151 tablet_bootstrap.cc:492] T bd9b59f0864a497faf281edd6077668d P fd27ab6665304aaea53adbd102dd4f58: Bootstrap starting.
I20250901 01:52:37.556093 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 464) took 4056 ms. Trace:
I20250901 01:52:37.556283 1484 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet a8483e7212d94daba7e5401256c33aef
I20250901 01:52:37.556618 2113 rpcz_store.cc:276] 0901 01:52:33.499669 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:33.499734 (+ 65us) service_pool.cc:225] Handling call
0901 01:52:37.527538 (+4027804us) ts_tablet_manager.cc:1145] Deleting tablet 92f0ad59fbbb435ea872c669a52f854a
0901 01:52:37.527549 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.533001 (+ 5452us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.535806 (+ 2805us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.556072 (+ 20266us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4027556}
I20250901 01:52:37.502336 2956 tablet.cc:1722] T 2dacb608897d47d28512df87dc320927 P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250901 01:52:37.558030 2118 log.cc:1199] T 858218c4427c41e980b2b5c969a5af40 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/858218c4427c41e980b2b5c969a5af40
I20250901 01:52:37.558435 2118 ts_tablet_manager.cc:1939] T 858218c4427c41e980b2b5c969a5af40 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:37.558779 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 342) took 3162 ms. Trace:
I20250901 01:52:37.558845 2118 rpcz_store.cc:276] 0901 01:52:34.396703 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:34.396754 (+ 51us) service_pool.cc:225] Handling call
0901 01:52:37.455531 (+3058777us) ts_tablet_manager.cc:1145] Deleting tablet 858218c4427c41e980b2b5c969a5af40
0901 01:52:37.455542 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.466722 (+ 11180us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.488919 (+ 22197us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.558762 (+ 69843us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3058604}
I20250901 01:52:37.559216 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 858218c4427c41e980b2b5c969a5af40 (table default.Table_6d47266af0ed433eb1f2abbc81d18e69 [id=264cc469a73841e79538843479a165ab]) successfully deleted
I20250901 01:52:37.559331 1492 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: default.Table_63498be0da4f4dfb9fd9e73a721eab13 [id=8a1a77329ab740dda2d6a5d78bbff906] alter complete (version 1)
I20250901 01:52:37.559918 2118 tablet_replica.cc:331] T 508406601c994e42a5c5b1f665e74957 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:37.560112 2118 raft_consensus.cc:2241] T 508406601c994e42a5c5b1f665e74957 P 00c282f779c1477d920f24dc0dc16628 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:37.560308 1488 master_service.cc:948] ReplaceTablet: received request to replace tablet 89d40f0331114490b51de3ba6d773571 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:37.560391 2118 raft_consensus.cc:2270] T 508406601c994e42a5c5b1f665e74957 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:37.560752 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 92f0ad59fbbb435ea872c669a52f854a (table default.Table_4aa01c0e60364804a5e99d552ca3956c [id=59306e9184ca4715b1cd675609807eca]) successfully deleted
I20250901 01:52:37.560997 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 466) took 3954 ms. Trace:
I20250901 01:52:37.561069 2113 rpcz_store.cc:276] 0901 01:52:33.606928 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:33.606989 (+ 61us) service_pool.cc:225] Handling call
0901 01:52:37.560887 (+3953898us) ts_tablet_manager.cc:1145] Deleting tablet 8151bad707c04ac687c5bddfea573e3f
0901 01:52:37.560897 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.560979 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3953712}
I20250901 01:52:37.561061 2118 ts_tablet_manager.cc:1905] T 508406601c994e42a5c5b1f665e74957 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250901 01:52:37.561431 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 8151bad707c04ac687c5bddfea573e3f because the tablet was not found. No further retry: Not found: Tablet not found: 8151bad707c04ac687c5bddfea573e3f
W20250901 01:52:37.561735 1484 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet a8483e7212d94daba7e5401256c33aef: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:37.562132 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 468) took 3949 ms. Trace:
I20250901 01:52:37.562445 893 tablet_service.cc:1515] Processing DeleteTablet for tablet a8483e7212d94daba7e5401256c33aef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:37 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:37.557351 1151 tablet_bootstrap.cc:654] T bd9b59f0864a497faf281edd6077668d P fd27ab6665304aaea53adbd102dd4f58: Neither blocks nor log segments found. Creating new log.
I20250901 01:52:37.562685 2113 rpcz_store.cc:276] 0901 01:52:33.613074 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:33.613144 (+ 70us) service_pool.cc:225] Handling call
0901 01:52:37.562052 (+3948908us) ts_tablet_manager.cc:1145] Deleting tablet 7442c6ff699141e1aff208ebdc9d192a
0901 01:52:37.562061 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.562115 (+ 54us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3948729}
W20250901 01:52:37.563239 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 7442c6ff699141e1aff208ebdc9d192a because the tablet was not found. No further retry: Not found: Tablet not found: 7442c6ff699141e1aff208ebdc9d192a
E20250901 01:52:37.526700 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f5e3316adeae47cd848833dd84b972bb, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_fc962c8bc653498e85cc0bb41639e9ad'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.550745 3020 tablet.cc:1722] T 5afae316ff164413be0d25d4bc0f58dc P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:37.563620 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 472) took 3940 ms. Trace:
E20250901 01:52:37.563386 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 6ce9521b925b4f97bb4a810ce77e147e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_666a4667dfa84c5ca860560cef32d55c'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.564191 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2f61edb50d7a42198bfd70639c7bf4de, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_cd56686217424ab8b95e6bb776d4177a'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.563970 2113 rpcz_store.cc:276] 0901 01:52:33.622797 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:33.623182 (+ 385us) service_pool.cc:225] Handling call
0901 01:52:37.563531 (+3940349us) ts_tablet_manager.cc:1145] Deleting tablet f3962b471bdb49aea6e9fa95f1f9dfe2
0901 01:52:37.563540 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.563605 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3940175}
I20250901 01:52:37.564484 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet a8483e7212d94daba7e5401256c33aef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:37 UTC) from {username='slave'} at 127.0.0.1:35946
W20250901 01:52:37.564958 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet f3962b471bdb49aea6e9fa95f1f9dfe2 because the tablet was not found. No further retry: Not found: Tablet not found: f3962b471bdb49aea6e9fa95f1f9dfe2
I20250901 01:52:37.565269 2113 tablet_replica.cc:331] T 008d23f236c348b6be9aee718f8698af P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:37.565663 2113 raft_consensus.cc:2241] T 008d23f236c348b6be9aee718f8698af P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:37.565593 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_206f62066d2748c7bfba66fe38cf4a17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.566124 2113 raft_consensus.cc:2270] T 008d23f236c348b6be9aee718f8698af P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:37.567243 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_bda0db40078348d69593b06b563db930"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.569217 2613 raft_consensus.cc:491] T 3697821cd29446bb908b020599df0db9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:37.569327 2613 raft_consensus.cc:513] T 3697821cd29446bb908b020599df0db9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:37.569878 2613 leader_election.cc:290] T 3697821cd29446bb908b020599df0db9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:37.569145 2113 ts_tablet_manager.cc:1905] T 008d23f236c348b6be9aee718f8698af P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:37.573053 2613 raft_consensus.cc:491] T 315c15cf0afc44d4a79cb29294281d60 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:37.573203 2613 raft_consensus.cc:513] T 315c15cf0afc44d4a79cb29294281d60 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:37.573864 2613 leader_election.cc:290] T 315c15cf0afc44d4a79cb29294281d60 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:37.574702 1485 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet fd2ce82a4ee24d0faa44980a3967e187
I20250901 01:52:37.574753 1151 tablet_bootstrap.cc:492] T bd9b59f0864a497faf281edd6077668d P fd27ab6665304aaea53adbd102dd4f58: No bootstrap required, opened a new log
I20250901 01:52:37.574898 1151 ts_tablet_manager.cc:1397] T bd9b59f0864a497faf281edd6077668d P fd27ab6665304aaea53adbd102dd4f58: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.002s
I20250901 01:52:37.576157 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b54d2365b72d4cbaa58679a75526e23a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.576453 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_31af121dc55840a7b6c179bb60a1936a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:37.569898 992 leader_election.cc:343] T c422881b40f54971a68530abfbd1945b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: c422881b40f54971a68530abfbd1945b
I20250901 01:52:37.576700 1151 raft_consensus.cc:357] T bd9b59f0864a497faf281edd6077668d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:37.576872 1151 raft_consensus.cc:383] T bd9b59f0864a497faf281edd6077668d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250901 01:52:37.576939 1151 raft_consensus.cc:738] T bd9b59f0864a497faf281edd6077668d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Initialized, Role: FOLLOWER
I20250901 01:52:37.576157 1488 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 89d40f0331114490b51de3ba6d773571
W20250901 01:52:37.577096 1488 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 89d40f0331114490b51de3ba6d773571: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:37.577168 1151 consensus_queue.cc:260] T bd9b59f0864a497faf281edd6077668d P fd27ab6665304aaea53adbd102dd4f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:37.577759 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 89d40f0331114490b51de3ba6d773571 with delete_type TABLET_DATA_DELETED (replaced by tablet 2e52ec21fa72424f92c945f4804e2059) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:37.577770 1151 ts_tablet_manager.cc:1428] T bd9b59f0864a497faf281edd6077668d P fd27ab6665304aaea53adbd102dd4f58: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250901 01:52:37.578154 1488 catalog_manager.cc:6652] ReplaceTablet: tablet 89d40f0331114490b51de3ba6d773571 deleted and replaced by tablet 2e52ec21fa72424f92c945f4804e2059
I20250901 01:52:37.578217 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 89d40f0331114490b51de3ba6d773571 with delete_type TABLET_DATA_DELETED (replaced by tablet 2e52ec21fa72424f92c945f4804e2059) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:37.581517 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5e91ee6c48f340ffa8060ae420d88c5e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.581637 1696 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet fd2ce82a4ee24d0faa44980a3967e187
I20250901 01:52:37.576759 992 leader_election.cc:304] T c422881b40f54971a68530abfbd1945b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:37.583961 2613 raft_consensus.cc:2747] T c422881b40f54971a68530abfbd1945b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:37.584278 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f353799464364ccdbd25742c84ed61d7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.585371 2954 tablet.cc:1722] T a8483e7212d94daba7e5401256c33aef P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250901 01:52:37.585644 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_25d32a7da6b244a482c96abead0f3f77"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:37.586627 992 consensus_peers.cc:489] T 5adf6b492640417ca071c94702b48cb9 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Couldn't send request to peer 00c282f779c1477d920f24dc0dc16628. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5adf6b492640417ca071c94702b48cb9. This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:37.591076 2954 tablet.cc:1722] T 81c36693199945d98465579b509852e2 P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250901 01:52:37.594251 992 consensus_peers.cc:489] T b164b4a67a9143ceaff69b0ce49d63fe P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250901 01:52:37.594393 992 consensus_peers.cc:489] T 81c36693199945d98465579b509852e2 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250901 01:52:37.594574 992 leader_election.cc:336] T 5d68ea97c71942c7b55b967ba870e0d7 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:37.595973 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "315c15cf0afc44d4a79cb29294281d60" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:37.596201 913 raft_consensus.cc:2466] T 315c15cf0afc44d4a79cb29294281d60 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:37.599990 2113 ts_tablet_manager.cc:1918] T 008d23f236c348b6be9aee718f8698af P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:37.601023 1482 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_d24e3a18ef4843fdb7b74ce23d47cf9e" } modify_external_catalogs: true
I20250901 01:52:37.601275 1482 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_d24e3a18ef4843fdb7b74ce23d47cf9e" } modify_external_catalogs: true
I20250901 01:52:37.607127 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c91083f84c414855a9fa77538439a0a7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.609272 1485 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_8ad4807386b64e4195236b6fe222ad8e" } modify_external_catalogs: true
I20250901 01:52:37.609509 1485 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_8ad4807386b64e4195236b6fe222ad8e" } modify_external_catalogs: true
I20250901 01:52:37.610620 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_819cd96147d24cfb8549661ae486ec3c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.611737 1488 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_df16f2a0280645a0949bc071a2af532f" } modify_external_catalogs: true
I20250901 01:52:37.611881 1488 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_df16f2a0280645a0949bc071a2af532f" } modify_external_catalogs: true
I20250901 01:52:37.612041 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a4c929c045b54918a5988fb1598f5729"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.612950 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_db5943df746d47829cd5411a25344457"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.613804 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_60c33e8fed1a4629a26991a3eee5c499"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.614701 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_17fab05ff1454a8abf0b609b7c88e065"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.615696 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_967696e82fc9454ca7fe5c3748d8b75e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.618408 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_61621961cdab47afb1bed4fb0056ac5d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.619359 1492 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_8db3eec03fc7427096bb410c9da63608" } new_table_name: "default.Table_9f14a37542af45978c2a16cfd925c35c" modify_external_catalogs: true
E20250901 01:52:37.621975 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 494b6488aa0d4355b4caa46004267465, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_dc77f0e1986d409dbf72b15b05150586'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.620280 1482 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 5a7fa68d29644ae8afd2711e39dde59a
I20250901 01:52:37.622413 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f2b57c3a6065453b9aa8961bb827736b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.629396 2956 tablet.cc:1722] T a8483e7212d94daba7e5401256c33aef P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20250901 01:52:37.630173 1492 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ff2bd017417c41b5b8225b8f00283e63 has no consensus state
I20250901 01:52:37.630335 3020 tablet.cc:1722] T 81c36693199945d98465579b509852e2 P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:37.631953 1484 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_335c0676987d4bf3a4d64b239be8067e" } new_table_name: "default.Table_2847042ef2784ca3b387e6bbf4360167" modify_external_catalogs: true
I20250901 01:52:37.651577 2118 ts_tablet_manager.cc:1918] T 508406601c994e42a5c5b1f665e74957 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:37.652729 2887 webserver.cc:480] Webserver started at http://127.0.104.61:46533/ using document root <none> and password file <none>
I20250901 01:52:37.652900 2118 log.cc:1199] T 508406601c994e42a5c5b1f665e74957 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/508406601c994e42a5c5b1f665e74957
I20250901 01:52:37.654038 2887 fs_manager.cc:362] Metadata directory not provided
I20250901 01:52:37.654201 2887 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
E20250901 01:52:37.622109 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 01deaae6abb141789f0cd240f3058c73, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7baa92b46d98454a891222be49a6ca48'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.654922 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9f62a9d97f514747b940b488907369f3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d53a456dbca64daf92aeb5fb4517b417'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.656299 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ff2bd017417c41b5b8225b8f00283e63, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_9f14a37542af45978c2a16cfd925c35c'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.656335 2118 ts_tablet_manager.cc:1939] T 508406601c994e42a5c5b1f665e74957 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:37.656702 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 344) took 3247 ms. Trace:
I20250901 01:52:37.656808 2118 rpcz_store.cc:276] 0901 01:52:34.409464 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:34.409516 (+ 52us) service_pool.cc:225] Handling call
0901 01:52:37.559875 (+3150359us) ts_tablet_manager.cc:1145] Deleting tablet 508406601c994e42a5c5b1f665e74957
0901 01:52:37.559884 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.639139 (+ 79255us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.651483 (+ 12344us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.656688 (+ 5205us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3150131}
E20250901 01:52:37.656751 1696 catalog_manager.cc:6205] Error selecting replicas for tablet dcc60bb000f440f1aca9133928b5ce20, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_5a19f7d6321b49249c4edd7afb1d24ef'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.657181 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 465c02a130c54e09b5a74fb4dd2ef762, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_8e4ca21f21734e049d174baa0a9335a4'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.657244 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 508406601c994e42a5c5b1f665e74957 (table default.Table_ebba044988d5435aba124b1f971356d1 [id=4aabc328f6374ec5b1dea26f778208b2]) successfully deleted
I20250901 01:52:37.657429 2118 tablet_replica.cc:331] T 4b4aa063ffe948e69e47f03102581899 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:37.657586 2118 raft_consensus.cc:2241] T 4b4aa063ffe948e69e47f03102581899 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:37.657768 2118 raft_consensus.cc:2270] T 4b4aa063ffe948e69e47f03102581899 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
E20250901 01:52:37.657636 1696 catalog_manager.cc:6205] Error selecting replicas for tablet a5609f111ab34c0ea9fabdbce06c81f9, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_4505d77e4269443995012fef50c7e7b5'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.657981 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9f2416b2cc144054a8d2effc64157615, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_69575b1a9efd4573a2ee01d3f432800b'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.658223 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 75883d9fd00e429aa8853903a9fdd7e4, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_a9f6ac8609554a38ab632b2a3f7113d8'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.658282 2118 ts_tablet_manager.cc:1905] T 4b4aa063ffe948e69e47f03102581899 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
E20250901 01:52:37.658587 1696 catalog_manager.cc:6205] Error selecting replicas for tablet b1ba0e19dc0b462e80c537bf2d183f5b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_f9cac02fe78f4902a8c730f2706f7139'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.658882 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 086aa948f93644da8e5835b0fac87ddd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b4fbc34da806475c912052d0fab3fc53'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.659116 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 082db24bb3fd4089950504145a20a075, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ebba044988d5435aba124b1f971356d1'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.659387 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 93aa22afc0ad4d7fa24807ea3c5d5c21, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0f4b4094d2974b7591875375eda6bed0'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.659638 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 97418fffe6614dfba4ebd13a6ed57e67, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_360f4423e7f341339188b71cc82a1149'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.659876 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 3a62034e688348ed9e2ac9de009ed3a2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_90623cc9a13044b5934c922f14b85470'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.660125 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 18498c9cb09141d08c1c5c0854b63ebd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_df16f2a0280645a0949bc071a2af532f'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.660372 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f5e3316adeae47cd848833dd84b972bb, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_fc962c8bc653498e85cc0bb41639e9ad'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.660710 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 6ce9521b925b4f97bb4a810ce77e147e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_666a4667dfa84c5ca860560cef32d55c'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.661067 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2f61edb50d7a42198bfd70639c7bf4de, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_cd56686217424ab8b95e6bb776d4177a'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.661394 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2e52ec21fa72424f92c945f4804e2059, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_9fb44a0a69994184bc2ffe62c55d2c25'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.632452 2911 raft_consensus.cc:491] T ced4813805004bba9c4667a79ac201fc P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250901 01:52:37.632540 992 leader_election.cc:343] T 3697821cd29446bb908b020599df0db9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 3697821cd29446bb908b020599df0db9
I20250901 01:52:37.665879 2911 raft_consensus.cc:513] T ced4813805004bba9c4667a79ac201fc P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:37.661939 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_194718a85aaa4b958f9927b5e3150c9f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.666069 992 leader_election.cc:304] T 315c15cf0afc44d4a79cb29294281d60 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:37.666400 2613 raft_consensus.cc:2802] T 315c15cf0afc44d4a79cb29294281d60 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:37.666476 2613 raft_consensus.cc:491] T 315c15cf0afc44d4a79cb29294281d60 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:37.666555 2613 raft_consensus.cc:3058] T 315c15cf0afc44d4a79cb29294281d60 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:37.666740 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_67c1cffa203e4a82aec57506503520d4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.667340 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5977271e82c6422e842b5283bb371d88"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.663239 1491 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_19fb556a9c4941709db9cd00c545c0eb" } new_table_name: "default.Table_46b75a7344104e2f9ad89f26512d0a0a" modify_external_catalogs: true
I20250901 01:52:37.664965 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5c7b7b0410724865b449f54c32d055ec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.669976 2613 raft_consensus.cc:513] T 315c15cf0afc44d4a79cb29294281d60 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:37.670490 2613 leader_election.cc:290] T 315c15cf0afc44d4a79cb29294281d60 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:37.664923 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2a51707fe874413c8bfa0604198fa4f6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.665409 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d54ac63a9ae14f73b75e168faf1cff9c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.665511 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_cf007941e8224329980599119efa6f7a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.665695 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_0d1412b7494b496f99f5de21f2383e94"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.665946 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4997a198b46c4d5b822f74a20f5dd208"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.674641 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b742fdc3954d4e9491d87f8da735804a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.680487 2113 log.cc:1199] T 008d23f236c348b6be9aee718f8698af P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/008d23f236c348b6be9aee718f8698af
I20250901 01:52:37.680938 2113 ts_tablet_manager.cc:1939] T 008d23f236c348b6be9aee718f8698af P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:37.681818 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 473) took 4058 ms. Trace:
I20250901 01:52:37.681938 2113 rpcz_store.cc:276] 0901 01:52:33.623484 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:33.623532 (+ 48us) service_pool.cc:225] Handling call
0901 01:52:37.565220 (+3941688us) ts_tablet_manager.cc:1145] Deleting tablet 008d23f236c348b6be9aee718f8698af
0901 01:52:37.565228 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.592505 (+ 27277us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.599885 (+ 7380us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.681794 (+ 81909us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3941544}
I20250901 01:52:37.682390 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 008d23f236c348b6be9aee718f8698af (table default.Table_f9cac02fe78f4902a8c730f2706f7139 [id=0d9b514272444aa1914c4ff033b8f611]) successfully deleted
I20250901 01:52:37.682814 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 474) took 4059 ms. Trace:
I20250901 01:52:37.682912 2113 rpcz_store.cc:276] 0901 01:52:33.623802 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:33.623877 (+ 75us) service_pool.cc:225] Handling call
0901 01:52:37.682696 (+4058819us) ts_tablet_manager.cc:1145] Deleting tablet b85d64e02cc646e380581ac9f59dee9e
0901 01:52:37.682706 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.682798 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4058678}
W20250901 01:52:37.683310 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet b85d64e02cc646e380581ac9f59dee9e because the tablet was not found. No further retry: Not found: Tablet not found: b85d64e02cc646e380581ac9f59dee9e
I20250901 01:52:37.685467 2113 tablet_replica.cc:331] T 088ef4861dc14f81ae470c3aaee5b4e1 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:37.685737 2113 raft_consensus.cc:2241] T 088ef4861dc14f81ae470c3aaee5b4e1 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:37.685956 2113 raft_consensus.cc:2270] T 088ef4861dc14f81ae470c3aaee5b4e1 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:37.689904 2113 ts_tablet_manager.cc:1905] T 088ef4861dc14f81ae470c3aaee5b4e1 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:37.693226 2118 ts_tablet_manager.cc:1918] T 4b4aa063ffe948e69e47f03102581899 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:37.693849 1488 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 18498c9cb09141d08c1c5c0854b63ebd
W20250901 01:52:37.694406 992 consensus_peers.cc:489] T d0b7c84ec9394a20ad77add3808e555a P fd27ab6665304aaea53adbd102dd4f58 -> Peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Couldn't send request to peer 00c282f779c1477d920f24dc0dc16628. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d0b7c84ec9394a20ad77add3808e555a. This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:37.694645 992 leader_election.cc:304] T 5d68ea97c71942c7b55b967ba870e0d7 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:37.695235 2613 raft_consensus.cc:2802] T 5d68ea97c71942c7b55b967ba870e0d7 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:37.695657 2613 raft_consensus.cc:695] T 5d68ea97c71942c7b55b967ba870e0d7 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:37.696643 2613 consensus_queue.cc:237] T 5d68ea97c71942c7b55b967ba870e0d7 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:37.697328 992 leader_election.cc:336] T 3697821cd29446bb908b020599df0db9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:37.697464 992 leader_election.cc:304] T 3697821cd29446bb908b020599df0db9 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:37.697690 2948 raft_consensus.cc:2747] T 3697821cd29446bb908b020599df0db9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250901 01:52:37.697786 992 leader_election.cc:336] T 315c15cf0afc44d4a79cb29294281d60 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:37.698314 1485 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 2936fdd7f303487db7b03dccff71867d
W20250901 01:52:37.698467 1485 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 2936fdd7f303487db7b03dccff71867d: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:37.699245 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 2936fdd7f303487db7b03dccff71867d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:37 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:37.704123 3022 tablet.cc:1722] T 3c1fd0820d8a4995abe027f685da7f7b P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250901 01:52:37.707616 1485 master_service.cc:948] ReplaceTablet: received request to replace tablet b4278a013ae84082bc7a8497b04ac147 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:37.708307 1485 master_service.cc:948] ReplaceTablet: received request to replace tablet ea846f88b3534beebd0001b7f17b5ca9 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:37.708902 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_23400604e4e843408e61638ce1ee5b7a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.710775 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a1c89274399a4955bd672273ebb1078c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.711474 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_af65ecf27d9e423ea385375f10cb8fb5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.712136 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_90cba39d98d14653bb905b41504d173b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:37.712364 992 leader_election.cc:336] T 315c15cf0afc44d4a79cb29294281d60 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:37.709026 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e5dce81eae394ac58199160bfc1e8d7d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.709143 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_45504cb6d35d40ae90865f7e0fae1699"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.709260 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3b58a33c41704b3d965e80a1254ac368"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.709386 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_0e44adc56ecd40cb818e39db16e30742"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.708560 1491 master_service.cc:948] ReplaceTablet: received request to replace tablet 858218c4427c41e980b2b5c969a5af40 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:37.709514 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_bcf597e0cfbb40828686025019c84508"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.709545 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_16ca91047cdf4ae6968a0a26b045580f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.708482 1488 master_service.cc:948] ReplaceTablet: received request to replace tablet 6c0abbfc8d904dce989ce51961e4237d from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:37.699934 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 2936fdd7f303487db7b03dccff71867d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:37 UTC) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:37.702541 911 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "315c15cf0afc44d4a79cb29294281d60" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:37.717715 911 raft_consensus.cc:3058] T 315c15cf0afc44d4a79cb29294281d60 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
E20250901 01:52:37.707683 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 494b6488aa0d4355b4caa46004267465, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_dc77f0e1986d409dbf72b15b05150586'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.718029 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 01deaae6abb141789f0cd240f3058c73, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7baa92b46d98454a891222be49a6ca48'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.718159 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9f62a9d97f514747b940b488907369f3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d53a456dbca64daf92aeb5fb4517b417'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.718281 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ff2bd017417c41b5b8225b8f00283e63, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_9f14a37542af45978c2a16cfd925c35c'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.718412 1696 catalog_manager.cc:6205] Error selecting replicas for tablet dcc60bb000f440f1aca9133928b5ce20, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_5a19f7d6321b49249c4edd7afb1d24ef'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.718519 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 465c02a130c54e09b5a74fb4dd2ef762, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_8e4ca21f21734e049d174baa0a9335a4'. Need at least 3 replicas, but only 2 tablet servers are available
W20250901 01:52:37.709748 1474 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.61:42375, user_credentials={real_user=slave}} blocked reactor thread for 55204.9us
E20250901 01:52:37.718627 1696 catalog_manager.cc:6205] Error selecting replicas for tablet a5609f111ab34c0ea9fabdbce06c81f9, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_4505d77e4269443995012fef50c7e7b5'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.719720 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9f2416b2cc144054a8d2effc64157615, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_69575b1a9efd4573a2ee01d3f432800b'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.719858 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 75883d9fd00e429aa8853903a9fdd7e4, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_a9f6ac8609554a38ab632b2a3f7113d8'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.719966 1696 catalog_manager.cc:6205] Error selecting replicas for tablet b1ba0e19dc0b462e80c537bf2d183f5b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_f9cac02fe78f4902a8c730f2706f7139'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.720062 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 086aa948f93644da8e5835b0fac87ddd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b4fbc34da806475c912052d0fab3fc53'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.720165 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 082db24bb3fd4089950504145a20a075, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ebba044988d5435aba124b1f971356d1'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.720258 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 93aa22afc0ad4d7fa24807ea3c5d5c21, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0f4b4094d2974b7591875375eda6bed0'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.720358 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 97418fffe6614dfba4ebd13a6ed57e67, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_360f4423e7f341339188b71cc82a1149'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.720480 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 3a62034e688348ed9e2ac9de009ed3a2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_90623cc9a13044b5934c922f14b85470'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.720580 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f5e3316adeae47cd848833dd84b972bb, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_fc962c8bc653498e85cc0bb41639e9ad'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.720681 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 6ce9521b925b4f97bb4a810ce77e147e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_666a4667dfa84c5ca860560cef32d55c'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.720786 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2f61edb50d7a42198bfd70639c7bf4de, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_cd56686217424ab8b95e6bb776d4177a'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.720993 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2e52ec21fa72424f92c945f4804e2059, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_9fb44a0a69994184bc2ffe62c55d2c25'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.721004 2118 log.cc:1199] T 4b4aa063ffe948e69e47f03102581899 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/4b4aa063ffe948e69e47f03102581899
I20250901 01:52:37.721426 2118 ts_tablet_manager.cc:1939] T 4b4aa063ffe948e69e47f03102581899 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:37.724479 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 347) took 3033 ms. Trace:
W20250901 01:52:37.725687 992 consensus_peers.cc:489] T 482a9a2cb81f448bb2823892e35920d4 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Couldn't send request to peer 00c282f779c1477d920f24dc0dc16628. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 482a9a2cb81f448bb2823892e35920d4. This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:37.727322 3022 tablet.cc:1722] T 9127f9c5239e445ba86012951b7c75d9 P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250901 01:52:37.727829 2118 rpcz_store.cc:276] 0901 01:52:34.691139 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:34.691185 (+ 46us) service_pool.cc:225] Handling call
0901 01:52:37.657389 (+2966204us) ts_tablet_manager.cc:1145] Deleting tablet 4b4aa063ffe948e69e47f03102581899
0901 01:52:37.657398 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.661186 (+ 3788us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.693119 (+ 31933us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.724457 (+ 31338us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2966042}
I20250901 01:52:37.728315 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 4b4aa063ffe948e69e47f03102581899 (table default.Table_14e7f1e7dd314cdcaee09ac2fdf28993 [id=76e521ff309947a58840a7135e783477]) successfully deleted
I20250901 01:52:37.728677 2118 tablet_replica.cc:331] T 5f5396045979450193d461ba1334a5c9 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:37.728896 2118 raft_consensus.cc:2241] T 5f5396045979450193d461ba1334a5c9 P 00c282f779c1477d920f24dc0dc16628 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:37.729305 2118 raft_consensus.cc:2270] T 5f5396045979450193d461ba1334a5c9 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250901 01:52:37.729715 1465 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet ff2bd017417c41b5b8225b8f00283e63 (table default.Table_9f14a37542af45978c2a16cfd925c35c [id=305490ecfddb4d46927f7bb128c7f878], current schema version=4) failed: Not found: failed to reset TS proxy: No leader found: Tablet ff2bd017417c41b5b8225b8f00283e63 has no consensus state
I20250901 01:52:37.729971 2118 ts_tablet_manager.cc:1905] T 5f5396045979450193d461ba1334a5c9 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:37.730273 1484 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_63498be0da4f4dfb9fd9e73a721eab13" } new_table_name: "default.Table_e40051a54d4a4d6784a666e826d4fa72" modify_external_catalogs: true
I20250901 01:52:37.733314 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 438ae684af234c868a419fbab43b0b98 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:24 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:37.734674 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet bd9b59f0864a497faf281edd6077668d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:31 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:37.723151 911 raft_consensus.cc:2466] T 315c15cf0afc44d4a79cb29294281d60 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:37.738375 992 leader_election.cc:304] T 315c15cf0afc44d4a79cb29294281d60 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters: 0cf599f74322478d91475df234e58965
I20250901 01:52:37.738425 2911 leader_election.cc:290] T ced4813805004bba9c4667a79ac201fc P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:37.738714 2911 raft_consensus.cc:2802] T 315c15cf0afc44d4a79cb29294281d60 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:37.740258 919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ced4813805004bba9c4667a79ac201fc" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:37.740466 919 raft_consensus.cc:2466] T ced4813805004bba9c4667a79ac201fc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:37.738878 2911 raft_consensus.cc:695] T 315c15cf0afc44d4a79cb29294281d60 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:37.742484 2911 consensus_queue.cc:237] T 315c15cf0afc44d4a79cb29294281d60 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:37.744073 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_67464dc9cab1453bbf45fc1279d38ee7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.744132 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_523353800b3a42cbb3f9ac57093ba974"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.744248 3020 tablet.cc:1722] T 3c1fd0820d8a4995abe027f685da7f7b P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250901 01:52:37.745131 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b2281d926ac34d95b4efc48d8a57a1d4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.745970 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_712d516e3d2b421aac0e464fbf98cb7f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.752084 1484 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: default.Table_ba32ff5aeabb4d0caa98298e418ab717 [id=b3914f33e856466e8479d0a2263ec56f] alter complete (version 3)
I20250901 01:52:37.752897 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ff9f1aa0aefa4c73b5af99372146fced"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.753840 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c7acd90b00fa4e5199543434f849226e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.754675 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d5c15c40e09541ceb8b3c17ae9736ace"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.755399 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_364e4a20f6d34e65978114cf3989bff1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.755813 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b443cb6fbaa64b0085948a3e7ff34cc7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.756219 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_96968e2317cc4987b8045ca78de6b4b6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.761117 3020 tablet.cc:1722] T 9127f9c5239e445ba86012951b7c75d9 P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
E20250901 01:52:37.762689 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 494b6488aa0d4355b4caa46004267465, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_dc77f0e1986d409dbf72b15b05150586'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.763950 992 leader_election.cc:304] T ced4813805004bba9c4667a79ac201fc P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:37.766394 3030 raft_consensus.cc:2802] T ced4813805004bba9c4667a79ac201fc P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:37.766541 3030 raft_consensus.cc:491] T ced4813805004bba9c4667a79ac201fc P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:37.766660 3030 raft_consensus.cc:3058] T ced4813805004bba9c4667a79ac201fc P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:37.767235 3022 tablet.cc:1722] T 5afae316ff164413be0d25d4bc0f58dc P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250901 01:52:37.769129 3030 raft_consensus.cc:513] T ced4813805004bba9c4667a79ac201fc P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:37.773950 3030 leader_election.cc:290] T ced4813805004bba9c4667a79ac201fc P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
W20250901 01:52:37.775758 992 leader_election.cc:336] T ced4813805004bba9c4667a79ac201fc P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:37.777066 919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ced4813805004bba9c4667a79ac201fc" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:37.777220 919 raft_consensus.cc:3058] T ced4813805004bba9c4667a79ac201fc P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:37.777685 3020 tablet.cc:1722] T 5afae316ff164413be0d25d4bc0f58dc P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250901 01:52:37.779819 919 raft_consensus.cc:2466] T ced4813805004bba9c4667a79ac201fc P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:37.783010 2887 fs_manager.cc:714] Time spent opening directory manager: real 0.090s user 0.005s sys 0.000s
I20250901 01:52:37.783330 3029 consensus_queue.cc:1035] T c83ce95fd6fa41478de0e2168be201b1 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:37.773016 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_186ee9c199364bf582878e457cf588b4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:37.789203 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 01deaae6abb141789f0cd240f3058c73, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7baa92b46d98454a891222be49a6ca48'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.789290 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9f62a9d97f514747b940b488907369f3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d53a456dbca64daf92aeb5fb4517b417'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.789347 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ff2bd017417c41b5b8225b8f00283e63, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_9f14a37542af45978c2a16cfd925c35c'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.789402 1696 catalog_manager.cc:6205] Error selecting replicas for tablet dcc60bb000f440f1aca9133928b5ce20, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_5a19f7d6321b49249c4edd7afb1d24ef'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.789460 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 465c02a130c54e09b5a74fb4dd2ef762, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_8e4ca21f21734e049d174baa0a9335a4'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.789542 1696 catalog_manager.cc:6205] Error selecting replicas for tablet a5609f111ab34c0ea9fabdbce06c81f9, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_4505d77e4269443995012fef50c7e7b5'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.773653 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_bad3b26a18a340e3803c7dfe692c554a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:37.789619 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9f2416b2cc144054a8d2effc64157615, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_69575b1a9efd4573a2ee01d3f432800b'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.774410 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e062b31dd53b46168b5f803a75c08555"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:37.789685 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 75883d9fd00e429aa8853903a9fdd7e4, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_a9f6ac8609554a38ab632b2a3f7113d8'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.789742 1696 catalog_manager.cc:6205] Error selecting replicas for tablet b1ba0e19dc0b462e80c537bf2d183f5b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_f9cac02fe78f4902a8c730f2706f7139'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.789800 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 086aa948f93644da8e5835b0fac87ddd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b4fbc34da806475c912052d0fab3fc53'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.789858 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 082db24bb3fd4089950504145a20a075, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ebba044988d5435aba124b1f971356d1'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.789912 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 93aa22afc0ad4d7fa24807ea3c5d5c21, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0f4b4094d2974b7591875375eda6bed0'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.789968 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 97418fffe6614dfba4ebd13a6ed57e67, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_360f4423e7f341339188b71cc82a1149'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.790020 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 3a62034e688348ed9e2ac9de009ed3a2, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_90623cc9a13044b5934c922f14b85470'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.790077 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f5e3316adeae47cd848833dd84b972bb, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_fc962c8bc653498e85cc0bb41639e9ad'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.790132 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 6ce9521b925b4f97bb4a810ce77e147e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_666a4667dfa84c5ca860560cef32d55c'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.790184 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2f61edb50d7a42198bfd70639c7bf4de, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_cd56686217424ab8b95e6bb776d4177a'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.790247 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2e52ec21fa72424f92c945f4804e2059, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_9fb44a0a69994184bc2ffe62c55d2c25'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.790407 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_eba93e07392b4696a689ce3d8f0d7f21"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.791149 1482 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_d28d7742171c4465a8b438d2e337b5f7" } new_table_name: "default.Table_8e94a246afd14f90afc30853e6ab1d1c" modify_external_catalogs: true
I20250901 01:52:37.766661 2113 ts_tablet_manager.cc:1918] T 088ef4861dc14f81ae470c3aaee5b4e1 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:37.791535 2113 log.cc:1199] T 088ef4861dc14f81ae470c3aaee5b4e1 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/088ef4861dc14f81ae470c3aaee5b4e1
I20250901 01:52:37.791667 992 leader_election.cc:304] T ced4813805004bba9c4667a79ac201fc P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:37.791863 2113 ts_tablet_manager.cc:1939] T 088ef4861dc14f81ae470c3aaee5b4e1 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:37.774827 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_67690133114348e7a38d2b0e4fa74db6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.774842 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9a761c12f6834c56849fdf2d9d84f82e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.775215 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_0a6c4bf3c520403391ae920fe210d36c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.795589 3029 raft_consensus.cc:2802] T ced4813805004bba9c4667a79ac201fc P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:37.795698 3029 raft_consensus.cc:695] T ced4813805004bba9c4667a79ac201fc P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:37.795943 3029 consensus_queue.cc:237] T ced4813805004bba9c4667a79ac201fc P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:37.798372 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 475) took 4173 ms. Trace:
I20250901 01:52:37.798462 2113 rpcz_store.cc:276] 0901 01:52:33.624457 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:33.624587 (+ 130us) service_pool.cc:225] Handling call
0901 01:52:37.685028 (+4060441us) ts_tablet_manager.cc:1145] Deleting tablet 088ef4861dc14f81ae470c3aaee5b4e1
0901 01:52:37.685040 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.763617 (+ 78577us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.766581 (+ 2964us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.798351 (+ 31770us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5747,"tablet-delete.queue_time_us":4060288}
I20250901 01:52:37.798873 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 477) took 4173 ms. Trace:
I20250901 01:52:37.798949 2113 rpcz_store.cc:276] 0901 01:52:33.625468 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:33.625818 (+ 350us) service_pool.cc:225] Handling call
0901 01:52:37.798766 (+4172948us) ts_tablet_manager.cc:1145] Deleting tablet e2aa32450a584395ae07b49f76a7b26d
0901 01:52:37.798776 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.798854 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4172791}
I20250901 01:52:37.799247 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 471) took 4176 ms. Trace:
I20250901 01:52:37.799317 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 088ef4861dc14f81ae470c3aaee5b4e1 (table default.Table_666a4667dfa84c5ca860560cef32d55c [id=d68960dbe1154bd8a8016f9a5718e5ad]) successfully deleted
I20250901 01:52:37.799317 2113 rpcz_store.cc:276] 0901 01:52:33.622676 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:33.626059 (+ 3383us) service_pool.cc:225] Handling call
0901 01:52:37.799175 (+4173116us) ts_tablet_manager.cc:1145] Deleting tablet 74b709cb5ec64006b2f12bead8217d80
0901 01:52:37.799185 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.799233 (+ 48us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4172971}
I20250901 01:52:37.799552 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 478) took 4174 ms. Trace:
I20250901 01:52:37.799638 2113 rpcz_store.cc:276] 0901 01:52:33.625574 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:33.626533 (+ 959us) service_pool.cc:225] Handling call
0901 01:52:37.799488 (+4172955us) ts_tablet_manager.cc:1145] Deleting tablet e8e7d2a72f704f52a8f8d46911a8a9e4
0901 01:52:37.799495 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.799539 (+ 44us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4172791}
I20250901 01:52:37.799886 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 479) took 4174 ms. Trace:
I20250901 01:52:37.799959 2113 rpcz_store.cc:276] 0901 01:52:33.625662 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:33.626733 (+ 1071us) service_pool.cc:225] Handling call
0901 01:52:37.799817 (+4173084us) ts_tablet_manager.cc:1145] Deleting tablet 75345ea2a3dc40fc8c9f8090d8839d2d
0901 01:52:37.799826 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.799873 (+ 47us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4172951}
I20250901 01:52:37.800215 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 480) took 4174 ms. Trace:
W20250901 01:52:37.800305 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet e2aa32450a584395ae07b49f76a7b26d because the tablet was not found. No further retry: Not found: Tablet not found: e2aa32450a584395ae07b49f76a7b26d
I20250901 01:52:37.800275 2113 rpcz_store.cc:276] 0901 01:52:33.625746 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:33.626896 (+ 1150us) service_pool.cc:225] Handling call
0901 01:52:37.800150 (+4173254us) ts_tablet_manager.cc:1145] Deleting tablet cb7d7e1796c0456f8de98fb812b9e374
0901 01:52:37.800158 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.800203 (+ 45us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4173118}
I20250901 01:52:37.774956 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_435cf677906d43e08367b8bc141e9ba2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:37.800523 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 74b709cb5ec64006b2f12bead8217d80 because the tablet was not found. No further retry: Not found: Tablet not found: 74b709cb5ec64006b2f12bead8217d80
W20250901 01:52:37.800655 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet e8e7d2a72f704f52a8f8d46911a8a9e4 because the tablet was not found. No further retry: Not found: Tablet not found: e8e7d2a72f704f52a8f8d46911a8a9e4
W20250901 01:52:37.800783 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 75345ea2a3dc40fc8c9f8090d8839d2d because the tablet was not found. No further retry: Not found: Tablet not found: 75345ea2a3dc40fc8c9f8090d8839d2d
I20250901 01:52:37.801441 2113 tablet_replica.cc:331] T 50472a7aa2f04e0fbd65ac6c86c34b3e P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:37.801828 2113 raft_consensus.cc:2241] T 50472a7aa2f04e0fbd65ac6c86c34b3e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250901 01:52:37.802062 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet cb7d7e1796c0456f8de98fb812b9e374 because the tablet was not found. No further retry: Not found: Tablet not found: cb7d7e1796c0456f8de98fb812b9e374
I20250901 01:52:37.802888 2113 raft_consensus.cc:2270] T 50472a7aa2f04e0fbd65ac6c86c34b3e P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:37.777772 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_25e834f86af9411a94fb950cccecdf80"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.780339 1491 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: default.Table_2847042ef2784ca3b387e6bbf4360167 [id=ab670bb02c6e4ed491e7831ad73cdfc6] alter complete (version 2)
I20250901 01:52:37.786084 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_cacf070d8a0c481bab54b206cec7998b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.805279 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_976434bedd8846a7ad0b7e3d3c4d91b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.806221 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_65cb18c8712f408e9b5c4601521b5006"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.807329 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8d98d9adf20a4b62bf70334bfd63c4ab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.808045 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_53f0ea61afa84a4daab6050ca5fdc2ff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.808110 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_510c6d62204e4e678ff0214fae695707"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.808959 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a1a2b46024bb4effa19510ccd2701997"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:37.809291 992 leader_election.cc:336] T ced4813805004bba9c4667a79ac201fc P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
W20250901 01:52:37.809615 992 consensus_peers.cc:489] T 5afae316ff164413be0d25d4bc0f58dc P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250901 01:52:37.809706 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a3a2c5771cb04b8da8eaa2231dd3a4b6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.810473 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_72b6e000266d4b86ba36dc65c11c726c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.811280 1491 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_dc77f0e1986d409dbf72b15b05150586" } modify_external_catalogs: true
I20250901 01:52:37.811344 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f83d5d6a98f74f96a9b1812886e29ceb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.811422 1491 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_dc77f0e1986d409dbf72b15b05150586" } modify_external_catalogs: true
I20250901 01:52:37.812393 2113 ts_tablet_manager.cc:1905] T 50472a7aa2f04e0fbd65ac6c86c34b3e P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:37.814392 1483 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_90623cc9a13044b5934c922f14b85470" } modify_external_catalogs: true
I20250901 01:52:37.816455 1483 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_90623cc9a13044b5934c922f14b85470" } modify_external_catalogs: true
I20250901 01:52:37.820154 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_37778fdac5e340ffbab2cebd1aab750f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.820958 1484 master_service.cc:948] ReplaceTablet: received request to replace tablet d0c3c363dda444519a17dca63ad9caa7 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:37.821954 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_65d153df4cb841518415d6afcc57f27d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.822445 1491 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 494b6488aa0d4355b4caa46004267465
I20250901 01:52:37.822957 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_6a73edd06aba494195e0c40c13d5bf5d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.822994 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_91341830779949899d9deaad1e8c109e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.826675 1483 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 3a62034e688348ed9e2ac9de009ed3a2
I20250901 01:52:37.828229 2118 ts_tablet_manager.cc:1918] T 5f5396045979450193d461ba1334a5c9 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:37.830628 3022 tablet.cc:1722] T 615b2306fca84003b264da9d5f95ddab P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 8 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 9
I20250901 01:52:37.834465 1492 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: default.Table_e40051a54d4a4d6784a666e826d4fa72 [id=8a1a77329ab740dda2d6a5d78bbff906] alter complete (version 2)
I20250901 01:52:37.834569 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_63533b82563a4201a8751dc19b15b625"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.834985 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4c0bbe0e0009432f85cb7240ac684ca3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.835453 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_57aa103e49bf46ed868dfe085321b252"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.836247 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4989f88f66a94d11896840e365206618"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.837194 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b708d1477318431694ec3c8429251fbe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.838029 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e93b16520ea5486aba092fa4947bc43c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.839025 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_74de988bbb8843e59514649e5fec0fb6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.839390 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7e4c289faf874d66a998325578c8e1e0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.839993 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_caf73da079454ef2b27099a80ccadb47"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.840122 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f8a94587aad74a33ace5e5b71c0d212f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:37.845791 992 consensus_peers.cc:489] T 4b4aa063ffe948e69e47f03102581899 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Couldn't send request to peer 00c282f779c1477d920f24dc0dc16628. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4b4aa063ffe948e69e47f03102581899. This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:37.847877 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f304e090ed53485991427552db2ed862"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.847965 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f4a718a1d915496e8ea4d857ed200865"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.848867 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f504b589626d464ca02c4f0b94b16908"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.849290 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d053b3ccb9ee416ea770fca071b17757"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.849844 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a4a16ec6f48f4557bf51efa6508d5357"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.836549 3020 tablet.cc:1722] T 615b2306fca84003b264da9d5f95ddab P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 8 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 9
I20250901 01:52:37.852911 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4c8fb96c332849c1844525486b309d52"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.854125 2118 log.cc:1199] T 5f5396045979450193d461ba1334a5c9 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/5f5396045979450193d461ba1334a5c9
I20250901 01:52:37.854853 2118 ts_tablet_manager.cc:1939] T 5f5396045979450193d461ba1334a5c9 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
W20250901 01:52:37.856112 992 consensus_peers.cc:489] T df01d5df9fc2424eaf9de4e68e781803 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250901 01:52:37.858160 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2831a28fbee049ca902f44d640eae292"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.865139 3038 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250901 01:52:37.869817 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8e1a5ae18e6744fabad44fa9c2b5ddc8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.870883 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a7cc78a6531c4af59e7cec3b702efe5b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.869841 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_689490f7f1bf4917b2b526fcfd87d4f3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.871924 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_cd8a8aebf74f418198e4e63b341aa9a5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:37.872503 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 01deaae6abb141789f0cd240f3058c73, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7baa92b46d98454a891222be49a6ca48'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.872747 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_574019d98600487d9b6d5426c5b3664e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.875725 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 346) took 3184 ms. Trace:
I20250901 01:52:37.873276 1483 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_9f14a37542af45978c2a16cfd925c35c" } modify_external_catalogs: true
I20250901 01:52:37.876436 1483 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_9f14a37542af45978c2a16cfd925c35c" } modify_external_catalogs: true
I20250901 01:52:37.873407 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c709af926bbb4f6282120f74dc31e185"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.869807 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_efc1e51970e24a3798cdd1d639afd37f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.877159 2113 ts_tablet_manager.cc:1918] T 50472a7aa2f04e0fbd65ac6c86c34b3e P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:37.877274 2113 log.cc:1199] T 50472a7aa2f04e0fbd65ac6c86c34b3e P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/50472a7aa2f04e0fbd65ac6c86c34b3e
I20250901 01:52:37.869784 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c1b56b23495b4b0d8d1ff79a6f9b57c4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.875454 1485 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_b4fbc34da806475c912052d0fab3fc53" } new_table_name: "default.Table_59d45086d2784ae0a55afbf521492127" modify_external_catalogs: true
I20250901 01:52:37.877765 1490 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_666a4667dfa84c5ca860560cef32d55c" } new_table_name: "default.Table_b20bbcbcba6444e090d9626144d173ae" modify_external_catalogs: true
I20250901 01:52:37.880035 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_65a266ce64d44042b2b429fd1a98ffae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:37.877516 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9f62a9d97f514747b940b488907369f3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d53a456dbca64daf92aeb5fb4517b417'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.880381 1696 catalog_manager.cc:6205] Error selecting replicas for tablet ff2bd017417c41b5b8225b8f00283e63, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_9f14a37542af45978c2a16cfd925c35c'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.883450 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_839af921701349d3b89ba3546a8e44dd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:37.885601 1485 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 086aa948f93644da8e5835b0fac87ddd has no consensus state
I20250901 01:52:37.885614 3029 raft_consensus.cc:491] T bd9b59f0864a497faf281edd6077668d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:37.885754 3029 raft_consensus.cc:513] T bd9b59f0864a497faf281edd6077668d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:37.886348 3029 leader_election.cc:290] T bd9b59f0864a497faf281edd6077668d P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:37.886345 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_0e5aa8e37ad44367a64b696eb11f6d32"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.886574 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d2a6db530ee14f8a9fc3dad3ddb3da01"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.887197 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f657ae41403045f48e26515805ae1ba7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.887398 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c3538f2da4a3448abbc69b03b8241afd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.888171 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_dee23c3e5e5d428c944735a96c92a84b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.889215 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_474cf9561caf4cfeb77dc8161f8f1978"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.889839 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e557b761a29b4bcc83fc6992fc753198"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:37.890283 1696 catalog_manager.cc:6205] Error selecting replicas for tablet dcc60bb000f440f1aca9133928b5ce20, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_5a19f7d6321b49249c4edd7afb1d24ef'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.889839 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5f618eceae504839bf8182edb6f7ef8e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:37.890372 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 465c02a130c54e09b5a74fb4dd2ef762, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_8e4ca21f21734e049d174baa0a9335a4'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.891175 1696 catalog_manager.cc:6205] Error selecting replicas for tablet a5609f111ab34c0ea9fabdbce06c81f9, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_4505d77e4269443995012fef50c7e7b5'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.891301 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9f2416b2cc144054a8d2effc64157615, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_69575b1a9efd4573a2ee01d3f432800b'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.891404 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 75883d9fd00e429aa8853903a9fdd7e4, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_a9f6ac8609554a38ab632b2a3f7113d8'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.891505 1696 catalog_manager.cc:6205] Error selecting replicas for tablet b1ba0e19dc0b462e80c537bf2d183f5b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_f9cac02fe78f4902a8c730f2706f7139'. Need at least 3 replicas, but only 2 tablet servers are available
W20250901 01:52:37.891539 1490 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6ce9521b925b4f97bb4a810ce77e147e has no consensus state
E20250901 01:52:37.891589 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 086aa948f93644da8e5835b0fac87ddd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_59d45086d2784ae0a55afbf521492127'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.891700 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 082db24bb3fd4089950504145a20a075, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ebba044988d5435aba124b1f971356d1'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.891948 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b0b1fbfa02f64520a48e275b5863d101"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.875810 2118 rpcz_store.cc:276] 0901 01:52:34.691026 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:34.692884 (+ 1858us) service_pool.cc:225] Handling call
0901 01:52:37.728631 (+3035747us) ts_tablet_manager.cc:1145] Deleting tablet 5f5396045979450193d461ba1334a5c9
0901 01:52:37.728641 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.784682 (+ 56041us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.828120 (+ 43438us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.875701 (+ 47581us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3035557}
I20250901 01:52:37.892944 1491 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: default.Table_46b75a7344104e2f9ad89f26512d0a0a [id=cecdd8d6ca6d445c9fd82decd2a95366] alter complete (version 9)
I20250901 01:52:37.893011 2113 ts_tablet_manager.cc:1939] T 50472a7aa2f04e0fbd65ac6c86c34b3e P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:37.893342 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet 5f5396045979450193d461ba1334a5c9 (table default.Table_2b946a63bd574ac0971fa3d3b9c9a6a4 [id=9c3930e9fcfe4b49ba4bea67a2b0ad07]) successfully deleted
I20250901 01:52:37.893936 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 348) took 3152 ms. Trace:
I20250901 01:52:37.894019 2118 rpcz_store.cc:276] 0901 01:52:34.741197 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:34.741258 (+ 61us) service_pool.cc:225] Handling call
0901 01:52:37.893845 (+3152587us) ts_tablet_manager.cc:1145] Deleting tablet 764bfeb2a7d14c40815b5ec1b07bdd62
0901 01:52:37.893856 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.893919 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3152391}
I20250901 01:52:37.894322 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 481) took 4263 ms. Trace:
W20250901 01:52:37.894373 1472 catalog_manager.cc:4895] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): delete failed for tablet 764bfeb2a7d14c40815b5ec1b07bdd62 because the tablet was not found. No further retry: Not found: Tablet not found: 764bfeb2a7d14c40815b5ec1b07bdd62
I20250901 01:52:37.894429 2113 rpcz_store.cc:276] 0901 01:52:33.631291 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:33.632036 (+ 745us) service_pool.cc:225] Handling call
0901 01:52:37.801392 (+4169356us) ts_tablet_manager.cc:1145] Deleting tablet 50472a7aa2f04e0fbd65ac6c86c34b3e
0901 01:52:37.801403 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.844375 (+ 42972us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.877044 (+ 32669us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.894297 (+ 17253us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4169172}
I20250901 01:52:37.895382 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_931b7452985449a48b45e20b5928b095"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.898103 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 50472a7aa2f04e0fbd65ac6c86c34b3e (table default.Table_489b87a28dac48d093c241f301d5cccc [id=5fdd352d5e5e47a3a2bd8fd608a263dd]) successfully deleted
I20250901 01:52:37.898494 2113 tablet_replica.cc:331] T f23df6f253a943fa8e146bb8c811ef51 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:37.898732 2113 raft_consensus.cc:2241] T f23df6f253a943fa8e146bb8c811ef51 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:37.898901 2113 raft_consensus.cc:2270] T f23df6f253a943fa8e146bb8c811ef51 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:37.899449 2113 ts_tablet_manager.cc:1905] T f23df6f253a943fa8e146bb8c811ef51 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:37.899497 2887 fs_manager.cc:730] Time spent opening block manager: real 0.114s user 0.011s sys 0.001s
I20250901 01:52:37.899716 2887 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/master-1/data,/tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/master-1/wal
uuid: "d2ae2cb6af0f4b3a8f47ab1cdd6567d3"
format_stamp: "Formatted at 2025-09-01 01:52:05 on dist-test-slave-l1vd"
I20250901 01:52:37.900766 2887 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250901 01:52:37.902653 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_683f5f1099a741ff91cda3eb13194676"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:37.903374 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 93aa22afc0ad4d7fa24807ea3c5d5c21, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0f4b4094d2974b7591875375eda6bed0'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.903626 2118 tablet_replica.cc:331] T f13b972e25214f168421439fe3943e8a P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
W20250901 01:52:37.904569 992 leader_election.cc:343] T bd9b59f0864a497faf281edd6077668d P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: bd9b59f0864a497faf281edd6077668d
E20250901 01:52:37.905190 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 97418fffe6614dfba4ebd13a6ed57e67, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_360f4423e7f341339188b71cc82a1149'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.905911 2118 raft_consensus.cc:2241] T f13b972e25214f168421439fe3943e8a P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
E20250901 01:52:37.906181 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f5e3316adeae47cd848833dd84b972bb, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_fc962c8bc653498e85cc0bb41639e9ad'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.906606 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 6ce9521b925b4f97bb4a810ce77e147e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b20bbcbcba6444e090d9626144d173ae'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.906853 2118 raft_consensus.cc:2270] T f13b972e25214f168421439fe3943e8a P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:37.904743 3029 raft_consensus.cc:491] T 98c56eb62ad84f79b04f17a43d44c479 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:37.907037 3029 raft_consensus.cc:513] T 98c56eb62ad84f79b04f17a43d44c479 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
W20250901 01:52:37.907299 992 consensus_peers.cc:489] T c83ce95fd6fa41478de0e2168be201b1 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:37.907537 3029 leader_election.cc:290] T 98c56eb62ad84f79b04f17a43d44c479 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
E20250901 01:52:37.907881 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2f61edb50d7a42198bfd70639c7bf4de, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_cd56686217424ab8b95e6bb776d4177a'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.908002 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2e52ec21fa72424f92c945f4804e2059, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_9fb44a0a69994184bc2ffe62c55d2c25'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.908730 2118 ts_tablet_manager.cc:1905] T f13b972e25214f168421439fe3943e8a P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:37.910880 911 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "98c56eb62ad84f79b04f17a43d44c479" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628" is_pre_election: true
I20250901 01:52:37.911288 911 raft_consensus.cc:2466] T 98c56eb62ad84f79b04f17a43d44c479 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 0.
I20250901 01:52:37.915904 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_cbc66c2c30df452bbb5df070c692218b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:37.916996 992 leader_election.cc:336] T bd9b59f0864a497faf281edd6077668d P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:37.918016 992 leader_election.cc:304] T bd9b59f0864a497faf281edd6077668d P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:37.917855 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_81370db461244fc6befbc65e2e155304"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.919225 992 leader_election.cc:304] T 98c56eb62ad84f79b04f17a43d44c479 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:37.919982 1483 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet ff2bd017417c41b5b8225b8f00283e63
I20250901 01:52:37.919878 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_03b2fa0489eb43e2b7359ac3fab523d4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.920548 2911 raft_consensus.cc:2747] T bd9b59f0864a497faf281edd6077668d P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:37.920724 2911 raft_consensus.cc:2802] T 98c56eb62ad84f79b04f17a43d44c479 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250901 01:52:37.920759 2113 ts_tablet_manager.cc:1918] T f23df6f253a943fa8e146bb8c811ef51 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:37.920789 2911 raft_consensus.cc:491] T 98c56eb62ad84f79b04f17a43d44c479 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250901 01:52:37.920866 2911 raft_consensus.cc:3058] T 98c56eb62ad84f79b04f17a43d44c479 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:37.920770 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2f46677c01ee4c5bbdbba6b2b05d9b63"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.920866 2113 log.cc:1199] T f23df6f253a943fa8e146bb8c811ef51 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/f23df6f253a943fa8e146bb8c811ef51
I20250901 01:52:37.921520 2113 ts_tablet_manager.cc:1939] T f23df6f253a943fa8e146bb8c811ef51 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:37.922286 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 482) took 4289 ms. Trace:
I20250901 01:52:37.922803 2113 rpcz_store.cc:276] 0901 01:52:33.632468 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:33.632517 (+ 49us) service_pool.cc:225] Handling call
0901 01:52:37.898420 (+4265903us) ts_tablet_manager.cc:1145] Deleting tablet f23df6f253a943fa8e146bb8c811ef51
0901 01:52:37.898436 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.905791 (+ 7355us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.920651 (+ 14860us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.922264 (+ 1613us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4265737}
I20250901 01:52:37.923111 2113 tablet_replica.cc:331] T 3697821cd29446bb908b020599df0db9 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:37.923308 2113 raft_consensus.cc:2241] T 3697821cd29446bb908b020599df0db9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:37.923442 2113 raft_consensus.cc:2270] T 3697821cd29446bb908b020599df0db9 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:37.923506 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3032aba98fc94a61b15edb13f17b5250"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.923583 2911 raft_consensus.cc:513] T 98c56eb62ad84f79b04f17a43d44c479 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:37.923985 2113 ts_tablet_manager.cc:1905] T 3697821cd29446bb908b020599df0db9 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:37.924077 2911 leader_election.cc:290] T 98c56eb62ad84f79b04f17a43d44c479 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:37.924669 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet f23df6f253a943fa8e146bb8c811ef51 (table default.Table_59039012009944fdbe0179002ea44633 [id=218947d08fb24732b5d0375d4af973e9]) successfully deleted
I20250901 01:52:37.925040 3022 tablet.cc:1722] T a9b7f2e6548043a7865e25387cdd06c9 P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:37.926010 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_af026098c7c74e388c7b466bd5223b50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.926913 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_698c0ddc9592422f8a92b1a6b29eb27a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.928134 919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "98c56eb62ad84f79b04f17a43d44c479" candidate_uuid: "fd27ab6665304aaea53adbd102dd4f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c282f779c1477d920f24dc0dc16628"
I20250901 01:52:37.928151 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_fa063e72f2594d19af9fcf10b4543167"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.928360 919 raft_consensus.cc:3058] T 98c56eb62ad84f79b04f17a43d44c479 P 00c282f779c1477d920f24dc0dc16628 [term 0 FOLLOWER]: Advancing to term 1
I20250901 01:52:37.929097 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1582fc7f82bd478cbaaf2a7ece3227a5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.930071 1490 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_2847042ef2784ca3b387e6bbf4360167" } new_table_name: "default.Table_1af06edaa32f45cfb57831cf5fa77254" modify_external_catalogs: true
I20250901 01:52:37.931326 919 raft_consensus.cc:2466] T 98c56eb62ad84f79b04f17a43d44c479 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27ab6665304aaea53adbd102dd4f58 in term 1.
I20250901 01:52:37.932551 3020 tablet.cc:1722] T a9b7f2e6548043a7865e25387cdd06c9 P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250901 01:52:37.934301 1483 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_69575b1a9efd4573a2ee01d3f432800b" } modify_external_catalogs: true
I20250901 01:52:37.934453 1483 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_69575b1a9efd4573a2ee01d3f432800b" } modify_external_catalogs: true
I20250901 01:52:37.934567 1491 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_290a3dc98f0943fd816d3cf2966436a4" } modify_external_catalogs: true
I20250901 01:52:37.934691 1491 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_290a3dc98f0943fd816d3cf2966436a4" } modify_external_catalogs: true
I20250901 01:52:37.935114 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_25900e3d3c66405fba15cd680499ca7c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.935904 1485 master_service.cc:948] ReplaceTablet: received request to replace tablet ed1b41340ce742a88c3e4fd85cd25ca6 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:37.945694 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c306aa9d649845fcbf1569ad96143872"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.946765 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5fb25af8e5cb44d0a4ec97a9e98ed057"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.947690 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b96d26d06f11441f8fa0a6574bcb3d3e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
E20250901 01:52:37.949419 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 01deaae6abb141789f0cd240f3058c73, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7baa92b46d98454a891222be49a6ca48'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.950343 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_27c38a4a46ca42dab672bfef286d0e4b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.950649 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3d11e70b289248ab971af5cd7a01a5d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.951108 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4d6da3cf194f40e3bc50cd5fa4862c2c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.951237 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ab98e597e30940a380ea76fcddb012c4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.952265 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1b7286eb56d64fc8aec7965b9878d717"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.951152 1485 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: default.Table_8e94a246afd14f90afc30853e6ab1d1c [id=5bb33f9da2dc4a9cbf17b31cd14fdd23] alter complete (version 1)
E20250901 01:52:37.950881 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9f62a9d97f514747b940b488907369f3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d53a456dbca64daf92aeb5fb4517b417'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.953161 1696 catalog_manager.cc:6205] Error selecting replicas for tablet dcc60bb000f440f1aca9133928b5ce20, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_5a19f7d6321b49249c4edd7afb1d24ef'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.953307 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 465c02a130c54e09b5a74fb4dd2ef762, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_8e4ca21f21734e049d174baa0a9335a4'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.953430 1696 catalog_manager.cc:6205] Error selecting replicas for tablet a5609f111ab34c0ea9fabdbce06c81f9, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_4505d77e4269443995012fef50c7e7b5'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.953619 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_513ce5ed05794dc684be559249663282"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.953660 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_58e1087a901a4e5a979ee814515f515d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.956005 1485 master_service.cc:948] ReplaceTablet: received request to replace tablet 0e4009963cfe42d6ba164fc828897f09 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:37.930182 2113 ts_tablet_manager.cc:1918] T 3697821cd29446bb908b020599df0db9 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:37.957728 2113 log.cc:1199] T 3697821cd29446bb908b020599df0db9 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/3697821cd29446bb908b020599df0db9
W20250901 01:52:37.948956 992 leader_election.cc:336] T 98c56eb62ad84f79b04f17a43d44c479 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:37.958770 2113 ts_tablet_manager.cc:1939] T 3697821cd29446bb908b020599df0db9 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:37.958889 992 leader_election.cc:304] T 98c56eb62ad84f79b04f17a43d44c479 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c282f779c1477d920f24dc0dc16628, fd27ab6665304aaea53adbd102dd4f58; no voters:
I20250901 01:52:37.959219 2613 raft_consensus.cc:2802] T 98c56eb62ad84f79b04f17a43d44c479 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Leader election won for term 1
I20250901 01:52:37.959350 2613 raft_consensus.cc:695] T 98c56eb62ad84f79b04f17a43d44c479 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Becoming Leader. State: Replica: fd27ab6665304aaea53adbd102dd4f58, State: Running, Role: LEADER
I20250901 01:52:37.959609 2613 consensus_queue.cc:237] T 98c56eb62ad84f79b04f17a43d44c479 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:37.961194 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 483) took 4328 ms. Trace:
I20250901 01:52:37.961268 2113 rpcz_store.cc:276] 0901 01:52:33.632872 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:33.632918 (+ 46us) service_pool.cc:225] Handling call
0901 01:52:37.923066 (+4290148us) ts_tablet_manager.cc:1145] Deleting tablet 3697821cd29446bb908b020599df0db9
0901 01:52:37.923076 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.927057 (+ 3981us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.930100 (+ 3043us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.961175 (+ 31075us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4289989}
I20250901 01:52:37.961736 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 3697821cd29446bb908b020599df0db9 (table default.Table_efc44c841f0b4ae2a168be904751b65c [id=8106fbf3fbe244dabcc6f134e512a9d4]) successfully deleted
I20250901 01:52:37.964260 2113 tablet_replica.cc:331] T 3dae00aa38204ee3ac77325eb334be42 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
E20250901 01:52:37.953878 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9f2416b2cc144054a8d2effc64157615, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_69575b1a9efd4573a2ee01d3f432800b'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.964644 2113 raft_consensus.cc:2241] T 3dae00aa38204ee3ac77325eb334be42 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:37.965013 2113 pending_rounds.cc:70] T 3dae00aa38204ee3ac77325eb334be42 P fd27ab6665304aaea53adbd102dd4f58: Trying to abort 1 pending ops.
E20250901 01:52:37.965057 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 75883d9fd00e429aa8853903a9fdd7e4, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_a9f6ac8609554a38ab632b2a3f7113d8'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.965109 2113 pending_rounds.cc:77] T 3dae00aa38204ee3ac77325eb334be42 P fd27ab6665304aaea53adbd102dd4f58: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7195408613352837120 op_type: NO_OP noop_request { }
E20250901 01:52:37.965245 1696 catalog_manager.cc:6205] Error selecting replicas for tablet b1ba0e19dc0b462e80c537bf2d183f5b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_f9cac02fe78f4902a8c730f2706f7139'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.965297 2113 raft_consensus.cc:2887] T 3dae00aa38204ee3ac77325eb334be42 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
E20250901 01:52:37.965394 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 086aa948f93644da8e5835b0fac87ddd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_59d45086d2784ae0a55afbf521492127'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.966133 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 082db24bb3fd4089950504145a20a075, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ebba044988d5435aba124b1f971356d1'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.966285 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 93aa22afc0ad4d7fa24807ea3c5d5c21, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0f4b4094d2974b7591875375eda6bed0'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.966394 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 97418fffe6614dfba4ebd13a6ed57e67, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_360f4423e7f341339188b71cc82a1149'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.966495 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f5e3316adeae47cd848833dd84b972bb, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_fc962c8bc653498e85cc0bb41639e9ad'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.966593 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 6ce9521b925b4f97bb4a810ce77e147e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b20bbcbcba6444e090d9626144d173ae'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.966696 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2f61edb50d7a42198bfd70639c7bf4de, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_cd56686217424ab8b95e6bb776d4177a'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.966794 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2e52ec21fa72424f92c945f4804e2059, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_9fb44a0a69994184bc2ffe62c55d2c25'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.970759 2118 ts_tablet_manager.cc:1918] T f13b972e25214f168421439fe3943e8a P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 01:52:37.970934 2118 log.cc:1199] T f13b972e25214f168421439fe3943e8a P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/f13b972e25214f168421439fe3943e8a
I20250901 01:52:37.971323 2118 ts_tablet_manager.cc:1939] T f13b972e25214f168421439fe3943e8a P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:37.972102 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 349) took 3000 ms. Trace:
I20250901 01:52:37.965487 2911 raft_consensus.cc:491] T 438ae684af234c868a419fbab43b0b98 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:37.972805 1484 master_service.cc:948] ReplaceTablet: received request to replace tablet 34ebb441c27747ec860ff2e25f9a0adc from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:37.972800 2911 raft_consensus.cc:513] T 438ae684af234c868a419fbab43b0b98 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:37.972239 2118 rpcz_store.cc:276] 0901 01:52:34.972025 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:34.972077 (+ 52us) service_pool.cc:225] Handling call
0901 01:52:37.903580 (+2931503us) ts_tablet_manager.cc:1145] Deleting tablet f13b972e25214f168421439fe3943e8a
0901 01:52:37.903590 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.950095 (+ 46505us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.970648 (+ 20553us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.972082 (+ 1434us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2931320}
I20250901 01:52:37.973464 2911 leader_election.cc:290] T 438ae684af234c868a419fbab43b0b98 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:37.973618 1472 catalog_manager.cc:4928] TS 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): tablet f13b972e25214f168421439fe3943e8a (table default.Table_63ef1dcd92db46d298446117a62cfba2 [id=c1483e9d0db64d468b12344500f9cac8]) successfully deleted
W20250901 01:52:37.966001 992 leader_election.cc:336] T 98c56eb62ad84f79b04f17a43d44c479 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:37.974331 2118 tablet_replica.cc:331] T ea846f88b3534beebd0001b7f17b5ca9 P 00c282f779c1477d920f24dc0dc16628: stopping tablet replica
I20250901 01:52:37.974555 2118 raft_consensus.cc:2241] T ea846f88b3534beebd0001b7f17b5ca9 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:37.974767 2118 raft_consensus.cc:2270] T ea846f88b3534beebd0001b7f17b5ca9 P 00c282f779c1477d920f24dc0dc16628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:37.974943 3022 tablet.cc:1722] T 9127f9c5239e445ba86012951b7c75d9 P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250901 01:52:37.975325 2118 ts_tablet_manager.cc:1905] T ea846f88b3534beebd0001b7f17b5ca9 P 00c282f779c1477d920f24dc0dc16628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:37.976163 1483 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 9f2416b2cc144054a8d2effc64157615
I20250901 01:52:37.977424 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_844ef84183784d1580421b7924aadac3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.977708 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_00fd8f0ec15f405481c9fbbba8cadb11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.978269 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8edbd119507b42698afa6ffda493d0bc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:37.978710 992 consensus_peers.cc:489] T 5dd0c6b806574caf9126dad94833fcf7 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Couldn't send request to peer 0cf599f74322478d91475df234e58965. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250901 01:52:37.978968 992 leader_election.cc:343] T 438ae684af234c868a419fbab43b0b98 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 438ae684af234c868a419fbab43b0b98
I20250901 01:52:37.979816 2113 raft_consensus.cc:2270] T 3dae00aa38204ee3ac77325eb334be42 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:37.980531 2113 ts_tablet_manager.cc:1905] T 3dae00aa38204ee3ac77325eb334be42 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:37.980522 3020 tablet.cc:1722] T 9127f9c5239e445ba86012951b7c75d9 P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250901 01:52:37.984961 1484 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 34ebb441c27747ec860ff2e25f9a0adc
W20250901 01:52:37.985170 1484 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 34ebb441c27747ec860ff2e25f9a0adc: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:37.985872 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 34ebb441c27747ec860ff2e25f9a0adc with delete_type TABLET_DATA_DELETED (replaced by tablet aa34e819bf13480cb63b06c66151829d) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:37.986191 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 34ebb441c27747ec860ff2e25f9a0adc with delete_type TABLET_DATA_DELETED (replaced by tablet aa34e819bf13480cb63b06c66151829d) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:37.986447 1484 catalog_manager.cc:6652] ReplaceTablet: tablet 34ebb441c27747ec860ff2e25f9a0adc deleted and replaced by tablet aa34e819bf13480cb63b06c66151829d
W20250901 01:52:37.986738 992 leader_election.cc:336] T 438ae684af234c868a419fbab43b0b98 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:37.986907 992 leader_election.cc:304] T 438ae684af234c868a419fbab43b0b98 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
E20250901 01:52:37.987146 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 01deaae6abb141789f0cd240f3058c73, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7baa92b46d98454a891222be49a6ca48'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.987406 3034 raft_consensus.cc:2747] T 438ae684af234c868a419fbab43b0b98 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
E20250901 01:52:37.987581 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9f62a9d97f514747b940b488907369f3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d53a456dbca64daf92aeb5fb4517b417'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.987843 1696 catalog_manager.cc:6205] Error selecting replicas for tablet dcc60bb000f440f1aca9133928b5ce20, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_5a19f7d6321b49249c4edd7afb1d24ef'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.987962 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 465c02a130c54e09b5a74fb4dd2ef762, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_8e4ca21f21734e049d174baa0a9335a4'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.988068 1696 catalog_manager.cc:6205] Error selecting replicas for tablet a5609f111ab34c0ea9fabdbce06c81f9, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_4505d77e4269443995012fef50c7e7b5'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.988161 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 75883d9fd00e429aa8853903a9fdd7e4, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_a9f6ac8609554a38ab632b2a3f7113d8'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.988255 1696 catalog_manager.cc:6205] Error selecting replicas for tablet b1ba0e19dc0b462e80c537bf2d183f5b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_f9cac02fe78f4902a8c730f2706f7139'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.988350 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 086aa948f93644da8e5835b0fac87ddd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_59d45086d2784ae0a55afbf521492127'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.988482 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 082db24bb3fd4089950504145a20a075, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ebba044988d5435aba124b1f971356d1'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.988581 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 93aa22afc0ad4d7fa24807ea3c5d5c21, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0f4b4094d2974b7591875375eda6bed0'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.988682 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 97418fffe6614dfba4ebd13a6ed57e67, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_360f4423e7f341339188b71cc82a1149'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.988793 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f5e3316adeae47cd848833dd84b972bb, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_fc962c8bc653498e85cc0bb41639e9ad'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.988901 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 6ce9521b925b4f97bb4a810ce77e147e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b20bbcbcba6444e090d9626144d173ae'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.988997 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2f61edb50d7a42198bfd70639c7bf4de, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_cd56686217424ab8b95e6bb776d4177a'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:37.989092 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2e52ec21fa72424f92c945f4804e2059, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_9fb44a0a69994184bc2ffe62c55d2c25'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:37.994599 2113 ts_tablet_manager.cc:1918] T 3dae00aa38204ee3ac77325eb334be42 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:37.994771 2113 log.cc:1199] T 3dae00aa38204ee3ac77325eb334be42 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/3dae00aa38204ee3ac77325eb334be42
I20250901 01:52:37.995167 2113 ts_tablet_manager.cc:1939] T 3dae00aa38204ee3ac77325eb334be42 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
W20250901 01:52:37.996053 992 consensus_peers.cc:489] T aec951f2cc84499b9162b9fa24d5440e P fd27ab6665304aaea53adbd102dd4f58 -> Peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Couldn't send request to peer 00c282f779c1477d920f24dc0dc16628. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aec951f2cc84499b9162b9fa24d5440e. This is attempt 1: this message will repeat every 5th retry.
I20250901 01:52:37.996080 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 484) took 4362 ms. Trace:
I20250901 01:52:37.998152 2113 rpcz_store.cc:276] 0901 01:52:33.633253 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:33.633295 (+ 42us) service_pool.cc:225] Handling call
0901 01:52:37.964188 (+4330893us) ts_tablet_manager.cc:1145] Deleting tablet 3dae00aa38204ee3ac77325eb334be42
0901 01:52:37.964206 (+ 18us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.983697 (+ 19491us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.994470 (+ 10773us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:37.996058 (+ 1588us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4330756}
I20250901 01:52:37.998685 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 485) took 4365 ms. Trace:
I20250901 01:52:37.998736 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 3dae00aa38204ee3ac77325eb334be42 (table default.Table_faf7ef12324f454396e5ae64d64ebe51 [id=5812902df9ea406ba59ff0ff23f130b6]) successfully deleted
I20250901 01:52:37.999311 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_82d8fd52e6cc48d4a0e1eddc58c9c17b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:37.999579 2113 rpcz_store.cc:276] 0901 01:52:33.633630 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:33.634138 (+ 508us) service_pool.cc:225] Handling call
0901 01:52:37.998561 (+4364423us) ts_tablet_manager.cc:1145] Deleting tablet 85322a864d154f299f2d670fdd2a7bfb
0901 01:52:37.998584 (+ 23us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:37.998665 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4364227}
W20250901 01:52:38.000068 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 85322a864d154f299f2d670fdd2a7bfb because the tablet was not found. No further retry: Not found: Tablet not found: 85322a864d154f299f2d670fdd2a7bfb
I20250901 01:52:38.000169 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a81151f470b44159b259d2e30f6ee2e3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.000313 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_435ae156224041b7a832eb85092b7f1e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.000478 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 486) took 4366 ms. Trace:
I20250901 01:52:38.000560 2113 rpcz_store.cc:276] 0901 01:52:33.634487 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:33.634535 (+ 48us) service_pool.cc:225] Handling call
0901 01:52:38.000350 (+4365815us) ts_tablet_manager.cc:1145] Deleting tablet 42c1156e8dd84c6fa1396b46de13eb6f
0901 01:52:38.000362 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:38.000455 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4365663}
I20250901 01:52:38.000860 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e498b242be4f4195b21046b66aaba8eb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:38.000944 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 42c1156e8dd84c6fa1396b46de13eb6f because the tablet was not found. No further retry: Not found: Tablet not found: 42c1156e8dd84c6fa1396b46de13eb6f
I20250901 01:52:38.001719 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 487) took 4366 ms. Trace:
I20250901 01:52:38.002223 3034 raft_consensus.cc:491] T d27c8d99528a42ed8dd1c5a7f373bbae P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:38.002350 3034 raft_consensus.cc:513] T d27c8d99528a42ed8dd1c5a7f373bbae P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:38.002483 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_6fbe6a9027dc442c867f2a24bab3adf6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.003023 3034 leader_election.cc:290] T d27c8d99528a42ed8dd1c5a7f373bbae P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:38.003814 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a79acd13b0ef425297ce3a83f0d34871"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:38.004496 992 leader_election.cc:343] T d27c8d99528a42ed8dd1c5a7f373bbae P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: d27c8d99528a42ed8dd1c5a7f373bbae
I20250901 01:52:38.004946 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_68e4e16df6af4521b3458939661b9715"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.005103 1491 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 61d8f4252767490f842fb28f3edc1a96
W20250901 01:52:38.005465 1491 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 61d8f4252767490f842fb28f3edc1a96: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:38.002074 2113 rpcz_store.cc:276] 0901 01:52:33.634971 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:33.635028 (+ 57us) service_pool.cc:225] Handling call
0901 01:52:38.001636 (+4366608us) ts_tablet_manager.cc:1145] Deleting tablet 061b760d8e6647c6ab0b5d538840a230
0901 01:52:38.001645 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:38.001703 (+ 58us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4366461}
I20250901 01:52:38.006040 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 61d8f4252767490f842fb28f3edc1a96 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:37 UTC) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:38.006546 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 61d8f4252767490f842fb28f3edc1a96 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-01 01:52:37 UTC) from {username='slave'} at 127.0.0.1:34472
W20250901 01:52:38.006774 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 061b760d8e6647c6ab0b5d538840a230 because the tablet was not found. No further retry: Not found: Tablet not found: 061b760d8e6647c6ab0b5d538840a230
W20250901 01:52:38.007129 992 leader_election.cc:336] T d27c8d99528a42ed8dd1c5a7f373bbae P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:38.007270 992 leader_election.cc:304] T d27c8d99528a42ed8dd1c5a7f373bbae P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:38.007508 3034 raft_consensus.cc:2747] T d27c8d99528a42ed8dd1c5a7f373bbae P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:38.007174 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 488) took 4371 ms. Trace:
I20250901 01:52:38.007838 2113 rpcz_store.cc:276] 0901 01:52:33.635350 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:33.635391 (+ 41us) service_pool.cc:225] Handling call
0901 01:52:38.007086 (+4371695us) ts_tablet_manager.cc:1145] Deleting tablet 641a83a3e50d451189c33c7b7b4ca070
0901 01:52:38.007098 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:38.007157 (+ 59us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4371552}
I20250901 01:52:38.008723 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 489) took 4373 ms. Trace:
E20250901 01:52:38.009043 1696 catalog_manager.cc:6205] Error selecting replicas for tablet aa34e819bf13480cb63b06c66151829d, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_da6f34d486984dedbc970080cf14482b'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:38.009052 2113 rpcz_store.cc:276] 0901 01:52:33.635784 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:33.635840 (+ 56us) service_pool.cc:225] Handling call
0901 01:52:38.008644 (+4372804us) ts_tablet_manager.cc:1145] Deleting tablet ff013ea889d44cba8797834e4611b489
0901 01:52:38.008654 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:38.008708 (+ 54us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4372599}
W20250901 01:52:38.009961 992 consensus_peers.cc:489] T 81ce448b148447ddbc41875d863e0518 P fd27ab6665304aaea53adbd102dd4f58 -> Peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Couldn't send request to peer 00c282f779c1477d920f24dc0dc16628. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 81ce448b148447ddbc41875d863e0518. This is attempt 1: this message will repeat every 5th retry.
E20250901 01:52:38.009205 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 01deaae6abb141789f0cd240f3058c73, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_7baa92b46d98454a891222be49a6ca48'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:38.013393 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 9f62a9d97f514747b940b488907369f3, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_d53a456dbca64daf92aeb5fb4517b417'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:38.010767 1485 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_6df4c021a4f44a0bbcfe1c5758999915" } new_table_name: "default.Table_6203ecd4781e4b88aaf3fe2297b8832b" modify_external_catalogs: true
I20250901 01:52:38.009690 1492 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_a9f6ac8609554a38ab632b2a3f7113d8" } new_table_name: "default.Table_6ad99928ce1e41a2bf22a24227f69b67" modify_external_catalogs: true
W20250901 01:52:38.009778 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet ff013ea889d44cba8797834e4611b489 because the tablet was not found. No further retry: Not found: Tablet not found: ff013ea889d44cba8797834e4611b489
I20250901 01:52:38.014916 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_19497803d6ea48fcb3e6ee55d67b1706"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.009814 1491 master_service.cc:948] ReplaceTablet: received request to replace tablet 2dacb608897d47d28512df87dc320927 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:38.015822 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d40b814bad5e4f78826f0c3dddac7d19"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.024974 2911 consensus_queue.cc:1035] T 5dd0c6b806574caf9126dad94833fcf7 P fd27ab6665304aaea53adbd102dd4f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250901 01:52:38.030114 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_58eae87a355545be82eb0eb7f188fd3f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.030290 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_40bd83e4ce294df2892e81bb0e9c702e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:38.030238 1492 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 75883d9fd00e429aa8853903a9fdd7e4 has no consensus state
I20250901 01:52:38.030848 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5e58e184e41f42c69a3a831e6a4e73a4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.030114 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3b90b447e99a4a5abf8e0856ef3a8b38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.031603 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_635e0b86ec3b41ebb9ca5fe0a2e40298"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.031661 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7cc913fd27994a0e8c30b13127763b6e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.032461 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d42b085daae5446da7cce44a36b21bd6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.032780 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_6578952fcd0742ddbf97ea2d099aa6dd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.033030 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3609937d623c489baf00821c6a369f9c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.033397 1488 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_5a19f7d6321b49249c4edd7afb1d24ef" } modify_external_catalogs: true
I20250901 01:52:38.033533 1488 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_5a19f7d6321b49249c4edd7afb1d24ef" } modify_external_catalogs: true
I20250901 01:52:38.032438 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_34e8f5b4b0904fe8bca0375eee25ca2a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.034461 2113 tablet_replica.cc:331] T 435b1cca5a7d4d49959a310cb250b2a0 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:38.034721 2113 raft_consensus.cc:2241] T 435b1cca5a7d4d49959a310cb250b2a0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:38.035499 2113 raft_consensus.cc:2270] T 435b1cca5a7d4d49959a310cb250b2a0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:38.042541 1492 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: default.Table_1af06edaa32f45cfb57831cf5fa77254 [id=ab670bb02c6e4ed491e7831ad73cdfc6] alter complete (version 3)
I20250901 01:52:38.048481 1492 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_fc962c8bc653498e85cc0bb41639e9ad" } modify_external_catalogs: true
I20250901 01:52:38.048667 1492 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_fc962c8bc653498e85cc0bb41639e9ad" } modify_external_catalogs: true
I20250901 01:52:38.048856 1491 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 2dacb608897d47d28512df87dc320927
W20250901 01:52:38.049443 1491 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 2dacb608897d47d28512df87dc320927: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:38.049712 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4184b0dfcccc4fb1bff7d733fe6bcbe5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.050220 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 2dacb608897d47d28512df87dc320927 with delete_type TABLET_DATA_DELETED (replaced by tablet 08e0b22e9d9e4e4196ef008afac17964) from {username='slave'} at 127.0.0.1:34472
I20250901 01:52:38.050336 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 2dacb608897d47d28512df87dc320927 with delete_type TABLET_DATA_DELETED (replaced by tablet 08e0b22e9d9e4e4196ef008afac17964) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:38.050853 1491 catalog_manager.cc:6652] ReplaceTablet: tablet 2dacb608897d47d28512df87dc320927 deleted and replaced by tablet 08e0b22e9d9e4e4196ef008afac17964
I20250901 01:52:38.051059 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ff30df2f58764d008727e6e218ea4ef9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.051512 2113 ts_tablet_manager.cc:1905] T 435b1cca5a7d4d49959a310cb250b2a0 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:38.052098 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b3aa2b1e99034755ab2062b784dfe1eb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.053251 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_40efe900057c459bb3415d878248b5ef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.053594 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f13c1df33026449883907b5761b861b9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.053917 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9e83aa953dc042aaaa2615da634b73b1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.054456 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_234d8b16e8a446eab5b33dc4153a17cc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.055133 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d9e382d6ca854af5a2c9a0cc6d8ed684"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.055545 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b7ac09076e1346898aae0badbdbf3094"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.056660 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_633542283c4f4848925ee5a2a24301ac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.058102 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_30fe0e2993b44d5a9871d0a3a56d05f2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.058353 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_187fe37c8f004ade89beadc519710723"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.058164 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f5fb60be99cf42a5832e24a3b62a9358"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.059209 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5b049e4d34de4036844a44a58e824e06"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.060384 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4adf28832ece47b780f410616447250f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.060740 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_30e60808f1e548a8a52bda8336de892f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.060971 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_dbaa5cbbbd3545c5a57f12e2e864e5e7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.061403 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_20aa1e11b9e0462990ac85e13aa82e0e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.061540 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_708ef36b1ba8432cb2aa267c0a4fb0b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.062716 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_cce65742629d48ab985fdabc4f6b3dcd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.063141 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_25be173d5d23496c8e42be0f95c75e0f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.063628 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_fc95f7b44ee94078a634fa667dcb274f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.063863 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ac88abce11f84b34a38de0e0c9702cc9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.065707 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_dbe83db2865446c986d7b6f5bfa21c48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.066418 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a59f7be2519c4ba09b0fa5c4ea192aa3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.067129 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3d20590878d4423e8186c1e91a5dcd11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.067711 2113 ts_tablet_manager.cc:1918] T 435b1cca5a7d4d49959a310cb250b2a0 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250901 01:52:38.067864 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_14a7fb92a2014107b09aa33f86a5d65d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.068022 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_eb3f895dfa5c440b9455eb734bfdc893"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.068673 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3dfc66191df54f239b38f9c4368cb36b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.069525 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_55be904f16f84ed597bae177f0189527"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.070250 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ba2fc1eee23447f0a1d89b9ecb2b3b96"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.070923 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_12b726d341944250bb25831b01eff47e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.071285 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d578d3ca44e64508a01e071cfbefb47b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.071744 2113 log.cc:1199] T 435b1cca5a7d4d49959a310cb250b2a0 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/435b1cca5a7d4d49959a310cb250b2a0
I20250901 01:52:38.072165 2113 ts_tablet_manager.cc:1939] T 435b1cca5a7d4d49959a310cb250b2a0 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:38.071070 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b1cd694d30c449a78d1b0ed36d8c4677"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.073200 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 490) took 4437 ms. Trace:
I20250901 01:52:38.073735 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b152baade491406399e6484b7a5b5687"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.074568 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e8875d8d9cf94fd1adf22673a03d7f79"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.075001 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ef30d6f9ed5a4123ae1a63e2c63ccce1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.073287 2113 rpcz_store.cc:276] 0901 01:52:33.636226 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:33.636278 (+ 52us) service_pool.cc:225] Handling call
0901 01:52:38.009892 (+4373614us) ts_tablet_manager.cc:1145] Deleting tablet 435b1cca5a7d4d49959a310cb250b2a0
0901 01:52:38.034405 (+ 24513us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:38.056915 (+ 22510us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:38.067571 (+ 10656us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:38.073178 (+ 5607us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4373423}
I20250901 01:52:38.075515 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d6226f829f5a4fe3a738d075294fa77f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.076148 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 435b1cca5a7d4d49959a310cb250b2a0 (table default.Table_b989608c88204be2b6f706a9426c47bb [id=126b298fef6c4653952d672adff1db7e]) successfully deleted
I20250901 01:52:38.076232 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_540f39c8c5ed4aff87b03cf9882854fb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.076861 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b8e457707d4b4f7e9a7cab93216da1e1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.077481 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 491) took 4440 ms. Trace:
I20250901 01:52:38.077531 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a9c3279ce9be408cb22cd61c12d6ef2f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.077661 2113 rpcz_store.cc:276] 0901 01:52:33.636651 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:33.636695 (+ 44us) service_pool.cc:225] Handling call
0901 01:52:38.076994 (+4440299us) ts_tablet_manager.cc:1145] Deleting tablet d10c193452954c198bba686169b8025b
0901 01:52:38.077007 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:38.077120 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4440120}
I20250901 01:52:38.078466 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 494) took 4437 ms. Trace:
I20250901 01:52:38.078869 2113 rpcz_store.cc:276] 0901 01:52:33.641227 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:33.650691 (+ 9464us) service_pool.cc:225] Handling call
0901 01:52:38.078340 (+4427649us) ts_tablet_manager.cc:1145] Deleting tablet 56f2c4d2e6ab4bb4bdd6d025843589b6
0901 01:52:38.078350 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:38.078445 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4427414}
W20250901 01:52:38.079294 1473 catalog_manager.cc:4895] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): delete failed for tablet 56f2c4d2e6ab4bb4bdd6d025843589b6 because the tablet was not found. No further retry: Not found: Tablet not found: 56f2c4d2e6ab4bb4bdd6d025843589b6
I20250901 01:52:38.079640 2113 tablet_replica.cc:331] T 8ca5cf4e99014c0d84a3acad4ef713af P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:38.079902 2113 raft_consensus.cc:2241] T 8ca5cf4e99014c0d84a3acad4ef713af P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250901 01:52:38.080899 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_63f38254df504639a80960f71b743aab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.080945 2113 raft_consensus.cc:2270] T 8ca5cf4e99014c0d84a3acad4ef713af P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:38.081673 2113 ts_tablet_manager.cc:1905] T 8ca5cf4e99014c0d84a3acad4ef713af P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:38.081311 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_48e58a844df24316a04f0fc057e94ab1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.081547 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_bd3b6f8cfc7b47fca0cdac859edbdf24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.085448 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d5f22ee3cd4d4607b857768e46d88f4a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.087181 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f01fa6e476aa4a4c80f5c5f4357c0d7b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.087864 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5fbbc7b293e846b8bf318a0ec06d63be"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.088610 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7215611412674ca58039289eaa430c12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.090103 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_6440734f47594c6abc1075da8376d5ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.090340 2113 ts_tablet_manager.cc:1918] T 8ca5cf4e99014c0d84a3acad4ef713af P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 01:52:38.090552 2113 log.cc:1199] T 8ca5cf4e99014c0d84a3acad4ef713af P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/8ca5cf4e99014c0d84a3acad4ef713af
I20250901 01:52:38.090783 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8fcb31b9367641b0bd0ec766d1331e49"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.090837 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a2a83e2d36e14b968b1f78317b5f4b9d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.091167 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_faf3b53124a6473c83c4619205685377"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.092039 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a6527d80e49040cea2b6ac655a2a478f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.092759 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_48775e67617942dab4a500166ac9d1b1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.093256 2113 ts_tablet_manager.cc:1939] T 8ca5cf4e99014c0d84a3acad4ef713af P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:38.093477 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_730a47fcc2554b79b5758f05cddbebd8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.094039 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1ce718553f384022a5c9dea4dd1fad95"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.094677 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_cca5a32670344664baa7f6c4b8dfb629"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.095091 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 496) took 4351 ms. Trace:
I20250901 01:52:38.095212 2113 rpcz_store.cc:276] 0901 01:52:33.743887 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:33.743942 (+ 55us) service_pool.cc:225] Handling call
0901 01:52:38.079577 (+4335635us) ts_tablet_manager.cc:1145] Deleting tablet 8ca5cf4e99014c0d84a3acad4ef713af
0901 01:52:38.079590 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:38.085334 (+ 5744us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:38.090239 (+ 4905us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:38.095069 (+ 4830us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4335438}
I20250901 01:52:38.095645 1486 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_e549eeded6454e85b5bb2a0a66f96a61" } new_table_name: "default.Table_ff79b86b490344ba8063c73fd6d3eeae" modify_external_catalogs: true
I20250901 01:52:38.095700 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 8ca5cf4e99014c0d84a3acad4ef713af (table default.Table_8e4ca21f21734e049d174baa0a9335a4 [id=303e76c918bc4c39aa47a4d24849a358]) successfully deleted
I20250901 01:52:38.096148 2113 tablet_replica.cc:331] T ad45c90ba0da43c9860745e534664a98 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:38.096407 2113 raft_consensus.cc:2241] T ad45c90ba0da43c9860745e534664a98 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:38.098565 2113 raft_consensus.cc:2270] T ad45c90ba0da43c9860745e534664a98 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:38.098731 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_0601ad8e6e5b496c876003b5ca600d87"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.099345 2113 ts_tablet_manager.cc:1905] T ad45c90ba0da43c9860745e534664a98 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
E20250901 01:52:38.099434 1696 catalog_manager.cc:6205] Error selecting replicas for tablet dcc60bb000f440f1aca9133928b5ce20, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_5a19f7d6321b49249c4edd7afb1d24ef'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:38.100816 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 465c02a130c54e09b5a74fb4dd2ef762, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_8e4ca21f21734e049d174baa0a9335a4'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:38.101310 1696 catalog_manager.cc:6205] Error selecting replicas for tablet a5609f111ab34c0ea9fabdbce06c81f9, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_4505d77e4269443995012fef50c7e7b5'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:38.101994 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 75883d9fd00e429aa8853903a9fdd7e4, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_6ad99928ce1e41a2bf22a24227f69b67'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:38.102385 1696 catalog_manager.cc:6205] Error selecting replicas for tablet b1ba0e19dc0b462e80c537bf2d183f5b, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_f9cac02fe78f4902a8c730f2706f7139'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:38.103416 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 086aa948f93644da8e5835b0fac87ddd, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_59d45086d2784ae0a55afbf521492127'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:38.103730 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 082db24bb3fd4089950504145a20a075, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_ebba044988d5435aba124b1f971356d1'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:38.104028 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 93aa22afc0ad4d7fa24807ea3c5d5c21, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_0f4b4094d2974b7591875375eda6bed0'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:38.104660 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 97418fffe6614dfba4ebd13a6ed57e67, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_360f4423e7f341339188b71cc82a1149'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:38.104974 1696 catalog_manager.cc:6205] Error selecting replicas for tablet f5e3316adeae47cd848833dd84b972bb, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_fc962c8bc653498e85cc0bb41639e9ad'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:38.105268 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 6ce9521b925b4f97bb4a810ce77e147e, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_b20bbcbcba6444e090d9626144d173ae'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:38.105540 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2f61edb50d7a42198bfd70639c7bf4de, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_cd56686217424ab8b95e6bb776d4177a'. Need at least 3 replicas, but only 2 tablet servers are available
E20250901 01:52:38.106295 1696 catalog_manager.cc:6205] Error selecting replicas for tablet 2e52ec21fa72424f92c945f4804e2059, reason: Invalid argument: Not enough tablet servers are online for table 'default.Table_9fb44a0a69994184bc2ffe62c55d2c25'. Need at least 3 replicas, but only 2 tablet servers are available
I20250901 01:52:38.106509 2113 ts_tablet_manager.cc:1918] T ad45c90ba0da43c9860745e534664a98 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250901 01:52:38.106700 2113 log.cc:1199] T ad45c90ba0da43c9860745e534664a98 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/ad45c90ba0da43c9860745e534664a98
I20250901 01:52:38.101706 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_245fc8b67ec94b8f9437c4562704153d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.107085 2113 ts_tablet_manager.cc:1939] T ad45c90ba0da43c9860745e534664a98 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:38.107849 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 498) took 4320 ms. Trace:
I20250901 01:52:38.107976 2113 rpcz_store.cc:276] 0901 01:52:33.787262 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:33.787322 (+ 60us) service_pool.cc:225] Handling call
0901 01:52:38.096077 (+4308755us) ts_tablet_manager.cc:1145] Deleting tablet ad45c90ba0da43c9860745e534664a98
0901 01:52:38.096089 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:38.103298 (+ 7209us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:38.106396 (+ 3098us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:38.107831 (+ 1435us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4308550}
W20250901 01:52:38.108726 1486 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 26e3ad68393547e5b6e3dd6b9ec1dbbe consensus state has no leader
I20250901 01:52:38.108875 2113 tablet_replica.cc:331] T d0b7c84ec9394a20ad77add3808e555a P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:38.109014 2118 ts_tablet_manager.cc:1918] T ea846f88b3534beebd0001b7f17b5ca9 P 00c282f779c1477d920f24dc0dc16628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 01:52:38.109272 2113 raft_consensus.cc:2241] T d0b7c84ec9394a20ad77add3808e555a P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:38.109680 2113 raft_consensus.cc:2270] T d0b7c84ec9394a20ad77add3808e555a P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250901 01:52:38.110333 2113 ts_tablet_manager.cc:1905] T d0b7c84ec9394a20ad77add3808e555a P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:38.111294 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_559d4aa15fce4bd4910c7f67410c2083"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.111443 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b01c237f164c479a8af3249b5751e19e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.111460 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b667f27ced664a5e9386d5a919dbc200"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.111696 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7ea6d57378ce4cc1bc4a05cdbc569b5d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.112206 1490 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_ebba044988d5435aba124b1f971356d1" } new_table_name: "default.Table_1c1c165a0ac94f9289028b3c19d26705" modify_external_catalogs: true
I20250901 01:52:38.113402 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3c26728dacc0495b8b5a35d614450a22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.113430 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1b9bbe6425104148a4f2b223e1d14f80"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.113955 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_60b1a9c470464cb6ade946509d05aba3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.115741 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_1eb1b96d07f241d6b99c8267aab7608b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.116608 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_80a88158e7e849518053c74612488a11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.115715 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_23350242289645af822a62d04a6e8a29"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.117661 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f1d896d6516643edb5136422f72069f7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.119007 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_6fc16e59e33842e88b4649eb696969dd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.119205 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_3cb2fbf6a23e481fa08ed02d7ac8d5e6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.119787 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d68ec9288adc4a52b06d611d70e9887f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.120033 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_52f5e60b09384e77a64672d591280d5b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.120532 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_bd6ad1f5fba044e28b15e49fe583f53f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.111054 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_bbeca05795eb41cd8ec694e37d05a892"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.124972 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8d60ed8f685c48afb9f8c1f464137d19"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.125079 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b168b8a03bb748f1a4e97c0602d2538d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.125630 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_999d66647ff94256ae1658425b28ea81"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.125734 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_abc8e321f060415995752cbb2058ec6f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.126377 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ac334ac59ea14b769559ec129b4730eb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.127084 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_00b3ea4092b943ec9c7867e1bb8a57c0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.127910 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8139fea616c24faea8ab6f1bcaa9578a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.128849 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_28036320f00f4684ae1e488a2cd9f956"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.129710 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c91e11a69ed649dda4ea92b993299946"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.129927 2887 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
W20250901 01:52:38.130833 1490 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 082db24bb3fd4089950504145a20a075 has no consensus state
I20250901 01:52:38.131083 2887 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250901 01:52:38.131366 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_8a402c5f35894667ae9560c5bcada69b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.131454 2887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250901 01:52:38.133212 2113 ts_tablet_manager.cc:1918] T d0b7c84ec9394a20ad77add3808e555a P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250901 01:52:38.133392 2113 log.cc:1199] T d0b7c84ec9394a20ad77add3808e555a P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/d0b7c84ec9394a20ad77add3808e555a
I20250901 01:52:38.134016 2113 ts_tablet_manager.cc:1939] T d0b7c84ec9394a20ad77add3808e555a P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:38.136358 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b907906a15f74a65ad9e5cd8c37142eb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.137203 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d64b56311fbc4dfd9bc985cae7869728"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.138283 1490 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_360f4423e7f341339188b71cc82a1149" } modify_external_catalogs: true
I20250901 01:52:38.138417 1490 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_360f4423e7f341339188b71cc82a1149" } modify_external_catalogs: true
I20250901 01:52:38.139719 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c6f84b0b4a244596830cad1472c7d64c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.139799 1488 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet dcc60bb000f440f1aca9133928b5ce20
I20250901 01:52:38.143493 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2bc25881f37c42f7981053b2627d3d01"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.143682 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5be73d3c97e544f0a803adfbdac8b8d4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.144002 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4301d2dd174f4d8c95f25db4fbcd8271"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.145979 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4bf61dd3a1484e21a36d9338d670b67d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.146389 1490 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 97418fffe6614dfba4ebd13a6ed57e67
I20250901 01:52:38.146760 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_eb72058b11c2432ca51da8fd71d6273d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.146821 1490 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_e40051a54d4a4d6784a666e826d4fa72" } new_table_name: "default.Table_b20934a4b1064b60818c10980d2bf0a0" modify_external_catalogs: true
I20250901 01:52:38.147002 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e78d8e5b4ff84e9b97cec1621115520c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.149049 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c0f3723d58c246d58aeee17f867ab582"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.149312 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c40ccec82ebe4c0bb13138b9c1fccee7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.149835 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9dda8e2ab21e43d6aa8a51a1217288f0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.149936 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_6fed2734229b489bb7590bdeb20b4bc3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.150578 2613 raft_consensus.cc:491] T 7304937048ca48a7bd153f3cdcfaeff0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:38.150785 2613 raft_consensus.cc:513] T 7304937048ca48a7bd153f3cdcfaeff0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } }
I20250901 01:52:38.151618 2613 leader_election.cc:290] T 7304937048ca48a7bd153f3cdcfaeff0 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:38.152848 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_058bf5b00ce14c34bc4c5fbb5c798f76"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250901 01:52:38.153141 992 leader_election.cc:343] T 7304937048ca48a7bd153f3cdcfaeff0 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 7304937048ca48a7bd153f3cdcfaeff0
W20250901 01:52:38.153824 992 leader_election.cc:336] T 7304937048ca48a7bd153f3cdcfaeff0 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:38.153949 992 leader_election.cc:304] T 7304937048ca48a7bd153f3cdcfaeff0 P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:38.154160 2613 raft_consensus.cc:2747] T 7304937048ca48a7bd153f3cdcfaeff0 P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:38.154808 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7b0c33737a0d43f6b93020e2d1402ace"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.155196 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_9478b1e312ef4077a0559b75be68568c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.156263 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_72a32a25a826478da1d259dad564c533"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.158705 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7b7f2515f2b6435fb067af07b678244c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.158890 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5b79487653d642b1a33249333b27be4a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.159467 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f876cf5fec9d4ae7b449e45a0ce71616"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.159572 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_bf3f316858ab4bc386318cfc6d47223d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.160615 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b3935481d44c4f89a1a22b8944b09545"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.160893 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_6126d1b512144fc8872d8736d09fa844"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.162000 3022 tablet.cc:1722] T 5afae316ff164413be0d25d4bc0f58dc P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250901 01:52:38.162452 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_767cad8f783a49719419b0742511ea6b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.162743 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_362a683e659a4c789392bceac187b83e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.163081 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_801ccec51cff43c9b3d92a1a9ae55fee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.163637 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_342d59be10f24e8d8aef9acdff5daaed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.165510 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_da9f0b7595364fb5b9eda16640a60142"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.166105 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c26c14a102364026892c33289667f163"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.166590 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_914261d9e9bf4584be29742e919aa1a9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.166911 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_6f407006ba8f480f858426c6c5fa8884"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.167601 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_ca7b5ebcb50844cf958c02685bd550a5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.168118 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_924eade9f75145afabb378444ea85096"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.168737 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_fb16158c5dc74a27bdbb5a4e253a4bf0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.169178 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_7d8113ee2079442eb3c60691f2f6c8fa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.170080 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_32fc138166404d9cbd60a33c86e14800"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.170128 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_4550164816fa403183ffe8bf8a90ce6d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.171087 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_6d2ab03018f046c88c900d5f3d8902d6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.171316 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_a137e2e5ad0348a8b17159d6b1fa254d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.172513 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_e6af43a3850f4c1da4723460bdaa6088"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.173007 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5c69529f61d74768b57cc7e93bc4a103"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.173094 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_aa1f0a1bc3bb4e5cb8ed34cda8b01bf8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.173597 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_6eaa2aff85c54b2a8801f2a32ccbcb32"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.174419 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_0076340ea52d4eca85a5a46efc426d6a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.174953 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_49d12e39f4ee4b3899af617d774c8d35"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.173718 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_2503a8db06694e7a9a2bd86e06a9439f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.173607 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d1cede15d70842dfb3c2f0cc343b73ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.176584 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_0fd72fd0f60b4fa5a3cc0da8bdeb4584"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.179539 1696 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet dcc60bb000f440f1aca9133928b5ce20
I20250901 01:52:38.181083 1696 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 97418fffe6614dfba4ebd13a6ed57e67
I20250901 01:52:38.164363 3020 tablet.cc:1722] T 5afae316ff164413be0d25d4bc0f58dc P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20250901 01:52:38.185771 858 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111) [suppressed 102 similar messages]
I20250901 01:52:38.194269 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 500) took 4345 ms. Trace:
I20250901 01:52:38.194881 2113 rpcz_store.cc:276] 0901 01:52:33.848511 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:33.848577 (+ 66us) service_pool.cc:225] Handling call
0901 01:52:38.108749 (+4260172us) ts_tablet_manager.cc:1145] Deleting tablet d0b7c84ec9394a20ad77add3808e555a
0901 01:52:38.108779 (+ 30us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:38.129683 (+ 20904us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:38.133104 (+ 3421us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:38.194247 (+ 61143us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4260011}
I20250901 01:52:38.196893 2613 raft_consensus.cc:491] T 2df2240631844defa433270a3829d65b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250901 01:52:38.197024 2613 raft_consensus.cc:513] T 2df2240631844defa433270a3829d65b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf599f74322478d91475df234e58965" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40731 } } peers { permanent_uuid: "fd27ab6665304aaea53adbd102dd4f58" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39447 } } peers { permanent_uuid: "00c282f779c1477d920f24dc0dc16628" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40561 } }
I20250901 01:52:38.197713 2613 leader_election.cc:290] T 2df2240631844defa433270a3829d65b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf599f74322478d91475df234e58965 (127.0.104.1:40731), 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561)
I20250901 01:52:38.197001 2113 tablet_replica.cc:331] T 482a9a2cb81f448bb2823892e35920d4 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:38.199523 2113 raft_consensus.cc:2241] T 482a9a2cb81f448bb2823892e35920d4 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:38.200129 2113 raft_consensus.cc:2270] T 482a9a2cb81f448bb2823892e35920d4 P fd27ab6665304aaea53adbd102dd4f58 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250901 01:52:38.202280 992 leader_election.cc:343] T 2df2240631844defa433270a3829d65b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00c282f779c1477d920f24dc0dc16628 (127.0.104.2:40561): Not found: Tablet not found: 2df2240631844defa433270a3829d65b
W20250901 01:52:38.203490 992 leader_election.cc:336] T 2df2240631844defa433270a3829d65b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0cf599f74322478d91475df234e58965 (127.0.104.1:40731): Network error: Client connection negotiation failed: client connection to 127.0.104.1:40731: connect: Connection refused (error 111)
I20250901 01:52:38.203598 992 leader_election.cc:304] T 2df2240631844defa433270a3829d65b P fd27ab6665304aaea53adbd102dd4f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd27ab6665304aaea53adbd102dd4f58; no voters: 00c282f779c1477d920f24dc0dc16628, 0cf599f74322478d91475df234e58965
I20250901 01:52:38.203859 2613 raft_consensus.cc:2747] T 2df2240631844defa433270a3829d65b P fd27ab6665304aaea53adbd102dd4f58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250901 01:52:38.204769 2118 log.cc:1199] T ea846f88b3534beebd0001b7f17b5ca9 P 00c282f779c1477d920f24dc0dc16628: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-1/wal/wals/ea846f88b3534beebd0001b7f17b5ca9
I20250901 01:52:38.204824 2113 ts_tablet_manager.cc:1905] T 482a9a2cb81f448bb2823892e35920d4 P fd27ab6665304aaea53adbd102dd4f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250901 01:52:38.204983 1696 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet f5e3316adeae47cd848833dd84b972bb
I20250901 01:52:38.205508 2118 ts_tablet_manager.cc:1939] T ea846f88b3534beebd0001b7f17b5ca9 P 00c282f779c1477d920f24dc0dc16628: Deleting consensus metadata
I20250901 01:52:38.180168 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d2d61ac59ff04cd880a53758b2b3172f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.182943 1492 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet f5e3316adeae47cd848833dd84b972bb
I20250901 01:52:38.208079 1492 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_64abca852f94488597386653bbb2ad52"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.208819 1492 master_service.cc:948] ReplaceTablet: received request to replace tablet 3c1fd0820d8a4995abe027f685da7f7b from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:38.213313 2113 ts_tablet_manager.cc:1918] T 482a9a2cb81f448bb2823892e35920d4 P fd27ab6665304aaea53adbd102dd4f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250901 01:52:38.213413 1482 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: default.Table_b20934a4b1064b60818c10980d2bf0a0 [id=8a1a77329ab740dda2d6a5d78bbff906] alter complete (version 3)
I20250901 01:52:38.213565 2113 log.cc:1199] T 482a9a2cb81f448bb2823892e35920d4 P fd27ab6665304aaea53adbd102dd4f58: Deleting WAL directory at /tmp/dist-test-taskddMNdm/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1756691524293796-416-0/minicluster-data/ts-2/wal/wals/482a9a2cb81f448bb2823892e35920d4
I20250901 01:52:38.213677 1482 master_service.cc:948] ReplaceTablet: received request to replace tablet 81c36693199945d98465579b509852e2 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:38.214071 2113 ts_tablet_manager.cc:1939] T 482a9a2cb81f448bb2823892e35920d4 P fd27ab6665304aaea53adbd102dd4f58: Deleting consensus metadata
I20250901 01:52:38.214187 1482 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_9fb44a0a69994184bc2ffe62c55d2c25" } modify_external_catalogs: true
I20250901 01:52:38.214347 1482 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_9fb44a0a69994184bc2ffe62c55d2c25" } modify_external_catalogs: true
I20250901 01:52:38.215062 2113 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35946 (request call id 502) took 4065 ms. Trace:
I20250901 01:52:38.215196 2113 rpcz_store.cc:276] 0901 01:52:34.149818 (+ 0us) service_pool.cc:168] Inserting onto call queue
0901 01:52:34.150214 (+ 396us) service_pool.cc:225] Handling call
0901 01:52:38.196938 (+4046724us) ts_tablet_manager.cc:1145] Deleting tablet 482a9a2cb81f448bb2823892e35920d4
0901 01:52:38.196954 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0901 01:52:38.209754 (+ 12800us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:38.213228 (+ 3474us) tablet_metadata.cc:641] Metadata flushed
0901 01:52:38.215042 (+ 1814us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4046508}
I20250901 01:52:38.218947 1492 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 7ce2288f784044eea010664e3def133f: Sending DeleteTablet for 3 replicas of tablet 3c1fd0820d8a4995abe027f685da7f7b
W20250901 01:52:38.219184 1492 catalog_manager.cc:5878] Failed to send DeleteReplica request for tablet 3c1fd0820d8a4995abe027f685da7f7b: Not found: failed to reset TS proxy: Could not find TS for UUID 0cf599f74322478d91475df234e58965
I20250901 01:52:38.219897 1492 catalog_manager.cc:6652] ReplaceTablet: tablet 3c1fd0820d8a4995abe027f685da7f7b deleted and replaced by tablet 12a2da087c7c4db48e9f58f171939957
I20250901 01:52:38.206314 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_eb9fccda7f1e4b2b8e3ffe66ae9a0578"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.225127 1492 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_b20bbcbcba6444e090d9626144d173ae" } modify_external_catalogs: true
I20250901 01:52:38.225339 1492 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_b20bbcbcba6444e090d9626144d173ae" } modify_external_catalogs: true
I20250901 01:52:38.225749 3022 tablet.cc:1722] T 14b466b2a5a249aa841f1b8b06301edc P fd27ab6665304aaea53adbd102dd4f58: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6
I20250901 01:52:38.227017 1484 master_service.cc:948] ReplaceTablet: received request to replace tablet 45f6e8ba99b74898a2922b36768c6c36 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:38.227432 1484 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_b05011d16cf648d88b5c45fd8c94dfea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.230453 1482 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 2e52ec21fa72424f92c945f4804e2059
I20250901 01:52:38.230666 1482 master_service.cc:948] ReplaceTablet: received request to replace tablet 6789505328b1409ea288e8b4eaf94935 from {username='slave'} at 127.0.0.1:33478
I20250901 01:52:38.231339 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_650666f117054fc39a9a99ed9dfeef80"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.216022 2113 tablet_replica.cc:331] T 5adf6b492640417ca071c94702b48cb9 P fd27ab6665304aaea53adbd102dd4f58: stopping tablet replica
I20250901 01:52:38.233260 1696 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 2e52ec21fa72424f92c945f4804e2059
I20250901 01:52:38.233712 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_da4c177f8a6e484cb7502f78ea5258a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.203718 1485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_65062eeb3d45416c81b11b7d76ebd437"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.235356 2113 raft_consensus.cc:2241] T 5adf6b492640417ca071c94702b48cb9 P fd27ab6665304aaea53adbd102dd4f58 [term 1 LEADER]: Raft consensus shutting down.
I20250901 01:52:38.236132 1491 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_f56922b0cbf4481a9a7e7ad68dcb9c9b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.224740 3020 tablet.cc:1722] T 14b466b2a5a249aa841f1b8b06301edc P 00c282f779c1477d920f24dc0dc16628: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6
I20250901 01:52:38.242274 2118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34472 (request call id 350) took 3260 ms. Trace:
I20250901 01:52:38.186621 1490 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d440a3e760e046819ae6e78fc19fed2a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.191572 1488 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d530c8d8eae5412db6038d36d3ca2bc9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.251911 1488 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33478:
table { table_name: "default.Table_59d45086d2784ae0a55afbf521492127" } new_table_name: "default.Table_c4e690f221614919b673c82775c446fc" modify_external_catalogs: true
I20250901 01:52:38.191929 1489 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_c18b994bbecb42249cb2c55d8ac17c50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.196605 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet d0b7c84ec9394a20ad77add3808e555a (table default.Table_85b9e2f3f8f7400aa833201daae93dc3 [id=cac94430e07342dfa9b266df6472b6e3]) successfully deleted
W20250901 01:52:38.252482 1473 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.3:39447, user_credentials={real_user=slave}} blocked reactor thread for 55884.8us
I20250901 01:52:38.253021 1473 catalog_manager.cc:4928] TS fd27ab6665304aaea53adbd102dd4f58 (127.0.104.3:39447): tablet 482a9a2cb81f448bb2823892e35920d4 (table default.Table_7baa92b46d98454a891222be49a6ca48 [id=b94e5ce799ee4046982550aace3a08d6]) successfully deleted
I20250901 01:52:38.253628 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 3c1fd0820d8a4995abe027f685da7f7b with delete_type TABLET_DATA_DELETED (replaced by tablet 12a2da087c7c4db48e9f58f171939957) from {username='slave'} at 127.0.0.1:35946
I20250901 01:52:38.180914 1483 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_5103a1564f604aa48b8d75bff2e30c98"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.239710 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_12c095f6e25e4bcc8ebb7695c60a9873"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.185614 1486 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_75850fb756fc4f2e97fec5ae00665a4d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.255798 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_bb15b6dcfa5440e1b529d2c314f3d82b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250901 01:52:38.260389 1482 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33478:
name: "default.Table_d2b7da8a7f8c4aa19afaa2fb8e6a17a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"