[==========] Running 4 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 4 tests from WebserverCrawlITest
[ RUN ] WebserverCrawlITest.TestAllWebPages/0
2026-08-12T17:58:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T17:58:43Z Disabled control of system clock
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260812 17:58:43.945593 229064 mini_chronyd.cc:189] Time spent starting chronyd: real 0.197s user 0.004s sys 0.016s
I20260812 17:58:43.950714 229064 external_mini_cluster.cc:1326] Running /data0/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu
/data0/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu
--fs_wal_dir=/data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1786557522090267-229064-0/minicluster-data/master-0/wal
--fs_data_dirs=/data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1786557522090267-229064-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1786557522090267-229064-0/minicluster-data/master-0/logs
--server_dump_info_path=/data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1786557522090267-229064-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.223.178.62:34763
--webserver_interface=127.223.178.62
--webserver_port=0
--builtin_ntp_servers=127.223.178.20:36841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.223.178.62:34763,127.223.178.61:44151,127.223.178.60:37935
--webserver_doc_root=/data0/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/../../../www with env {}
W20260812 17:58:46.449221 229107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:58:46.450747 229107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:58:46.452214 229107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:58:46.876721 229107 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:58:46.877625 229107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:58:46.878381 229107 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:58:46.879062 229107 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 17:58:47.389092 229107 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.223.178.20:36841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1786557522090267-229064-0/minicluster-data/master-0/data
--fs_wal_dir=/data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1786557522090267-229064-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.223.178.62:34763,127.223.178.61:44151,127.223.178.60:37935
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.223.178.62:34763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1786557522090267-229064-0/minicluster-data/master-0/data/info.pb
--webserver_doc_root=/data0/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/../../../www
--webserver_interface=127.223.178.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1786557522090267-229064-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 16:39:11 UTC on ccycloud.quasar-nphzfb.root.comops.site
build id 4582771
TSAN enabled
I20260812 17:58:47.391583 229107 env_posix.cc:2256] Not raising this process' open files per process limit of 1024000; it is already as high as it can go
I20260812 17:58:47.397027 229107 file_cache.cc:493] Constructed file cache file cache with capacity 409600
W20260812 17:58:47.594728 229115 instance_detector.cc:116] could not retrieve GCE instance metadata: Network error: curl error: Couldn't resolve host name: Could not resolve host: metadata.google.internal
W20260812 17:58:48.523214 229114 instance_detector.cc:116] could not retrieve Azure instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1009 milliseconds
W20260812 17:58:48.632464 229116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1042 milliseconds
W20260812 17:58:48.630573 229113 instance_detector.cc:116] could not retrieve AWS instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1040 milliseconds
I20260812 17:58:48.632773 229107 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260812 17:58:56.146291 229107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:58:56.151589 229107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:58:56.153314 229107 hybrid_clock.cc:648] HybridClock initialized: now 1786557536153264 us; error 61 us; skew 500 ppm
I20260812 17:58:56.155545 229107 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 17:58:56.245692 229107 webserver.cc:460] Webserver started at http://127.223.178.62:33645/ using document root /data0/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/../../../www and password file <none>
I20260812 17:58:56.248059 229107 fs_manager.cc:359] Metadata directory not provided
I20260812 17:58:56.248557 229107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:58:56.249512 229107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:58:56.343245 229107 fs_manager.cc:1063] Generated new instance metadata in path /data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1786557522090267-229064-0/minicluster-data/master-0/data/instance:
uuid: "6043727174a34ecb8abe162c8047f57c"
format_stamp: "Formatted at 2026-08-12 17:58:56 on ccycloud.quasar-nphzfb.root.comops.site"
I20260812 17:58:56.346146 229107 fs_manager.cc:1063] Generated new instance metadata in path /data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1786557522090267-229064-0/minicluster-data/master-0/wal/instance:
uuid: "6043727174a34ecb8abe162c8047f57c"
format_stamp: "Formatted at 2026-08-12 17:58:56 on ccycloud.quasar-nphzfb.root.comops.site"
I20260812 17:58:56.362636 229107 fs_manager.cc:691] Time spent creating directory manager: real 0.015s user 0.012s sys 0.001s
I20260812 17:58:56.449944 229125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:58:56.452878 229107 fs_manager.cc:725] Time spent opening block manager: real 0.082s user 0.004s sys 0.002s
I20260812 17:58:56.453653 229107 fs_manager.cc:642] Opened local filesystem: /data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1786557522090267-229064-0/minicluster-data/master-0/data,/data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1786557522090267-229064-0/minicluster-data/master-0/wal
uuid: "6043727174a34ecb8abe162c8047f57c"
format_stamp: "Formatted at 2026-08-12 17:58:56 on ccycloud.quasar-nphzfb.root.comops.site"
I20260812 17:58:56.454380 229107 fs_report.cc:383] FS layout report
--------------------
wal directory: /data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1786557522090267-229064-0/minicluster-data/master-0/wal
metadata directory: /data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1786557522090267-229064-0/minicluster-data/master-0/wal
1 data directories: /data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1786557522090267-229064-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:58:56.671013 229107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:58:56.673645 229107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 65536; it is already as high as it can go
I20260812 17:58:56.674427 229107 kserver.cc:163] Server-wide thread pool size limit: 6553
I20260812 17:58:57.198730 229107 rpc_server.cc:307] RPC server started. Bound to: 127.223.178.62:34763
I20260812 17:58:57.198884 229176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.223.178.62:34763 every 8 connection(s)
I20260812 17:58:57.282938 229107 server_base.cc:1166] Dumped server information to /data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1786557522090267-229064-0/minicluster-data/master-0/data/info.pb
I20260812 17:58:57.287895 229064 external_mini_cluster.cc:1388] Started /data0/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu as pid 229107
I20260812 17:58:57.288697 229064 external_mini_cluster.cc:1402] Reading /data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1786557522090267-229064-0/minicluster-data/master-0/wal/instance
I20260812 17:58:57.292600 229177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:58:57.297433 229064 external_mini_cluster.cc:1326] Running /data0/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu
/data0/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu
--fs_wal_dir=/data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1786557522090267-229064-0/minicluster-data/master-1/wal
--fs_data_dirs=/data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1786557522090267-229064-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1786557522090267-229064-0/minicluster-data/master-1/logs
--server_dump_info_path=/data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1786557522090267-229064-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.223.178.61:44151
--webserver_interface=127.223.178.61
--webserver_port=0
--builtin_ntp_servers=127.223.178.20:36841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.223.178.62:34763,127.223.178.61:44151,127.223.178.60:37935
--webserver_doc_root=/data0/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/../../../www with env {}
I20260812 17:58:57.386415 229177 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.223.178.62" port: 34763 } has no permanent_uuid. Determining permanent_uuid...
I20260812 17:58:57.588922 229177 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.223.178.61" port: 44151 } has no permanent_uuid. Determining permanent_uuid...
W20260812 17:58:57.594798 229126 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.223.178.61:44151: connect: Connection refused (error 111)
W20260812 17:58:57.600273 229177 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.223.178.61:44151: Network error: Client connection negotiation failed: client connection to 127.223.178.61:44151: connect: Connection refused (error 111)
I20260812 17:58:57.689272 229177 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.223.178.61" port: 44151 } attempt: 1
W20260812 17:58:57.698269 229177 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.223.178.61:44151: Network error: Client connection negotiation failed: client connection to 127.223.178.61:44151: connect: Connection refused (error 111)
I20260812 17:58:57.792867 229177 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.223.178.61" port: 44151 } attempt: 2
W20260812 17:58:57.804842 229177 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.223.178.61:44151: Network error: Client connection negotiation failed: client connection to 127.223.178.61:44151: connect: Connection refused (error 111)
I20260812 17:58:57.901242 229177 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.223.178.61" port: 44151 } attempt: 3
W20260812 17:58:57.910475 229177 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.223.178.61:44151: Network error: Client connection negotiation failed: client connection to 127.223.178.61:44151: connect: Connection refused (error 111)
I20260812 17:58:58.105891 229177 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.223.178.61" port: 44151 } attempt: 4
W20260812 17:58:58.115579 229177 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.223.178.61:44151: Network error: Client connection negotiation failed: client connection to 127.223.178.61:44151: connect: Connection refused (error 111)
I20260812 17:58:58.420034 229177 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.223.178.61" port: 44151 } attempt: 5
W20260812 17:58:58.427325 229177 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.223.178.61:44151: Network error: Client connection negotiation failed: client connection to 127.223.178.61:44151: connect: Connection refused (error 111)
I20260812 17:58:59.040766 229177 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.223.178.61" port: 44151 } attempt: 6
W20260812 17:58:59.059103 229177 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.223.178.61:44151: Network error: Client connection negotiation failed: client connection to 127.223.178.61:44151: connect: Connection refused (error 111)
W20260812 17:58:59.280373 229179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:58:59.281181 229179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:58:59.281991 229179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:58:59.689586 229179 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:58:59.690305 229179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:58:59.690874 229179 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:58:59.691594 229179 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 17:59:00.116194 229179 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.223.178.20:36841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1786557522090267-229064-0/minicluster-data/master-1/data
--fs_wal_dir=/data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1786557522090267-229064-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.223.178.62:34763,127.223.178.61:44151,127.223.178.60:37935
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.223.178.61:44151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1786557522090267-229064-0/minicluster-data/master-1/data/info.pb
--webserver_doc_root=/data0/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/../../../www
--webserver_interface=127.223.178.61
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1786557522090267-229064-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 16:39:11 UTC on ccycloud.quasar-nphzfb.root.comops.site
build id 4582771
TSAN enabled
I20260812 17:59:00.117900 229179 env_posix.cc:2256] Not raising this process' open files per process limit of 1024000; it is already as high as it can go
I20260812 17:59:00.187879 229177 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.223.178.61" port: 44151 } attempt: 7
I20260812 17:59:00.191625 229179 file_cache.cc:493] Constructed file cache file cache with capacity 409600
W20260812 17:59:00.203539 229177 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.223.178.61:44151: Network error: Client connection negotiation failed: client connection to 127.223.178.61:44151: connect: Connection refused (error 111)
W20260812 17:59:00.307302 229194 instance_detector.cc:116] could not retrieve GCE instance metadata: Network error: curl error: Couldn't resolve host name: Could not resolve host: metadata.google.internal
I20260812 17:59:02.294142 229177 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.223.178.61" port: 44151 } attempt: 8
W20260812 17:59:02.301645 229177 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.223.178.61:44151: Network error: Client connection negotiation failed: client connection to 127.223.178.61:44151: connect: Connection refused (error 111)
I20260812 17:59:06.447389 229177 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.223.178.61" port: 44151 } attempt: 9
W20260812 17:59:06.453634 229126 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.223.178.61:44151: connect: Connection refused (error 111) [suppressed 8 similar messages]
W20260812 17:59:06.458765 229177 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.223.178.61:44151: Network error: Client connection negotiation failed: client connection to 127.223.178.61:44151: connect: Connection refused (error 111)
W20260812 17:59:01.339762 229195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1032 milliseconds
W20260812 17:59:01.340325 229192 instance_detector.cc:116] could not retrieve AWS instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1035 milliseconds
I20260812 17:59:01.340045 229179 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20260812 17:59:01.230031 229193 instance_detector.cc:116] could not retrieve Azure instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1007 milliseconds
I20260812 17:59:07.393041 229179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:59:07.396926 229179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:59:07.399518 229179 hybrid_clock.cc:648] HybridClock initialized: now 1786557547399446 us; error 55 us; skew 500 ppm
I20260812 17:59:07.401418 229179 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 17:59:07.486034 229179 webserver.cc:460] Webserver started at http://127.223.178.61:36713/ using document root /data0/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/../../../www and password file <none>
I20260812 17:59:07.498235 229179 fs_manager.cc:359] Metadata directory not provided
I20260812 17:59:07.498922 229179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:59:07.501039 229179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:59:07.588707 229179 fs_manager.cc:1063] Generated new instance metadata in path /data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1786557522090267-229064-0/minicluster-data/master-1/data/instance:
uuid: "ab277f69e84345f49e335f7c49f2a1cb"
format_stamp: "Formatted at 2026-08-12 17:59:07 on ccycloud.quasar-nphzfb.root.comops.site"
I20260812 17:59:07.591724 229179 fs_manager.cc:1063] Generated new instance metadata in path /data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1786557522090267-229064-0/minicluster-data/master-1/wal/instance:
uuid: "ab277f69e84345f49e335f7c49f2a1cb"
format_stamp: "Formatted at 2026-08-12 17:59:07 on ccycloud.quasar-nphzfb.root.comops.site"
I20260812 17:59:07.605628 229179 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.008s sys 0.002s
I20260812 17:59:07.694227 229208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:59:07.696448 229179 fs_manager.cc:725] Time spent opening block manager: real 0.081s user 0.003s sys 0.001s
I20260812 17:59:07.697013 229179 fs_manager.cc:642] Opened local filesystem: /data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1786557522090267-229064-0/minicluster-data/master-1/data,/data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1786557522090267-229064-0/minicluster-data/master-1/wal
uuid: "ab277f69e84345f49e335f7c49f2a1cb"
format_stamp: "Formatted at 2026-08-12 17:59:07 on ccycloud.quasar-nphzfb.root.comops.site"
I20260812 17:59:07.697602 229179 fs_report.cc:383] FS layout report
--------------------
wal directory: /data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1786557522090267-229064-0/minicluster-data/master-1/wal
metadata directory: /data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1786557522090267-229064-0/minicluster-data/master-1/wal
1 data directories: /data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1786557522090267-229064-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:59:07.923450 229179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:59:07.925904 229179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 65536; it is already as high as it can go
I20260812 17:59:07.926563 229179 kserver.cc:163] Server-wide thread pool size limit: 6553
I20260812 17:59:08.648452 229179 rpc_server.cc:307] RPC server started. Bound to: 127.223.178.61:44151
I20260812 17:59:08.648548 229259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.223.178.61:44151 every 8 connection(s)
I20260812 17:59:08.654965 229179 server_base.cc:1166] Dumped server information to /data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1786557522090267-229064-0/minicluster-data/master-1/data/info.pb
I20260812 17:59:08.655695 229064 external_mini_cluster.cc:1388] Started /data0/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu as pid 229179
I20260812 17:59:08.656361 229064 external_mini_cluster.cc:1402] Reading /data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1786557522090267-229064-0/minicluster-data/master-1/wal/instance
I20260812 17:59:08.737574 229064 external_mini_cluster.cc:1326] Running /data0/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu
/data0/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu
--fs_wal_dir=/data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1786557522090267-229064-0/minicluster-data/master-2/wal
--fs_data_dirs=/data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1786557522090267-229064-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1786557522090267-229064-0/minicluster-data/master-2/logs
--server_dump_info_path=/data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1786557522090267-229064-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.223.178.60:37935
--webserver_interface=127.223.178.60
--webserver_port=0
--builtin_ntp_servers=127.223.178.20:36841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.223.178.62:34763,127.223.178.61:44151,127.223.178.60:37935
--webserver_doc_root=/data0/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/../../../www with env {}
I20260812 17:59:08.754500 229260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:59:08.853081 229260 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.223.178.62" port: 34763 } has no permanent_uuid. Determining permanent_uuid...
I20260812 17:59:09.157339 229260 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.223.178.61" port: 44151 } has no permanent_uuid. Determining permanent_uuid...
I20260812 17:59:09.272935 229260 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.223.178.60" port: 37935 } has no permanent_uuid. Determining permanent_uuid...
W20260812 17:59:09.277063 229212 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.223.178.60:37935: connect: Connection refused (error 111)
W20260812 17:59:09.359463 229260 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.223.178.60:37935: Network error: Client connection negotiation failed: client connection to 127.223.178.60:37935: connect: Connection refused (error 111)
I20260812 17:59:09.463693 229260 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.223.178.60" port: 37935 } attempt: 1
W20260812 17:59:09.475003 229260 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.223.178.60:37935: Network error: Client connection negotiation failed: client connection to 127.223.178.60:37935: connect: Connection refused (error 111)
I20260812 17:59:09.563267 229260 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.223.178.60" port: 37935 } attempt: 2
W20260812 17:59:09.576189 229260 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.223.178.60:37935: Network error: Client connection negotiation failed: client connection to 127.223.178.60:37935: connect: Connection refused (error 111)
I20260812 17:59:09.669883 229260 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.223.178.60" port: 37935 } attempt: 3
W20260812 17:59:09.679667 229260 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.223.178.60:37935: Network error: Client connection negotiation failed: client connection to 127.223.178.60:37935: connect: Connection refused (error 111)
I20260812 17:59:09.874284 229260 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.223.178.60" port: 37935 } attempt: 4
W20260812 17:59:09.889405 229260 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.223.178.60:37935: Network error: Client connection negotiation failed: client connection to 127.223.178.60:37935: connect: Connection refused (error 111)
I20260812 17:59:10.193742 229260 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.223.178.60" port: 37935 } attempt: 5
W20260812 17:59:10.201900 229260 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.223.178.60:37935: Network error: Client connection negotiation failed: client connection to 127.223.178.60:37935: connect: Connection refused (error 111)
I20260812 17:59:10.749503 229260 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.223.178.60" port: 37935 } attempt: 6
W20260812 17:59:10.829475 229260 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.223.178.60:37935: Network error: Client connection negotiation failed: client connection to 127.223.178.60:37935: connect: Connection refused (error 111)
W20260812 17:59:11.364919 229262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:59:11.442108 229262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:59:11.444136 229262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:59:11.782711 229262 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:59:11.783149 229262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:59:11.783493 229262 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:59:11.783792 229262 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 17:59:11.959182 229260 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.223.178.60" port: 37935 } attempt: 7
W20260812 17:59:11.973508 229260 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.223.178.60:37935: Network error: Client connection negotiation failed: client connection to 127.223.178.60:37935: connect: Connection refused (error 111)
I20260812 17:59:12.194644 229262 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.223.178.20:36841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1786557522090267-229064-0/minicluster-data/master-2/data
--fs_wal_dir=/data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1786557522090267-229064-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.223.178.62:34763,127.223.178.61:44151,127.223.178.60:37935
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.223.178.60:37935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1786557522090267-229064-0/minicluster-data/master-2/data/info.pb
--webserver_doc_root=/data0/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/../../../www
--webserver_interface=127.223.178.60
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1786557522090267-229064-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 16:39:11 UTC on ccycloud.quasar-nphzfb.root.comops.site
build id 4582771
TSAN enabled
I20260812 17:59:12.196270 229262 env_posix.cc:2256] Not raising this process' open files per process limit of 1024000; it is already as high as it can go
I20260812 17:59:12.198303 229262 file_cache.cc:493] Constructed file cache file cache with capacity 409600
W20260812 17:59:12.381994 229279 instance_detector.cc:116] could not retrieve GCE instance metadata: Network error: curl error: Couldn't resolve host name: Could not resolve host: metadata.google.internal
I20260812 17:59:14.064042 229260 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.223.178.60" port: 37935 } attempt: 8
W20260812 17:59:14.071465 229260 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.223.178.60:37935: Network error: Client connection negotiation failed: client connection to 127.223.178.60:37935: connect: Connection refused (error 111)
I20260812 17:59:14.672361 229177 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.223.178.61" port: 44151 } attempt: 10
I20260812 17:59:14.694089 229177 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.223.178.60" port: 37935 } has no permanent_uuid. Determining permanent_uuid...
W20260812 17:59:14.697486 229129 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.223.178.60:37935: connect: Connection refused (error 111)
W20260812 17:59:14.701467 229177 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.223.178.60:37935: Network error: Client connection negotiation failed: client connection to 127.223.178.60:37935: connect: Connection refused (error 111)
I20260812 17:59:14.775485 229177 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.223.178.60" port: 37935 } attempt: 1
W20260812 17:59:14.785676 229177 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.223.178.60:37935: Network error: Client connection negotiation failed: client connection to 127.223.178.60:37935: connect: Connection refused (error 111)
I20260812 17:59:14.877861 229177 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.223.178.60" port: 37935 } attempt: 2
W20260812 17:59:14.887188 229177 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.223.178.60:37935: Network error: Client connection negotiation failed: client connection to 127.223.178.60:37935: connect: Connection refused (error 111)
I20260812 17:59:14.991773 229177 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.223.178.60" port: 37935 } attempt: 3
W20260812 17:59:14.999568 229177 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.223.178.60:37935: Network error: Client connection negotiation failed: client connection to 127.223.178.60:37935: connect: Connection refused (error 111)
I20260812 17:59:15.189934 229177 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.223.178.60" port: 37935 } attempt: 4
W20260812 17:59:15.201679 229177 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.223.178.60:37935: Network error: Client connection negotiation failed: client connection to 127.223.178.60:37935: connect: Connection refused (error 111)
I20260812 17:59:15.502185 229177 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.223.178.60" port: 37935 } attempt: 5
W20260812 17:59:15.513773 229177 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.223.178.60:37935: Network error: Client connection negotiation failed: client connection to 127.223.178.60:37935: connect: Connection refused (error 111)
I20260812 17:59:16.052453 229177 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.223.178.60" port: 37935 } attempt: 6
W20260812 17:59:16.140875 229177 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.223.178.60:37935: Network error: Client connection negotiation failed: client connection to 127.223.178.60:37935: connect: Connection refused (error 111)
I20260812 17:59:17.270484 229177 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.223.178.60" port: 37935 } attempt: 7
W20260812 17:59:17.285830 229177 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.223.178.60:37935: Network error: Client connection negotiation failed: client connection to 127.223.178.60:37935: connect: Connection refused (error 111)
I20260812 17:59:18.217167 229260 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.223.178.60" port: 37935 } attempt: 9
W20260812 17:59:18.221210 229212 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.223.178.60:37935: connect: Connection refused (error 111) [suppressed 8 similar messages]
W20260812 17:59:18.225539 229260 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.223.178.60:37935: Network error: Client connection negotiation failed: client connection to 127.223.178.60:37935: connect: Connection refused (error 111)
W20260812 17:59:13.312737 229277 instance_detector.cc:116] could not retrieve AWS instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1011 milliseconds
W20260812 17:59:13.317420 229280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1016 milliseconds
W20260812 17:59:13.317852 229278 instance_detector.cc:116] could not retrieve Azure instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1016 milliseconds
I20260812 17:59:13.317627 229262 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260812 17:59:18.746264 229262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:59:18.750109 229262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:59:18.751679 229262 hybrid_clock.cc:648] HybridClock initialized: now 1786557558751634 us; error 45 us; skew 500 ppm
I20260812 17:59:18.753062 229262 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 17:59:18.760314 229262 webserver.cc:460] Webserver started at http://127.223.178.60:35119/ using document root /data0/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/../../../www and password file <none>
I20260812 17:59:18.761819 229262 fs_manager.cc:359] Metadata directory not provided
I20260812 17:59:18.762132 229262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:59:18.762810 229262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:59:18.840780 229262 fs_manager.cc:1063] Generated new instance metadata in path /data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1786557522090267-229064-0/minicluster-data/master-2/data/instance:
uuid: "9ff7142311674389b7ed69abe16dc022"
format_stamp: "Formatted at 2026-08-12 17:59:18 on ccycloud.quasar-nphzfb.root.comops.site"
I20260812 17:59:18.843017 229262 fs_manager.cc:1063] Generated new instance metadata in path /data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1786557522090267-229064-0/minicluster-data/master-2/wal/instance:
uuid: "9ff7142311674389b7ed69abe16dc022"
format_stamp: "Formatted at 2026-08-12 17:59:18 on ccycloud.quasar-nphzfb.root.comops.site"
I20260812 17:59:18.855031 229262 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.008s sys 0.002s
I20260812 17:59:18.865497 229300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:59:18.867291 229262 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.004s sys 0.001s
I20260812 17:59:18.867777 229262 fs_manager.cc:642] Opened local filesystem: /data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1786557522090267-229064-0/minicluster-data/master-2/data,/data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1786557522090267-229064-0/minicluster-data/master-2/wal
uuid: "9ff7142311674389b7ed69abe16dc022"
format_stamp: "Formatted at 2026-08-12 17:59:18 on ccycloud.quasar-nphzfb.root.comops.site"
I20260812 17:59:18.943624 229262 fs_report.cc:383] FS layout report
--------------------
wal directory: /data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1786557522090267-229064-0/minicluster-data/master-2/wal
metadata directory: /data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1786557522090267-229064-0/minicluster-data/master-2/wal
1 data directories: /data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1786557522090267-229064-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:59:19.155777 229262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:59:19.159011 229262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 65536; it is already as high as it can go
I20260812 17:59:19.159893 229262 kserver.cc:163] Server-wide thread pool size limit: 6553
I20260812 17:59:19.360694 229177 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.223.178.60" port: 37935 } attempt: 8
W20260812 17:59:19.370802 229177 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.223.178.60:37935: Network error: Client connection negotiation failed: client connection to 127.223.178.60:37935: connect: Connection refused (error 111)
I20260812 17:59:19.568377 229262 rpc_server.cc:307] RPC server started. Bound to: 127.223.178.60:37935
I20260812 17:59:19.569085 229354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.223.178.60:37935 every 8 connection(s)
I20260812 17:59:19.575608 229262 server_base.cc:1166] Dumped server information to /data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1786557522090267-229064-0/minicluster-data/master-2/data/info.pb
I20260812 17:59:19.584287 229064 external_mini_cluster.cc:1388] Started /data0/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu as pid 229262
I20260812 17:59:19.585060 229355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:59:19.585177 229064 external_mini_cluster.cc:1402] Reading /data0/somelongdirectorytoavoidrpathissues/src/kudutest/webserver-crawl-itest.0.WebserverCrawlITest.TestAllWebPages_0.1786557522090267-229064-0/minicluster-data/master-2/wal/instance
I20260812 17:59:19.689775 229355 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.223.178.62" port: 34763 } has no permanent_uuid. Determining permanent_uuid...
I20260812 17:59:19.897753 229355 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.223.178.61" port: 44151 } has no permanent_uuid. Determining permanent_uuid...
I20260812 17:59:20.002156 229355 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.223.178.60" port: 37935 } has no permanent_uuid. Determining permanent_uuid...
I20260812 17:59:20.422499 229355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ff7142311674389b7ed69abe16dc022: Bootstrap starting.
I20260812 17:59:20.513653 229355 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ff7142311674389b7ed69abe16dc022: Neither blocks nor log segments found. Creating new log.
I20260812 17:59:20.517791 229355 log.cc:824] T 00000000000000000000000000000000 P 9ff7142311674389b7ed69abe16dc022: Log is configured to *not* fsync() on all Append() calls
I20260812 17:59:20.526706 229355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ff7142311674389b7ed69abe16dc022: No bootstrap required, opened a new log
I20260812 17:59:20.732275 229355 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ff7142311674389b7ed69abe16dc022 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6043727174a34ecb8abe162c8047f57c" member_type: VOTER last_known_addr { host: "127.223.178.62" port: 34763 } } peers { permanent_uuid: "ab277f69e84345f49e335f7c49f2a1cb" member_type: VOTER last_known_addr { host: "127.223.178.61" port: 44151 } } peers { permanent_uuid: "9ff7142311674389b7ed69abe16dc022" member_type: VOTER last_known_addr { host: "127.223.178.60" port: 37935 } }
I20260812 17:59:20.733677 229355 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ff7142311674389b7ed69abe16dc022 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:59:20.734086 229355 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ff7142311674389b7ed69abe16dc022 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ff7142311674389b7ed69abe16dc022, State: Initialized, Role: FOLLOWER
I20260812 17:59:20.735482 229355 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ff7142311674389b7ed69abe16dc022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6043727174a34ecb8abe162c8047f57c" member_type: VOTER last_known_addr { host: "127.223.178.62" port: 34763 } } peers { permanent_uuid: "ab277f69e84345f49e335f7c49f2a1cb" member_type: VOTER last_known_addr { host: "127.223.178.61" port: 44151 } } peers { permanent_uuid: "9ff7142311674389b7ed69abe16dc022" member_type: VOTER last_known_addr { host: "127.223.178.60" port: 37935 } }
I20260812 17:59:20.813130 229366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ff7142311674389b7ed69abe16dc022 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6043727174a34ecb8abe162c8047f57c" member_type: VOTER last_known_addr { host: "127.223.178.62" port: 34763 } } peers { permanent_uuid: "ab277f69e84345f49e335f7c49f2a1cb" member_type: VOTER last_known_addr { host: "127.223.178.61" port: 44151 } } peers { permanent_uuid: "9ff7142311674389b7ed69abe16dc022" member_type: VOTER last_known_addr { host: "127.223.178.60" port: 37935 } } }
I20260812 17:59:20.816653 229366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ff7142311674389b7ed69abe16dc022 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 17:59:20.821000 229355 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ff7142311674389b7ed69abe16dc022 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 17:59:20.919580 229366 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ff7142311674389b7ed69abe16dc022 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 17:59:20.921227 229366 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ff7142311674389b7ed69abe16dc022 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6043727174a34ecb8abe162c8047f57c" member_type: VOTER last_known_addr { host: "127.223.178.62" port: 34763 } } peers { permanent_uuid: "ab277f69e84345f49e335f7c49f2a1cb" member_type: VOTER last_known_addr { host: "127.223.178.61" port: 44151 } } peers { permanent_uuid: "9ff7142311674389b7ed69abe16dc022" member_type: VOTER last_known_addr { host: "127.223.178.60" port: 37935 } }
I20260812 17:59:20.926621 229366 leader_election.cc:288] T 00000000000000000000000000000000 P 9ff7142311674389b7ed69abe16dc022 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6043727174a34ecb8abe162c8047f57c (127.223.178.62:34763), ab277f69e84345f49e335f7c49f2a1cb (127.223.178.61:44151)
W20260812 17:59:20.943645 229377 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9ff7142311674389b7ed69abe16dc022: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 17:59:20.944168 229377 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260812 17:59:21.021974 229306 tablet.cc:2369] T 00000000000000000000000000000000 P 9ff7142311674389b7ed69abe16dc022: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 17:59:21.340303 229351 debug-util.cc:398] Leaking SignalData structure 0x7b0800070060 after lost signal to thread 229270
W20260812 17:59:21.342718 229351 debug-util.cc:398] Leaking SignalData structure 0x7b08000707a0 after lost signal to thread 229354
I20260812 17:59:20.942203 229262 master_runner.cc:186] Error getting master registration for 127.223.178.62:34763: OK, instance_id { permanent_uuid: "6043727174a34ecb8abe162c8047f57c" instance_seqno: 1786557537086536 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 17:59:21.358084 229262 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260812 17:59:21.970099 229377 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9ff7142311674389b7ed69abe16dc022: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 17:59:21.970944 229377 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260812 17:59:22.583154 229302 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.223.178.62:34763 timed out after 1.541s (SENT)
W20260812 17:59:23.009343 229377 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9ff7142311674389b7ed69abe16dc022: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 17:59:23.009828 229377 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 17:59:23.523623 229177 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.223.178.60" port: 37935 } attempt: 9
I20260812 17:59:23.635737 229177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6043727174a34ecb8abe162c8047f57c: Bootstrap starting.
I20260812 17:59:23.648195 229177 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6043727174a34ecb8abe162c8047f57c: Neither blocks nor log segments found. Creating new log.
I20260812 17:59:23.651757 229177 log.cc:824] T 00000000000000000000000000000000 P 6043727174a34ecb8abe162c8047f57c: Log is configured to *not* fsync() on all Append() calls
I20260812 17:59:23.735772 229177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6043727174a34ecb8abe162c8047f57c: No bootstrap required, opened a new log
I20260812 17:59:23.857885 229177 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6043727174a34ecb8abe162c8047f57c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6043727174a34ecb8abe162c8047f57c" member_type: VOTER last_known_addr { host: "127.223.178.62" port: 34763 } } peers { permanent_uuid: "ab277f69e84345f49e335f7c49f2a1cb" member_type: VOTER last_known_addr { host: "127.223.178.61" port: 44151 } } peers { permanent_uuid: "9ff7142311674389b7ed69abe16dc022" member_type: VOTER last_known_addr { host: "127.223.178.60" port: 37935 } }
I20260812 17:59:23.859254 229177 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6043727174a34ecb8abe162c8047f57c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:59:23.859731 229177 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6043727174a34ecb8abe162c8047f57c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6043727174a34ecb8abe162c8047f57c, State: Initialized, Role: FOLLOWER
I20260812 17:59:23.861220 229177 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6043727174a34ecb8abe162c8047f57c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6043727174a34ecb8abe162c8047f57c" member_type: VOTER last_known_addr { host: "127.223.178.62" port: 34763 } } peers { permanent_uuid: "ab277f69e84345f49e335f7c49f2a1cb" member_type: VOTER last_known_addr { host: "127.223.178.61" port: 44151 } } peers { permanent_uuid: "9ff7142311674389b7ed69abe16dc022" member_type: VOTER last_known_addr { host: "127.223.178.60" port: 37935 } }
I20260812 17:59:23.941155 229385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6043727174a34ecb8abe162c8047f57c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6043727174a34ecb8abe162c8047f57c" member_type: VOTER last_known_addr { host: "127.223.178.62" port: 34763 } } peers { permanent_uuid: "ab277f69e84345f49e335f7c49f2a1cb" member_type: VOTER last_known_addr { host: "127.223.178.61" port: 44151 } } peers { permanent_uuid: "9ff7142311674389b7ed69abe16dc022" member_type: VOTER last_known_addr { host: "127.223.178.60" port: 37935 } } }
I20260812 17:59:23.946671 229385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6043727174a34ecb8abe162c8047f57c [sys.catalog]: This master's current role is: FOLLOWER
I20260812 17:59:23.947712 229385 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6043727174a34ecb8abe162c8047f57c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 17:59:23.948513 229385 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6043727174a34ecb8abe162c8047f57c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6043727174a34ecb8abe162c8047f57c" member_type: VOTER last_known_addr { host: "127.223.178.62" port: 34763 } } peers { permanent_uuid: "ab277f69e84345f49e335f7c49f2a1cb" member_type: VOTER last_known_addr { host: "127.223.178.61" port: 44151 } } peers { permanent_uuid: "9ff7142311674389b7ed69abe16dc022" member_type: VOTER last_known_addr { host: "127.223.178.60" port: 37935 } }
I20260812 17:59:23.952206 229177 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6043727174a34ecb8abe162c8047f57c [sys.catalog]: configured and running, proceeding with master startup.
I20260812 17:59:23.952454 229152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ff7142311674389b7ed69abe16dc022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6043727174a34ecb8abe162c8047f57c" is_pre_election: true
I20260812 17:59:23.952476 229151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ff7142311674389b7ed69abe16dc022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6043727174a34ecb8abe162c8047f57c" is_pre_election: true
I20260812 17:59:23.953685 229152 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6043727174a34ecb8abe162c8047f57c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ff7142311674389b7ed69abe16dc022 in term 0.
W20260812 17:59:23.954360 229152 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45426 (request call id 1) took 3026 ms (client timeout 1541 ms). Trace:
W20260812 17:59:23.954205 229151 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45426 (request call id 3) took 1364 ms (client timeout 1541 ms). Trace:
W20260812 17:59:23.954901 229151 rpcz_store.cc:269] 0812 17:59:22.589550 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 17:59:22.589840 (+ 290us) service_pool.cc:227] Handling call
0812 17:59:23.954095 (+1364255us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260812 17:59:23.954845 229152 rpcz_store.cc:269] 0812 17:59:20.928159 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 17:59:20.928377 (+ 218us) service_pool.cc:227] Handling call
0812 17:59:23.954276 (+3025899us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 17:59:23.959692 229385 leader_election.cc:288] T 00000000000000000000000000000000 P 6043727174a34ecb8abe162c8047f57c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab277f69e84345f49e335f7c49f2a1cb (127.223.178.61:44151), 9ff7142311674389b7ed69abe16dc022 (127.223.178.60:37935)
I20260812 17:59:23.960788 229330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6043727174a34ecb8abe162c8047f57c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ff7142311674389b7ed69abe16dc022" is_pre_election: true
I20260812 17:59:23.962110 229330 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9ff7142311674389b7ed69abe16dc022 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6043727174a34ecb8abe162c8047f57c in term 0.
I20260812 17:59:23.963999 229129 leader_election.cc:302] T 00000000000000000000000000000000 P 6043727174a34ecb8abe162c8047f57c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6043727174a34ecb8abe162c8047f57c, 9ff7142311674389b7ed69abe16dc022; no voters:
I20260812 17:59:23.965378 229385 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6043727174a34ecb8abe162c8047f57c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 17:59:23.965826 229385 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6043727174a34ecb8abe162c8047f57c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 17:59:23.966841 229385 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6043727174a34ecb8abe162c8047f57c [term 0 FOLLOWER]: Advancing to term 1
W20260812 17:59:24.050087 229131 tablet.cc:2369] T 00000000000000000000000000000000 P 6043727174a34ecb8abe162c8047f57c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 17:59:24.055235 229377 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9ff7142311674389b7ed69abe16dc022: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260812 17:59:24.055168 229385 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6043727174a34ecb8abe162c8047f57c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6043727174a34ecb8abe162c8047f57c" member_type: VOTER last_known_addr { host: "127.223.178.62" port: 34763 } } peers { permanent_uuid: "ab277f69e84345f49e335f7c49f2a1cb" member_type: VOTER last_known_addr { host: "127.223.178.61" port: 44151 } } peers { permanent_uuid: "9ff7142311674389b7ed69abe16dc022" member_type: VOTER last_known_addr { host: "127.223.178.60" port: 37935 } }
W20260812 17:59:24.056213 229377 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 17:59:24.058065 229385 leader_election.cc:288] T 00000000000000000000000000000000 P 6043727174a34ecb8abe162c8047f57c [CANDIDATE]: Term 1 election: Requested vote from peers ab277f69e84345f49e335f7c49f2a1cb (127.223.178.61:44151), 9ff7142311674389b7ed69abe16dc022 (127.223.178.60:37935)
I20260812 17:59:24.059675 229330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6043727174a34ecb8abe162c8047f57c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ff7142311674389b7ed69abe16dc022"
I20260812 17:59:24.060559 229330 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ff7142311674389b7ed69abe16dc022 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:59:24.069926 229330 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9ff7142311674389b7ed69abe16dc022 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6043727174a34ecb8abe162c8047f57c in term 1.
I20260812 17:59:24.071445 229129 leader_election.cc:302] T 00000000000000000000000000000000 P 6043727174a34ecb8abe162c8047f57c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6043727174a34ecb8abe162c8047f57c, 9ff7142311674389b7ed69abe16dc022; no voters:
W20260812 17:59:24.150014 229129 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.223.178.60:37935, user_credentials={real_user=systest}} blocked reactor thread for 58894.6us
I20260812 17:59:24.151316 229385 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6043727174a34ecb8abe162c8047f57c [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:59:24.154379 229385 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6043727174a34ecb8abe162c8047f57c [term 1 LEADER]: Becoming Leader. State: Replica: 6043727174a34ecb8abe162c8047f57c, State: Running, Role: LEADER
I20260812 17:59:24.156687 229385 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6043727174a34ecb8abe162c8047f57c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6043727174a34ecb8abe162c8047f57c" member_type: VOTER last_known_addr { host: "127.223.178.62" port: 34763 } } peers { permanent_uuid: "ab277f69e84345f49e335f7c49f2a1cb" member_type: VOTER last_known_addr { host: "127.223.178.61" port: 44151 } } peers { permanent_uuid: "9ff7142311674389b7ed69abe16dc022" member_type: VOTER last_known_addr { host: "127.223.178.60" port: 37935 } }
W20260812 17:59:24.164930 229397 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6043727174a34ecb8abe162c8047f57c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 17:59:24.165632 229397 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 17:59:24.250771 229384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6043727174a34ecb8abe162c8047f57c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6043727174a34ecb8abe162c8047f57c. Latest consensus state: current_term: 1 leader_uuid: "6043727174a34ecb8abe162c8047f57c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6043727174a34ecb8abe162c8047f57c" member_type: VOTER last_known_addr { host: "127.223.178.62" port: 34763 } } peers { permanent_uuid: "ab277f69e84345f49e335f7c49f2a1cb" member_type: VOTER last_known_addr { host: "127.223.178.61" port: 44151 } } peers { permanent_uuid: "9ff7142311674389b7ed69abe16dc022" member_type: VOTER last_known_addr { host: "127.223.178.60" port: 37935 } } }
I20260812 17:59:24.253297 229384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6043727174a34ecb8abe162c8047f57c [sys.catalog]: This master's current role is: LEADER
W20260812 17:59:24.251601 229301 leader_election.cc:334] T 00000000000000000000000000000000 P 9ff7142311674389b7ed69abe16dc022 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer ab277f69e84345f49e335f7c49f2a1cb (127.223.178.61:44151): Timed out: RequestConsensusVote RPC to 127.223.178.61:44151 timed out after 1.541s (SENT)
I20260812 17:59:24.260066 229400 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 17:59:24.259793 229301 leader_election.cc:302] T 00000000000000000000000000000000 P 9ff7142311674389b7ed69abe16dc022 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ff7142311674389b7ed69abe16dc022; no voters: 6043727174a34ecb8abe162c8047f57c, ab277f69e84345f49e335f7c49f2a1cb
I20260812 17:59:24.267393 229401 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 9ff7142311674389b7ed69abe16dc022 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 17:59:24.269862 229400 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 17:59:24.162676 229107 master_runner.cc:186] Error getting master registration for 127.223.178.61:44151: OK, instance_id { permanent_uuid: "ab277f69e84345f49e335f7c49f2a1cb" instance_seqno: 1786557548445897 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 17:59:24.459517 229107 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 17:59:24.482400 229330 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9ff7142311674389b7ed69abe16dc022 [term 1 FOLLOWER]: Refusing update from remote peer 6043727174a34ecb8abe162c8047f57c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 17:59:24.485697 229385 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6043727174a34ecb8abe162c8047f57c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ff7142311674389b7ed69abe16dc022" member_type: VOTER last_known_addr { host: "127.223.178.60" port: 37935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
F20260812 17:59:24.590375 229064 external_mini_cluster.cc:634] Check failed: leader_master() != nullptr Must have started at least 1 master before adding tablet servers
*** Check failure stack trace: ***
*** Aborted at 1786557564 (unix time) try "date -d @1786557564" if you are using GNU date ***
W20260812 17:59:24.688136 229318 rpcz_store.cc:267] Call kudu.master.MasterService.ConnectToMaster from 127.0.0.1:46718 (request call id 43) took 114 ms (client timeout 20 ms). Trace:
W20260812 17:59:24.688817 229318 rpcz_store.cc:269] 0812 17:59:24.574011 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 17:59:24.574377 (+ 366us) service_pool.cc:227] Handling call
0812 17:59:24.687286 (+112909us) spinlock_profiling.cc:243] Waited 77.7 ms on lock 0x7b54000118d4. stack: 00007fa34f26a033 00007fa34f269f30 00007fa34edeb534 00000000004db2b4 00000000004db249 00007fa3560ed471 00007fa356170fe9 00007fa3580fdbd6 00007fa35813bf87 00007fa355c70534 00007fa355c704c2 00007fa355c703f1 00007fa355c7036c 00007fa355c6f622 00007fa353f1894c 00007fa353f17d86
0812 17:59:24.687836 (+ 550us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":248593664}
I20260812 17:59:24.690557 229384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6043727174a34ecb8abe162c8047f57c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6043727174a34ecb8abe162c8047f57c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6043727174a34ecb8abe162c8047f57c" member_type: VOTER last_known_addr { host: "127.223.178.62" port: 34763 } } peers { permanent_uuid: "ab277f69e84345f49e335f7c49f2a1cb" member_type: VOTER last_known_addr { host: "127.223.178.61" port: 44151 } } peers { permanent_uuid: "9ff7142311674389b7ed69abe16dc022" member_type: VOTER last_known_addr { host: "127.223.178.60" port: 37935 } } }
I20260812 17:59:24.691538 229384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6043727174a34ecb8abe162c8047f57c [sys.catalog]: This master's current role is: LEADER
I20260812 17:59:24.691659 229401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ff7142311674389b7ed69abe16dc022 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6043727174a34ecb8abe162c8047f57c. Latest consensus state: current_term: 1 leader_uuid: "6043727174a34ecb8abe162c8047f57c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6043727174a34ecb8abe162c8047f57c" member_type: VOTER last_known_addr { host: "127.223.178.62" port: 34763 } } peers { permanent_uuid: "ab277f69e84345f49e335f7c49f2a1cb" member_type: VOTER last_known_addr { host: "127.223.178.61" port: 44151 } } peers { permanent_uuid: "9ff7142311674389b7ed69abe16dc022" member_type: VOTER last_known_addr { host: "127.223.178.60" port: 37935 } } }
I20260812 17:59:24.692760 229401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ff7142311674389b7ed69abe16dc022 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 17:59:24.694990 229400 catalog_manager.cc:1347] Generated new cluster ID: 7572ab14c2f54215aaecaafcdd82ca95
I20260812 17:59:24.695428 229400 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 17:59:24.696873 229406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ff7142311674389b7ed69abe16dc022 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6043727174a34ecb8abe162c8047f57c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6043727174a34ecb8abe162c8047f57c" member_type: VOTER last_known_addr { host: "127.223.178.62" port: 34763 } } peers { permanent_uuid: "ab277f69e84345f49e335f7c49f2a1cb" member_type: VOTER last_known_addr { host: "127.223.178.61" port: 44151 } } peers { permanent_uuid: "9ff7142311674389b7ed69abe16dc022" member_type: VOTER last_known_addr { host: "127.223.178.60" port: 37935 } } }
I20260812 17:59:24.698011 229406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ff7142311674389b7ed69abe16dc022 [sys.catalog]: This master's current role is: FOLLOWER
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800037ec8) received by PID 229064 (TID 0x7efee6a0b980) from PID 229064; stack trace: ***
@ 0x2b0c50 __tsan::CallUserSignalHandler() at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1915
@ 0x2b3340 rtl_sigaction() at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:2005
@ 0x7efeeca1d630 (unknown) at ??:0
I20260812 17:59:24.978106 229400 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 17:59:24.993366 229400 catalog_manager.cc:1504] Loading token signing keys...
@ 0x7efee7d84387 __GI_raise at ??:0
@ 0x7efee7d85a78 __GI_abort at ??:0
@ 0x2b5097 __interceptor_abort at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:263
I20260812 17:59:25.095352 229377 catalog_manager.cc:1259] Loaded cluster ID: 7572ab14c2f54215aaecaafcdd82ca95
I20260812 17:59:25.095700 229377 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9ff7142311674389b7ed69abe16dc022: loading cluster ID for follower catalog manager: success
@ 0x7efeeadafdf4 google::LogMessage::SendToLog() at ??:0
I20260812 17:59:25.190799 229400 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6043727174a34ecb8abe162c8047f57c: Generated new TSK 0
@ 0x7efeeadb09a0 google::LogMessage::Flush() at ??:0
I20260812 17:59:25.193738 229377 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9ff7142311674389b7ed69abe16dc022: acquiring CA information for follower catalog manager: success
I20260812 17:59:25.195341 229400 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 17:59:25.202906 229377 catalog_manager.cc:873] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
@ 0x7efeeadb5adb google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7efef4a00a2d kudu::cluster::ExternalMiniCluster::AddTabletServer() at ??:0
@ 0x7efef49feaab kudu::cluster::ExternalMiniCluster::Start() at ??:0
@ 0x356057 kudu::WebserverCrawlITest_TestAllWebPages_Test::TestBody() at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/webserver-crawl-itest.cc:?
@ 0x7efeeb90ae90 testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7efeeb8e8a92 testing::Test::Run() at ??:0
W20260812 17:59:25.613641 229126 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.223.178.61:44151 timed out after 1.532s (SENT) [suppressed 8 similar messages]
@ 0x7efeeb8e9da9 testing::TestInfo::Run() at ??:0
@ 0x7efeeb8ead85 testing::TestSuite::Run() at ??:0
@ 0x7efeeb8ff875 testing::internal::UnitTestImpl::RunAllTests() at ??:0
@ 0x7efeeb90bd50 testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7efeeb8fee2d testing::UnitTest::Run() at ??:0
@ 0x38c1cc RUN_ALL_TESTS() at /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/gtest/gtest.h:?
@ 0x38b08d main at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/test_main.cc:?
@ 0x7efee7d70555 __libc_start_main at ??:0
@ 0x2a9419 (unknown) at ??:?
I20260812 17:59:26.244768 229377 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9ff7142311674389b7ed69abe16dc022: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 17:59:26.444511 229260 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.223.178.60" port: 37935 } attempt: 10
I20260812 17:59:26.462891 229262 master_runner.cc:186] Error getting master registration for 127.223.178.61:44151: OK, instance_id { permanent_uuid: "ab277f69e84345f49e335f7c49f2a1cb" instance_seqno: 1786557548445897 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 17:59:26.464457 229262 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 17:59:26.571743 229260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab277f69e84345f49e335f7c49f2a1cb: Bootstrap starting.
I20260812 17:59:26.661558 229260 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab277f69e84345f49e335f7c49f2a1cb: Neither blocks nor log segments found. Creating new log.
I20260812 17:59:26.664888 229260 log.cc:824] T 00000000000000000000000000000000 P ab277f69e84345f49e335f7c49f2a1cb: Log is configured to *not* fsync() on all Append() calls
I20260812 17:59:26.677559 229260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab277f69e84345f49e335f7c49f2a1cb: No bootstrap required, opened a new log
I20260812 17:59:26.975543 229260 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab277f69e84345f49e335f7c49f2a1cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6043727174a34ecb8abe162c8047f57c" member_type: VOTER last_known_addr { host: "127.223.178.62" port: 34763 } } peers { permanent_uuid: "ab277f69e84345f49e335f7c49f2a1cb" member_type: VOTER last_known_addr { host: "127.223.178.61" port: 44151 } } peers { permanent_uuid: "9ff7142311674389b7ed69abe16dc022" member_type: VOTER last_known_addr { host: "127.223.178.60" port: 37935 } }
I20260812 17:59:26.977068 229260 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab277f69e84345f49e335f7c49f2a1cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:59:26.982417 229260 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab277f69e84345f49e335f7c49f2a1cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab277f69e84345f49e335f7c49f2a1cb, State: Initialized, Role: FOLLOWER
I20260812 17:59:26.984645 229260 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab277f69e84345f49e335f7c49f2a1cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6043727174a34ecb8abe162c8047f57c" member_type: VOTER last_known_addr { host: "127.223.178.62" port: 34763 } } peers { permanent_uuid: "ab277f69e84345f49e335f7c49f2a1cb" member_type: VOTER last_known_addr { host: "127.223.178.61" port: 44151 } } peers { permanent_uuid: "9ff7142311674389b7ed69abe16dc022" member_type: VOTER last_known_addr { host: "127.223.178.60" port: 37935 } }
I20260812 17:59:27.070160 229427 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab277f69e84345f49e335f7c49f2a1cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6043727174a34ecb8abe162c8047f57c" member_type: VOTER last_known_addr { host: "127.223.178.62" port: 34763 } } peers { permanent_uuid: "ab277f69e84345f49e335f7c49f2a1cb" member_type: VOTER last_known_addr { host: "127.223.178.61" port: 44151 } } peers { permanent_uuid: "9ff7142311674389b7ed69abe16dc022" member_type: VOTER last_known_addr { host: "127.223.178.60" port: 37935 } } }
I20260812 17:59:27.076854 229427 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab277f69e84345f49e335f7c49f2a1cb [sys.catalog]: This master's current role is: FOLLOWER
I20260812 17:59:27.090776 229235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ff7142311674389b7ed69abe16dc022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab277f69e84345f49e335f7c49f2a1cb" is_pre_election: true
I20260812 17:59:27.090657 229260 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab277f69e84345f49e335f7c49f2a1cb [sys.catalog]: configured and running, proceeding with master startup.
I20260812 17:59:27.092006 229235 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ab277f69e84345f49e335f7c49f2a1cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ff7142311674389b7ed69abe16dc022 in term 0.
I20260812 17:59:27.093153 229233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6043727174a34ecb8abe162c8047f57c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab277f69e84345f49e335f7c49f2a1cb" is_pre_election: true
I20260812 17:59:27.093931 229233 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ab277f69e84345f49e335f7c49f2a1cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6043727174a34ecb8abe162c8047f57c in term 0.
I20260812 17:59:27.169441 229229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6043727174a34ecb8abe162c8047f57c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab277f69e84345f49e335f7c49f2a1cb"
I20260812 17:59:27.175315 229229 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab277f69e84345f49e335f7c49f2a1cb [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:59:27.169165 229232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6043727174a34ecb8abe162c8047f57c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab277f69e84345f49e335f7c49f2a1cb"
I20260812 17:59:27.168838 229234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ff7142311674389b7ed69abe16dc022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab277f69e84345f49e335f7c49f2a1cb" is_pre_election: true
W20260812 17:59:27.178053 229232 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37346 (request call id 2) took 3119 ms (client timeout 1662 ms). Trace:
W20260812 17:59:27.092753 229235 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37362 (request call id 1) took 6165 ms (client timeout 1541 ms). Trace:
W20260812 17:59:27.170728 229233 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37346 (request call id 1) took 3213 ms (client timeout 1532 ms). Trace:
W20260812 17:59:27.180917 229235 rpcz_store.cc:269] 0812 17:59:20.927426 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 17:59:20.927674 (+ 248us) service_pool.cc:227] Handling call
0812 17:59:27.092640 (+6164966us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260812 17:59:27.178454 229232 rpcz_store.cc:269] 0812 17:59:24.058758 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 17:59:24.059090 (+ 332us) service_pool.cc:227] Handling call
0812 17:59:27.177981 (+3118891us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260812 17:59:27.182068 229233 rpcz_store.cc:269] 0812 17:59:23.957367 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 17:59:23.957637 (+ 270us) service_pool.cc:227] Handling call
0812 17:59:27.170597 (+3212960us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260812 17:59:27.177666 229234 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37362 (request call id 2) took 4583 ms (client timeout 1541 ms). Trace:
W20260812 17:59:27.189903 229234 rpcz_store.cc:269] 0812 17:59:22.593852 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 17:59:22.594079 (+ 227us) service_pool.cc:227] Handling call
0812 17:59:27.177580 (+4583501us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 17:59:27.186250 229230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6043727174a34ecb8abe162c8047f57c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab277f69e84345f49e335f7c49f2a1cb" is_pre_election: true
W20260812 17:59:27.192402 229230 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37346 (request call id 5) took 1571 ms (client timeout 1532 ms). Trace:
W20260812 17:59:27.192842 229230 rpcz_store.cc:269] 0812 17:59:25.621361 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 17:59:25.622863 (+ 1502us) service_pool.cc:227] Handling call
0812 17:59:27.192315 (+1569452us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 17:59:27.273543 229229 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ab277f69e84345f49e335f7c49f2a1cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6043727174a34ecb8abe162c8047f57c in term 1.
I20260812 17:59:27.276804 229231 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ab277f69e84345f49e335f7c49f2a1cb [term 1 FOLLOWER]: Refusing update from remote peer 6043727174a34ecb8abe162c8047f57c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260812 17:59:27.286972 229229 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37346 (request call id 6) took 1466 ms (client timeout 1662 ms). Trace:
W20260812 17:59:27.287437 229229 rpcz_store.cc:269] 0812 17:59:25.821010 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 17:59:25.821298 (+ 288us) service_pool.cc:227] Handling call
0812 17:59:27.286841 (+1465543us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 17:59:27.290875 229231 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37346 (request call id 4) took 2811 ms. Trace:
I20260812 17:59:27.291371 229231 rpcz_store.cc:276] 0812 17:59:24.479525 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 17:59:24.479819 (+ 294us) service_pool.cc:227] Handling call
0812 17:59:27.275224 (+2795405us) raft_consensus.cc:1457] Updating replica for 0 ops
0812 17:59:27.277723 (+ 2499us) raft_consensus.cc:1683] Filling consensus response to leader.
0812 17:59:27.290377 (+ 12654us) spinlock_profiling.cc:243] Waited 64.7 ms on lock 0x7b54000118d0. stack: 00007f18c3160033 00007f18c315ff30 00007f18c2ce1534 00000000004db2b4 00000000004db249 00007f18c9fe3471 00007f18ca05da09 00007f18cb03a2e9 00007f18c9811684 00007f18c9811612 00007f18c9811541 00007f18c98114bc 00007f18c9810772 00007f18c7e0e94c 00007f18c7e0dd86 00007f18c7e0d72f
0812 17:59:27.290739 (+ 362us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":206977408}
I20260812 17:59:27.294668 229384 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6043727174a34ecb8abe162c8047f57c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab277f69e84345f49e335f7c49f2a1cb" member_type: VOTER last_known_addr { host: "127.223.178.61" port: 44151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260812 17:59:27.378541 229214 tablet.cc:2369] T 00000000000000000000000000000000 P ab277f69e84345f49e335f7c49f2a1cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 17:59:27.590626 229441 log.cc:924] Time spent T 00000000000000000000000000000000 P ab277f69e84345f49e335f7c49f2a1cb: Append to log took a long time: real 0.101s user 0.002s sys 0.000s
I20260812 17:59:27.606025 229427 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab277f69e84345f49e335f7c49f2a1cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6043727174a34ecb8abe162c8047f57c. Latest consensus state: current_term: 1 leader_uuid: "6043727174a34ecb8abe162c8047f57c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6043727174a34ecb8abe162c8047f57c" member_type: VOTER last_known_addr { host: "127.223.178.62" port: 34763 } } peers { permanent_uuid: "ab277f69e84345f49e335f7c49f2a1cb" member_type: VOTER last_known_addr { host: "127.223.178.61" port: 44151 } } peers { permanent_uuid: "9ff7142311674389b7ed69abe16dc022" member_type: VOTER last_known_addr { host: "127.223.178.60" port: 37935 } } }
I20260812 17:59:27.606995 229427 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab277f69e84345f49e335f7c49f2a1cb [sys.catalog]: This master's current role is: FOLLOWER
I20260812 17:59:27.609977 229444 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab277f69e84345f49e335f7c49f2a1cb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6043727174a34ecb8abe162c8047f57c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6043727174a34ecb8abe162c8047f57c" member_type: VOTER last_known_addr { host: "127.223.178.62" port: 34763 } } peers { permanent_uuid: "ab277f69e84345f49e335f7c49f2a1cb" member_type: VOTER last_known_addr { host: "127.223.178.61" port: 44151 } } peers { permanent_uuid: "9ff7142311674389b7ed69abe16dc022" member_type: VOTER last_known_addr { host: "127.223.178.60" port: 37935 } } }
I20260812 17:59:27.610977 229444 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab277f69e84345f49e335f7c49f2a1cb [sys.catalog]: This master's current role is: FOLLOWER
I20260812 17:59:27.705009 229451 catalog_manager.cc:1259] Loaded cluster ID: 7572ab14c2f54215aaecaafcdd82ca95
I20260812 17:59:27.715590 229451 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ab277f69e84345f49e335f7c49f2a1cb: loading cluster ID for follower catalog manager: success
I20260812 17:59:27.809509 229451 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ab277f69e84345f49e335f7c49f2a1cb: acquiring CA information for follower catalog manager: success
I20260812 17:59:27.902514 229451 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ab277f69e84345f49e335f7c49f2a1cb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 17:59:27.910421 229438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6043727174a34ecb8abe162c8047f57c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6043727174a34ecb8abe162c8047f57c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6043727174a34ecb8abe162c8047f57c" member_type: VOTER last_known_addr { host: "127.223.178.62" port: 34763 } } peers { permanent_uuid: "ab277f69e84345f49e335f7c49f2a1cb" member_type: VOTER last_known_addr { host: "127.223.178.61" port: 44151 } } peers { permanent_uuid: "9ff7142311674389b7ed69abe16dc022" member_type: VOTER last_known_addr { host: "127.223.178.60" port: 37935 } } }
I20260812 17:59:27.912034 229438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6043727174a34ecb8abe162c8047f57c [sys.catalog]: This master's current role is: LEADER
W20260812 17:59:50.656040 229351 debug-util.cc:398] Leaking SignalData structure 0x7b08000b3ce0 after lost signal to thread 229270
W20260812 17:59:50.657117 229351 debug-util.cc:398] Leaking SignalData structure 0x7b08000b93a0 after lost signal to thread 229354
W20260812 18:00:31.997570 229173 debug-util.cc:398] Leaking SignalData structure 0x7b08000b7800 after lost signal to thread 229108
W20260812 18:00:31.998807 229173 debug-util.cc:398] Leaking SignalData structure 0x7b08000b8440 after lost signal to thread 229176
W20260812 18:00:51.541873 229173 debug-util.cc:398] Leaking SignalData structure 0x7b080008f540 after lost signal to thread 229108
W20260812 18:00:51.542913 229173 debug-util.cc:398] Leaking SignalData structure 0x7b08000a0560 after lost signal to thread 229176
W20260812 18:01:01.933137 229256 debug-util.cc:398] Leaking SignalData structure 0x7b08000a0240 after lost signal to thread 229183
W20260812 18:01:01.934309 229256 debug-util.cc:398] Leaking SignalData structure 0x7b08000ab280 after lost signal to thread 229259
W20260812 18:01:08.822624 229256 debug-util.cc:398] Leaking SignalData structure 0x7b08000a0680 after lost signal to thread 229183
W20260812 18:01:08.823509 229256 debug-util.cc:398] Leaking SignalData structure 0x7b08000a8400 after lost signal to thread 229259
I20260812 18:01:20.922492 229306 maintenance_manager.cc:419] P 9ff7142311674389b7ed69abe16dc022: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033363
I20260812 18:01:21.057777 229305 maintenance_manager.cc:643] P 9ff7142311674389b7ed69abe16dc022: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.134s user 0.056s sys 0.013s Metrics: {"bytes_written":6027,"cfile_init":1,"dirs.queue_time_us":1102,"dirs.run_cpu_time_us":356,"dirs.run_wall_time_us":454,"drs_written":1,"lbm_read_time_us":396,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3976,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":3,"thread_start_us":1244,"threads_started":2,"wal-append.queue_time_us":961}
I20260812 18:01:21.061822 229306 maintenance_manager.cc:419] P 9ff7142311674389b7ed69abe16dc022: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 185 bytes on disk
I20260812 18:01:21.064260 229305 maintenance_manager.cc:643] P 9ff7142311674389b7ed69abe16dc022: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":361,"lbm_reads_lt_1ms":4}
I20260812 18:01:24.199250 229131 maintenance_manager.cc:419] P 6043727174a34ecb8abe162c8047f57c: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033403
I20260812 18:01:24.273962 229130 maintenance_manager.cc:643] P 6043727174a34ecb8abe162c8047f57c: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.073s user 0.050s sys 0.010s Metrics: {"bytes_written":6027,"cfile_init":1,"dirs.queue_time_us":1014,"dirs.run_cpu_time_us":390,"dirs.run_wall_time_us":490,"drs_written":1,"lbm_read_time_us":298,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2980,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":3,"thread_start_us":1187,"threads_started":2,"wal-append.queue_time_us":1029}
I20260812 18:01:24.278055 229131 maintenance_manager.cc:419] P 6043727174a34ecb8abe162c8047f57c: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 185 bytes on disk
I20260812 18:01:24.280117 229130 maintenance_manager.cc:643] P 6043727174a34ecb8abe162c8047f57c: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":322,"lbm_reads_lt_1ms":4}
I20260812 18:01:27.313063 229214 maintenance_manager.cc:419] P ab277f69e84345f49e335f7c49f2a1cb: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033396
I20260812 18:01:27.401075 229213 maintenance_manager.cc:643] P ab277f69e84345f49e335f7c49f2a1cb: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.086s user 0.051s sys 0.014s Metrics: {"bytes_written":6027,"cfile_init":1,"dirs.queue_time_us":1196,"dirs.run_cpu_time_us":403,"dirs.run_wall_time_us":504,"drs_written":1,"lbm_read_time_us":345,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3190,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":3,"thread_start_us":1307,"threads_started":2,"wal-append.queue_time_us":1049}
I20260812 18:01:27.404855 229214 maintenance_manager.cc:419] P ab277f69e84345f49e335f7c49f2a1cb: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 185 bytes on disk
I20260812 18:01:27.407096 229213 maintenance_manager.cc:643] P ab277f69e84345f49e335f7c49f2a1cb: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":387,"lbm_reads_lt_1ms":4}
W20260812 18:01:32.373142 229256 debug-util.cc:398] Leaking SignalData structure 0x7b08000a6520 after lost signal to thread 229183
W20260812 18:01:32.375136 229256 debug-util.cc:398] Leaking SignalData structure 0x7b080009b980 after lost signal to thread 229259
W20260812 18:01:42.792119 229351 debug-util.cc:398] Leaking SignalData structure 0x7b080008cc00 after lost signal to thread 229270
W20260812 18:01:42.794042 229351 debug-util.cc:398] Leaking SignalData structure 0x7b080009a940 after lost signal to thread 229354